Ask ASTA
Back to Notes
March 15, 2024

Building OMNIX's Perception System

Problem

Standard LLMs are blind to the desktop. To make an autonomous agent, it needs to see UI elements to know where to click or type.

Approach

Initially, I tried using pure OCR, but it failed to understand UI hierarchy (like buttons vs text). I shifted to using Ultralytics YOLO to detect interactive bounding boxes combined with accessibility trees.

Failed attempts

Taking screenshots every 100ms melted the CPU. The agent loop had to become discrete: Observe (take one frame) -> Plan -> Act -> Verify.

Lesson

Vision-language models alone aren't enough for reliable computer control. You need a structured symbolic representation of the screen (bounding boxes) that an LLM can reason over.