
OMNIX
Autonomous AI Desktop Agent
User Goal
"Open notepad and check weather"
Perception
YOLO detects screen state & UI bounding boxes
Agent Brain
LLM analyzes intent vs current screen
Planner
Generates execution step sequence
Tool Execution
Mouse, Keyboard, OS control via Playwright
Memory
Stores contextual execution context for next turns
Result
Task successfully completed
Overview
Multimodal AI desktop agent that can understand natural-language goals, perceive a Windows screen, plan tasks, control apps, automate browsers, use memory, and recover during execution.
What I Built
A modular Windows desktop agent with reasoning, vision, memory, task planning, browser automation, voice input, and desktop control.
What I Learned
How to design AI systems that combine reasoning, perception, memory, planning, and real-world tool execution instead of treating an LLM as a chat-only API.
Challenges & Insights
Dynamic desktop environments are unpredictable. OMNIX uses observe, understand, plan, act, verify, and recover loops to make execution more reliable.
The Problem
Most LLM tools are text-in, text-out chatbots. They don't natively understand the graphical interface of a computer, making them unable to automate real, dynamic desktop tasks where UI elements constantly change position and state.
My Approach
Instead of hardcoding macros, I built an architecture that combines computer vision (YOLO) with LLM reasoning. The agent takes a screenshot, detects UI elements, plans a sequence of actions, executes them via PyAutoGUI, and verifies the outcome before proceeding.
Key Features & Metrics
- Autonomous agent
- Computer vision
- Semantic memory
- Browser automation
- Voice pipeline
System Architecture
Input + Context
Natural-language and voice commands are combined with screen state, system context, memory, and user intent.
AI Brain + Task Planner
LLM reasoning converts high-level goals into structured multi-step plans with skill selection and recovery paths.
Agent Controller
Coordinates planning, observation, execution, verification, and recovery across skills, vision, and automation.
Skills / Vision / Automation
PyQt6, Ultralytics YOLO, Playwright, PyAutoGUI, Sentence Transformers, LLM APIs, and Edge TTS work together to operate the desktop.
Technical Challenges
Separating reasoning from execution so new capabilities can be added as independent skills instead of hard-coded commands.
Making the agent understand a dynamic Windows environment through screen perception and UI-element detection.
Designing a dependable Observe + Understand + Plan + Act + Verify + Recover loop for real computer-based tasks.
Outcome
A working local AI desktop agent that can navigate Windows applications, read visual data, and execute multi-step workflows based purely on natural language goals.
Technology Stack
Categories
Key Features
- Autonomous agent
- Computer vision
- Semantic memory
- Browser automation
- Voice pipeline