Ask ASTA
Work/OMNIX
SYSTEM
BUILDING
2026

OMNIX

Autonomous AI Desktop Agent

OMNIX
Flagship System

OMNIX

Autonomous AI Desktop Agent

1

User Goal

"Open notepad and check weather"

2

Perception

YOLO detects screen state & UI bounding boxes

3

Agent Brain

LLM analyzes intent vs current screen

4

Planner

Generates execution step sequence

5

Tool Execution

Mouse, Keyboard, OS control via Playwright

6

Memory

Stores contextual execution context for next turns

7

Result

Task successfully completed

OMNIX Runtime
System: Waiting for goal...
User: "Open notepad and check weather"

Overview

Multimodal AI desktop agent that can understand natural-language goals, perceive a Windows screen, plan tasks, control apps, automate browsers, use memory, and recover during execution.

What I Built

A modular Windows desktop agent with reasoning, vision, memory, task planning, browser automation, voice input, and desktop control.

What I Learned

How to design AI systems that combine reasoning, perception, memory, planning, and real-world tool execution instead of treating an LLM as a chat-only API.

Challenges & Insights

Dynamic desktop environments are unpredictable. OMNIX uses observe, understand, plan, act, verify, and recover loops to make execution more reliable.

The Problem

Most LLM tools are text-in, text-out chatbots. They don't natively understand the graphical interface of a computer, making them unable to automate real, dynamic desktop tasks where UI elements constantly change position and state.

My Approach

Instead of hardcoding macros, I built an architecture that combines computer vision (YOLO) with LLM reasoning. The agent takes a screenshot, detects UI elements, plans a sequence of actions, executes them via PyAutoGUI, and verifies the outcome before proceeding.

Key Features & Metrics

  • Autonomous agent
  • Computer vision
  • Semantic memory
  • Browser automation
  • Voice pipeline

System Architecture

01

Input + Context

Natural-language and voice commands are combined with screen state, system context, memory, and user intent.

02

AI Brain + Task Planner

LLM reasoning converts high-level goals into structured multi-step plans with skill selection and recovery paths.

03

Agent Controller

Coordinates planning, observation, execution, verification, and recovery across skills, vision, and automation.

04

Skills / Vision / Automation

PyQt6, Ultralytics YOLO, Playwright, PyAutoGUI, Sentence Transformers, LLM APIs, and Edge TTS work together to operate the desktop.

Technical Challenges

!

Separating reasoning from execution so new capabilities can be added as independent skills instead of hard-coded commands.

!

Making the agent understand a dynamic Windows environment through screen perception and UI-element detection.

!

Designing a dependable Observe + Understand + Plan + Act + Verify + Recover loop for real computer-based tasks.

Outcome

A working local AI desktop agent that can navigate Windows applications, read visual data, and execute multi-step workflows based purely on natural language goals.

Technology Stack

PythonPyQt6Ultralytics YOLOPlaywrightPyAutoGUISentence TransformersEdge TTSLLM APIs

Categories

AI AGENTSAI / MLAUTOMATION

Key Features

  • Autonomous agent
  • Computer vision
  • Semantic memory
  • Browser automation
  • Voice pipeline