The Zaya universe · Capability guide

Many facets.
One companion.

A voice beside you. A helping hand on your desktop. A connection that travels with you. Discover the capabilities that come together in Zaya.

56catalogued capabilities
27connected categories
Youin control

A little possibility, everywhere

Find your facet.

Begin with the everyday. Explore the advanced when you’re ready. Each capability has its own requirements.

Nothing hidden behind the crystal

The complete feature collection.

Browse all 56 catalogued capabilities. Development labels describe components; they do not certify every flow in the public beta. Planned features are not available yet.

Showing 56 of 56 capabilities

Voice

03 capabilities
stablecross platform

Gemini Live Bidirectional Voice

Full-duplex conversational voice streaming over WebSockets with real-time speech synthesis.

Data route: cloud
Details & requirements

Streams microphone and response audio through the configured Gemini Live model for conversational voice. Response time depends on the provider and network.

  • Bidirectional audio streaming
  • Configurable voice settings
  • Conversational turns

Requires: Gemini API Key, Microphone, Audio Output

Read the guide ↗
stablewindows

Barge-In & Acoustic Interruption

Speak at any point during Zaya’s response to immediately halt queued speech and pivot.

Data route: local
Details & requirements

Local WebRTC VAD and energy-based speech detection catches owner speech during output playback, truncates speech buffers, and resumes active listening instantly.

  • Zero latency cutoff
  • Acoustic echo suppression
  • No provider round-trip needed

Requires: PyAudio, WebRTC VAD

Read the guide ↗
stablewindows

Local Wake-Word Gating & Push-to-Talk

Private on-device acoustic keyword spotting or hardware hotkey trigger.

Data route: local
Details & requirements

Keeps audio stream offline until the trained wake phrase or physical push-to-talk hotkey is held, preventing unintended ambient listening.

Requires: Local Hotkey Hook, OpenWakeWord Engine

Read the guide ↗

AI Engine

02 capabilities
stablewindows

Modular Local AI Engine

Split-pipeline local intelligence replacing cloud services with on-device models.

Data route: local
Details & requirements

Combines local WebRTC VAD, faster-whisper speech transcription, local LLM inference via Ollama / LM Studio / llama.cpp, and onnx Piper voice synthesis.

Requires: faster-whisper, Piper ONNX, Local LLM (Ollama/llama.cpp)

Read the guide ↗
stablecross platform

Multi-Key & Multi-Provider Failover

Automatic rate-limit recovery and multi-key rotation on 429 quota exhaustion.

Data route: optional cloud
Details & requirements

Maintains prioritized key rotation pools and provider fallbacks. When Gemini hits quota limits, Zaya seamlessly shifts to alternate keys or secondary providers without dropping conversation state.

Requires: Configured Keys in api_keys.json

Read the guide ↗

Computer

08 capabilities
stablewindows

Hierarchical Computer Operator

Multi-tiered desktop manipulation following an Observe → Identify → Act → Verify paradigm.

Data route: local
Details & requirements

Tiered execution engine attempting Native Control first, followed by Browser DOM, Windows UI Automation (UIA), Set-of-Mark visual tagging, vision segmentation, and precise coordinate fallback.

  • Never clicks blind coordinates
  • Post-action verification check
  • Atomic action rollback on failure

Requires: Windows UI Automation, PyAutoGUI / SendInput

Read the guide ↗
stablewindows

Application & Window Lifecycle

Open, minimize, focus, tile, and gracefully terminate desktop software.

Data route: local
Details & requirements

Direct Windows process and ShellExecute integration with title matching, multi-monitor coordinates, and process tree tree-kill safeguards.

Requires: win32gui, psutil

Read the guide ↗
betawindows

Floating Desktop Widgets

Create small desktop tools such as clocks, timers and calculators using Widget Forge.

Data route: local
Details & requirements

Create small desktop tools such as clocks, timers and calculators using Widget Forge.

Requires: Widget runtime

Read the guide ↗
betawindows

Companion Expressions & Controls

Use the taskbar character, expressions and Settings & Controls to personalize the desktop experience.

Data route: local
Details & requirements

Use the taskbar character, expressions and Settings & Controls to personalize the desktop experience.

Requires: Desktop companion

Read the guide ↗
betawindows

Runtime Logs & System Diagnostics

Inspect local runtime logs, resource information and diagnostic status when something needs attention.

Data route: local
Details & requirements

Inspect local runtime logs, resource information and diagnostic status when something needs attention.

Requires: Local desktop session

Read the guide ↗
betawindows

Selected Text Assistance

Read selected text and use it as context for your current request.

Data route: local
Details & requirements

Read selected text and use it as context for your current request.

Requires: Accessible selected text

Read the guide ↗
betawindows

Floating Content & Ghost Overlays

Display supported content and helper overlays alongside your desktop work.

Data route: local
Details & requirements

Display supported content and helper overlays alongside your desktop work.

Requires: Desktop display runtime

Read the guide ↗
betawindows

Computer Settings Assistance

Use supported system-settings actions while retaining owner permissions and confirmations.

Data route: local
Details & requirements

Use supported system-settings actions while retaining owner permissions and confirmations.

Requires: Windows settings access

Read the guide ↗

Browser

04 capabilities
stablecross platform

Hybrid Browser Automation & Recovery

Native browser launching alongside deep Playwright DOM automation.

Data route: local
Details & requirements

Launches your standard browser (Brave, Chrome, Firefox) for daily links, and creates isolated persistent Playwright contexts for deep programmatic navigation, form completion, and content extraction.

Requires: Playwright Chromium / Firefox

Read the guide ↗
betawindows

Web Search & Research

Search the web and gather context through configured search and browser tools.

Data route: optional cloud
Details & requirements

Search the web and gather context through configured search and browser tools.

Requires: Internet connection, configured providers

Read the guide ↗
betawindows

Weather Reports

Request weather information through the configured weather tool.

Data route: optional cloud
Details & requirements

Request weather information through the configured weather tool.

Requires: Internet connection, weather service

Read the guide ↗
betawindows

Flight Search Assistance

Look for flight options through the flight-finder integration. Availability depends on its configured services.

Data route: optional cloud
Details & requirements

Look for flight options through the flight-finder integration. Availability depends on its configured services.

Requires: Internet connection, flight search integration

Read the guide ↗

Vision

01 capabilities
stablewindows

On-Demand Visual Reasoning

High-resolution desktop or webcam snapshot analysis for contextual reasoning.

Data route: optional cloud
Details & requirements

Captures targeted window or desktop regions only when explicitly requested by user commands, analyzing UI layout, error popups, code snippets, or architectural diagrams.

Requires: Desktop Capture, Multimodal LLM / Gemini Vision

Read the guide ↗

Perception

01 capabilities
stablewindows

Continuous Local Screen Perception

Zero-cloud screen awareness monitoring active applications and UI region changes.

Data route: local
Details & requirements

Performs differential frame hashing, bounding-box motion tracking, and active window inspection locally at low CPU overhead. No live video stream is ever transmitted to cloud providers.

  • Completely private on-device
  • Diff-based low overhead
  • Selective OCR streaming

Requires: mss, OpenCV Contrib

Read the guide ↗

Files

02 capabilities
betawindows

File & Folder Management

Find, read, create, move and organize local files and folders with owner-controlled actions.

Data route: optional cloud
Details & requirements

Find, read, create, move and organize local files and folders with owner-controlled actions.

Requires: Filesystem access

Read the guide ↗
betawindows

Document Reading & Generation

Work with supported documents and generate files for your tasks. Format support depends on installed libraries.

Data route: optional cloud
Details & requirements

Work with supported documents and generate files for your tasks. Format support depends on installed libraries.

Requires: Document libraries

Read the guide ↗

Terminal

01 capabilities
betawindows

Owner-Controlled Terminal

Run terminal tasks in a working directory with the real execution policy and confirmation gate.

Data route: local
Details & requirements

Run terminal tasks in a working directory with the real execution policy and confirmation gate.

Requires: System shell

Read the guide ↗

Git

01 capabilities
stablecross platform

Verified Git Workflow Controller

Repository inspection, branch creation, commit signing, and verified push operations.

Data route: local
Details & requirements

Validates repository clean states, diff boundaries, author identity checks (sajancodes), and runs status checks before executing pushes to origin.

Requires: Git CLI

Read the guide ↗

Coding

02 capabilities
stablecross platform

Supervised OpenCode Orchestrator

Multi-pass coding assistant with structured briefs, tool-free review, and independent test verification.

Data route: local
Details & requirements

Compiles owner tasks into technical briefs, manages hidden OpenCode CLI worker processes, tracks git diffs, executes test suites, and enforces a strict three-pass correction ceiling.

Requires: OpenCode CLI, Git, Python / Node

Read the guide ↗
betawindows

Code Search & Helpers

Search project code and use configured code-generation helpers alongside supervised OpenCode work.

Data route: local
Details & requirements

Search project code and use configured code-generation helpers alongside supervised OpenCode work.

Requires: Project access, configured AI provider

Read the guide ↗

Missions

01 capabilities
stablewindows

Mission Supervisor & Durable State Engine

Long-running autonomous execution with bounded retries, proof-of-work, and restart recovery.

Data route: local
Details & requirements

Orchestrates complex multi-step user goals through an explicit 10-state state machine: planning, running, waiting confirmation, waiting clarification, waiting external, paused, interrupted, completed, completed unverified, failed, cancelled.

Requires: SQLite Mission Store

Read the guide ↗

Tasks

04 capabilities
betawindows

Reminders & Scheduled Calls

Schedule reminders and request a phone call at a specified time. Phone delivery needs a paired, reachable device.

Data route: optional cloud
Details & requirements

Schedule reminders and request a phone call at a specified time. Phone delivery needs a paired, reachable device.

Requires: Reminder scheduler, Android pairing for calls

Read the guide ↗
betawindows

Calendar & Meeting Scheduling

Query schedules and create or reschedule meetings through the configured calendar integration.

Data route: optional cloud
Details & requirements

Query schedules and create or reschedule meetings through the configured calendar integration.

Requires: Calendar connection

Read the guide ↗
betawindows

Background Task Monitoring

Follow ongoing work and inspect runtime status, progress and failures.

Data route: optional cloud
Details & requirements

Follow ongoing work and inspect runtime status, progress and failures.

Requires: Running desktop session

Read the guide ↗
betawindows

Proactive Owner Notices

Configure supported proactive notices and background checks for the owner, with delivery depending on the active session.

Data route: optional cloud
Details & requirements

Configure supported proactive notices and background checks for the owner, with delivery depending on the active session.

Requires: Desktop session, configured monitoring

Read the guide ↗

Memory

03 capabilities
stablewindows

Structured SQLite & FTS5 Knowledge Store

Continuous local conversation archive with semantic BM25 indexing and alias resolution.

Data route: local
Details & requirements

Stores private dialogue history in conversation.db. Fresh sessions start clean with zero context bloat; recalls at most five relevant messages on demand via indexed SQLite FTS5 queries.

Requires: SQLite3 with FTS5

Read the guide ↗
betawindows

Personal Profile & Preferences

Save supported personal preferences and profile information for your assistant.

Data route: local
Details & requirements

Save supported personal preferences and profile information for your assistant.

Requires: Local profile store

Read the guide ↗
betawindows

On-Demand Conversation History

Browse local conversation archives and request targeted recall without loading a previous session on startup.

Data route: local
Details & requirements

Browse local conversation archives and request targeted recall without loading a previous session on startup.

Requires: Local conversation archive

Read the guide ↗

Hand Control

01 capabilities
stablewindows

Optical Hand Landmark Tracking

Real-time 21-joint 3D hand skeleton inference running in an isolated sub-process.

Data route: local
Details & requirements

Uses Google MediaPipe Hand Landmarker model in an isolated process (.venv-hands) running at 30 FPS. Evaluates gestures: open palm (cursor), index poke (click), two fingers (selection), fist (scroll), thumbs up (Enter).

Requires: MediaPipe 0.10, OpenCV, Webcam

Read the guide ↗

Virtual Keyboard

01 capabilities
stablewindows

Depth-Invariant Air Keyboard

Floating mid-air typing interface with palm-relative depth adaptation and debouncing.

Data route: local
Details & requirements

Projects a translucent cyan keyboard plane. Automatically normalizes keystroke depth based on hand scale from the camera, enforcing single-strike latching and refusing to inject keystrokes into credential windows.

Requires: Ghost Keyboard Engine, SendInput

Read the guide ↗

Spatial

01 capabilities
stablewindows

3D Spatial Intelligence & CAD Workspace

Interactive WebGPU/WebGL 3D environment with GLB/glTF streaming and natural language planning.

Data route: local
Details & requirements

Babylon.js powered spatial canvas supporting progressive geometry streaming, exploded component views, X-ray inspection, semantic part labeling, and natural language scene transformations.

Requires: Babylon.js, Vite / WebGPU / WebGL

Read the guide ↗

WhatsApp

01 capabilities
stablecross platform

WhatsApp Hermes Linked Gateway

Persistent multi-device WhatsApp link supporting Observe, Draft, and Auto modes.

Data route: local
Details & requirements

Maintains a linked-device WhatsApp session via Hermes bridge. Supports per-contact mode assignment, urgent vocal escalation, and strict activity privacy boundaries (shares current song/task, never private chat history).

Requires: Hermes Gateway, Node.js

Read the guide ↗

Android

01 capabilities
stableandroid

Native Android Companion (Kotlin / Jetpack Compose)

A mobile connection to your laptop’s running Zaya instance.

Data route: local
Details & requirements

Native Kotlin app mirroring the companion state, live transcripts, active mission progress, WhatsApp messages, and file exchange over an encrypted local WebSocket bridge.

Requires: Android 10+ (API 29+), Encrypted Keystore

Read the guide ↗

Calls

02 capabilities
stableandroid

Zaya-Initiated Calls & Live Video Context

Incoming phone rings for urgent owner alerts and opt-in camera sharing.

Data route: local
Details & requirements

Zaya can proactively place priority calls to the owner phone for urgent alerts or completed tasks. During an active call, camera frames stream to the laptop for spatial visual reasoning.

Requires: Camera2 API, Android Telecom Framework

Read the guide ↗
betaandroid

Away & Presence Preferences

Tell Zaya when you are going out or back so phone-call delivery can use your stated availability.

Data route: local
Details & requirements

Tell Zaya when you are going out or back so phone-call delivery can use your stated availability.

Requires: Owner session, paired phone for delivery

Read the guide ↗

Phone Video

01 capabilities
betaandroid

Opt-In Phone Camera Context

Share phone camera frames during a call for visual context. Quality and latency depend on the device and connection.

Data route: optional cloud
Details & requirements

Share phone camera frames during a call for visual context. Quality and latency depend on the device and connection.

Requires: Paired Android companion, camera permission

Read the guide ↗

Media

02 capabilities
betawindows

Desktop Music & Media Controls

Control playback, track changes, mute and volume through Windows media controls.

Data route: local
Details & requirements

Control playback, track changes, mute and volume through Windows media controls.

Requires: Compatible media player, Windows

Read the guide ↗
betawindows

Video & YouTube Playback

Open videos and work with supported playback controls using your desktop and browser.

Data route: local
Details & requirements

Open videos and work with supported playback controls using your desktop and browser.

Requires: Browser or supported player

Read the guide ↗

Audio

02 capabilities
stablewindows

System Audio Ducking

Automatically attenuates background desktop sound during active speech.

Data route: local
Details & requirements

Windows CoreAudio session mixer drops background application audio to 20% when either owner or Zaya speaks, smoothly ramping back to full volume after release.

Requires: Windows CoreAudio / pycaw

Read the guide ↗
betawindows

Double-Clap Toggle

Use the optional microphone-based double-clap control where configured.

Data route: local
Details & requirements

Use the optional microphone-based double-clap control where configured.

Requires: Microphone, enabled clap detection

Read the guide ↗

Plugins

01 capabilities
betawindows

Plugin & Tool Extensions

Extend the desktop through configured plugins and discovered action tools. Additional integrations need their own setup.

Data route: optional cloud
Details & requirements

Extend the desktop through configured plugins and discovered action tools. Additional integrations need their own setup.

Requires: Plugin configuration

Read the guide ↗

Security

02 capabilities
stablecross platform

Owner Trust Boundary & ToolResult Architecture

Owner-only authority model treating external content as untrusted data.

Data route: local
Details & requirements

Strict privilege isolation: Web content, OCR text, WhatsApp messages, and tool outputs are strictly classified as DATA, NOT AUTHORITY. Destructive operations require on-screen owner confirmation.

Requires: Windows DPAPI, ExecutionPolicy

Read the guide ↗
stablewindows

Windows DPAPI Encrypted Vault

Windows user-account encryption for stored provider credentials.

Data route: local
Details & requirements

Uses Windows CryptProtectData to protect stored provider credentials under the Windows user account. This does not protect against a compromised, signed-in user session.

Requires: Windows Crypt32

Read the guide ↗

Remote Control

05 capabilities
betaandroid

Phone Contacts & Call Actions

Look up contacts and request supported calling actions on your paired phone.

Data route: local
Details & requirements

Look up contacts and request supported calling actions on your paired phone.

Requires: Android permissions, paired companion

Read the guide ↗
betaandroid

Phone Notifications & SMS

Access supported phone notifications and messaging actions after granting the required Android permissions.

Data route: local
Details & requirements

Access supported phone notifications and messaging actions after granting the required Android permissions.

Requires: Notification access or SMS permissions

Read the guide ↗
betaandroid

Phone Device & App Controls

Request supported app launches, media actions, flashlight controls and device status from your laptop.

Data route: local
Details & requirements

Request supported app launches, media actions, flashlight controls and device status from your laptop.

Requires: Paired Android companion, device permissions

Read the guide ↗
betaandroid

Phone Screen Sharing

Share a phone screen capture for visual assistance with explicit Android screen-capture consent.

Data route: local
Details & requirements

Share a phone screen capture for visual assistance with explicit Android screen-capture consent.

Requires: MediaProjection consent, vision provider

Read the guide ↗
betawindows

Owner-Requested Port Tunnels

Manage supported port tunnels for explicitly requested remote access. Exposing a service requires owner approval.

Data route: local
Details & requirements

Manage supported port tunnels for explicitly requested remote access. Exposing a service requires owner approval.

Requires: Tunnel provider, owner authorization

Read the guide ↗

Distribution

01 capabilities
betawindows

Windows Beta Installer

Install the published Windows beta using the official setup file and screenshot guide. The beta is unsigned.

Data route: local
Details & requirements

Install the published Windows beta using the official setup file and screenshot guide. The beta is unsigned.

Requires: Windows 10/11 64-bit

Read the guide ↗

Updates

02 capabilities
betawindows

Configured Game Update Assistance

Use the game-update tool for supported, configured update workflows.

Data route: optional cloud
Details & requirements

Use the game-update tool for supported, configured update workflows.

Requires: Supported game configuration, internet connection

Read the guide ↗
betawindows

Official Release & Update Checks

Find official releases and check update availability. Unsigned desktop betas require manual installation; automatic installation accepts signed releases only.

Data route: optional cloud
Details & requirements

Find official releases and check update availability. Unsigned desktop betas require manual installation; automatic installation accepts signed releases only.

Requires: Official release channel, internet connection

Read the guide ↗

Your first hello

Bring a little ice
to your everyday.

Start with the Windows beta. Choose your provider, set your permissions and make Zaya your own.

Get the Windows beta ↗