@spartaficial/spart-desk-agent

The agent layer of SPART-Desk: AI agents drive Windows applications through the engine, either from a shell with the spart-desk-cli command line or through the spart-desk-mcp MCP server. This page is for anyone who gives an agent a Windows application to drive, or builds on the session the tools delegate to.

Agents and humans use the same engine: every action goes through Desktop → Application → DesktopLocator, with the same resolver pipeline (the accessibility tree, and the screen where a locator says so), actionability checks, execution strategies and diagnostics. Nothing here talks to the helper executable, a screen provider or native input directly.

Human:  await app.window().getByRole("button", { name: "Save" }).click();
Shell:  spart-desk-cli snapshot app1 → c17 button "Save" → spart-desk-cli click c17
MCP:    desktop_snapshot → c17 = button "Save" → desktop_click({ ref: "c17" })

Command line: spart-desk-cli

For agents that have a shell (Claude Code, Copilot CLI, Codex, Cursor, …). There is no client configuration: the agent reads spart-desk-cli --help or the shipped skill.

npm i -g @spartaficial/spart-desk-agent
spart-desk-cli skill --install               # → .agents/skills/ (Codex, Copilot, Cursor, Gemini CLI) and .claude/skills/ (Claude Code); --install <dir> for another

spart-desk-cli start --allow-app charmap.exe # optional: a policy for this session (launch starts a default host otherwise)
spart-desk-cli launch charmap.exe            # app1 launched charmap.exe · window "Character Map"
spart-desk-cli snapshot app1 | findstr checkbox   # compact view inline; the complete snapshot is in .spart-desk-cli/app1-s1.txt
spart-desk-cli click c5                      # clicked c5 (checkbox "Advanced view") via uia.invoke
spart-desk-cli inspect c2                    # a read: exported as an assertion
spart-desk-cli export app1 --out tests/charmap.test.ts   # the recorded flow as a runnable test
spart-desk-cli stop

The skill is in the open Agent Skills format (SKILL.md with name and description), which every agent above reads; only the lookup directory differs, which is why --install writes the interoperable .agents/skills/ and Claude Code's .claude/skills/. An agent without skills support needs one line in its instructions file (AGENTS.md, CLAUDE.md, .github/copilot-instructions.md): for Windows desktop applications use spart-desk-cli; run spart-desk-cli skill first — the command line is self-describing.

Each command is a short-lived process. The applications and refs live in a session host — spart-desk-mcp --transport http — that the CLI starts on demand (detached) and that stops on stop or after an hour without commands. -s <name> / SPART_DESK_CLI_SESSION keep several sessions apart. The host listens on 127.0.0.1 only, on a random port; every request needs the bearer token from the session file %TEMP%\spart-desk-cli\<name>.json (SPART_DESK_CLI_HOME relocates it), and its log is next to that file.

Command	Effect
start [host options]	Start the host with a policy: the spart-desk-mcp flags below, plus --idle-timeout <minutes>.
status · sessions · stop [--all] [--force]	Inspect and end sessions; stop closes the applications the session launched first, leaves the ones it attached to running, and returns once the host has exited; a command run meanwhile waits for that too, then starts a new host. --force asks the host to stop for 3 s, then ends its process tree: the helpers and the applications it launched, never one it attached to.
launch <exe> [--arg <a>]... [--cwd <dir>] [--process <name> [--join-existing]] [--window-timeout <ms>] [--java | --no-java]	Start an application → app1. --process names the process a launcher stub hands off to (notepad.exe --process Notepad); --join-existing lets it hand over to an instance of that process that was already running; --java turns the Java Access Bridge on for it, --no-java never.
launch --app-id <id> [--arg <a>]...	Start a Store app or packaged application by its app ID (Microsoft.WindowsCalculator_8wekyb3d8bbwe!App), activated with its arguments → app1.
launch --uri <uri> [--app-id <id> | --process <name> | --title <title>] [--join-existing]	Open a URI by its protocol (ms-settings:display, calculator:, an application's own scheme) → app1; its scheme (ms-settings:) is allow-listed. --app-id activates that packaged application for it; --process or --title finds the application when the shell names no process.
apps · close <app>	List / terminate applications.
snapshot <app> [--root <ref>] [--depth <n>] [--max <n>] [--all] [--enrich <level>] [--changes] [--bounds] [--locators] [--query <text>] [--role <role>] [--limit <n>] [-q] [--no-save]	Compact view (interactive controls, capped) with refs, one line each (--json for the object form, --locators adds each control's stable locator; --query / --role keep only the controls that match, best first, at most --limit, each with its ancestors); the complete snapshot — every control with its ref and locator — is saved to .spart-desk-cli/<app>-<snapshot>.txt to grep and diff (-q prints only the header and the file name, --no-save skips it).
click <ref> [--right] [--double] [--modifiers <keys>] [--position <x>,<y>] · fill <ref> <text> · press <ref|app> <key> · type <ref|app> <text>	Act on controls by ref; press app1 Control+S and type app1 "Hi!" go to a window of the application: --window <title>'s, else the one in front, else the main window. click, fill and press take --until <ref>:<state> (or <ref>:value=<text>, <ref>:contains=<text>) and --sends <n> (1–5): sent again while the check fails.
set <ref> --check|--uncheck|--expand|--collapse|--option <text>|--window <state>|--bounds <x>,<y>,<width>,<height>	Set a state only when it does not hold (desktop_set). --bounds places a window's frame, in physical screen pixels, _ or nothing keeping a part as it is (--bounds _,_,1024,768 sizes it where it is).
hover <ref> · drag <ref> <to-ref> [--hold <ms>] [--steps <n>]	Bring the mouse over a control (tooltips, menus that open on hover); drag a control onto another with the real mouse.
batch <step> [+ <step>]...	Up to 20 of click, fill, set, press, type, scroll, hover and expect <ref> [<state>|value=<text>|contains=<text>] (a check that ends the batch when it does not hold), each written as its own command and separated by a lone +, in one call: batch fill c5 Ada + fill c6 Lovelace + click c9 + expect c12 contains=Saved. Every step is checked before the first runs; the first that fails ends the batch (exit 1). Prints what each step did and what changed.
confirm <id>	Run an action that waits for confirmation (a host started with --confirm <family>), once the user agreed to what its error said.
inspect <ref> [--enrich <level>] [--all] [--depth <n>] [--bounds] [--details] [--properties] [--timeout <ms>]	A control's own text and what is inside it, with refs (--all: text and structure inside it too). --details also reads what the control says of itself (help text, item status, keys, password, required, read-only, range, MSAA); --properties reads that and every property and pattern its provider reports, by UI Automation's names (desktop_inspect with all).
table <ref> [--offset <n>] [--limit <n>] [--scroll]	A grid's columns, then one line per row (--scroll: the whole of a virtualised grid).
row <ref> --where <column>=<text> [--where ...] [--contains] [--no-scroll] [--all-pages] · item <ref> <text> [--contains] [--no-scroll]	The one row of a grid by its cells (a text written /like this/i is a pattern; --all-pages also looks for a second one on every page), or the one item of a list by its text, paged into view: its ref and what it holds.
tree <ref> [--expand] [--depth <n>] [--max <n>]	A tree's items, indented by level, open or closed, selected, each with its ref (--expand: open closed items to --depth levels, as a user does).
scroll <ref> [--direction up|down|left|right] [--to top|bottom|view] [--wheel <n>] [--timeout <ms>]	Scroll a control or the view it sits in; --to view brings the control itself into view; --wheel notches, positive scrolls down.
wait <app> --ref <ref> | --role <r> --name <n> | --text … | --label … | --title … | --automation-id … | --ocr … | --vision … [--window <title>] [--state … | --rows <n>]	Wait until a control reaches a state, or until the table it holds shows n rows (0: empty; exported as toHaveRowCount / toBeEmpty). A query looks in the main window and its dialogs, or in the window --window names. --ocr is text read off the screen (getByOcr), where --text is what the accessibility tree reports.
screenshot <app> [--window <title>] [--out <file>]	PNG of a window of the application (--window's, else the one in front, else the main window); default .spart-desk-cli/<app>-<time>.png.
export <app> [--out <file>] [--title <t>] [--runner vitest|node|playwright] [--mode test|page]	The application's recorded flow as a runnable TypeScript test (launch in a before hook, every successful action as locator code, every inspect as an assertion, dispose after; --mode page puts the controls in a page class of getters). Printed, or written with --out; works after close; export before stop. Runner: Vitest when package.json depends on it, else Playwright Test when it depends on @playwright/test, else node:test.
attach [--process <name>] [--title <t> | --title-pattern <re>] [--pid <n>] [--hwnd <n>] · shell [--display <n|all>] · mouse <app> <x> <y> [--right] [--action click|move|down|up|wheel|drag] [--delta <n>] [--to <x>,<y>] · clipboard <app> [--write <text>] · screenshot --screen [--display <n|all>] [--out <file>] · run <program> [<arg>...] [--cwd <dir>] [--env <name>=<value>]... [--input <text>] [--timeout <ms>] (arguments that start with - after --)	Only on a host started with --allow-attach / --allow-shell / --allow-mouse / --allow-clipboard / --allow-screen / --allow-run <program>. --title-pattern is a regular expression the main window's title matches, only beside --process or --pid.
skill [--install [<dir>]]	Print SKILL.md, or install it for every agent (.agents/skills/ and .claude/skills/) or into <dir>.

Results that carry UI text (snapshot, inspect, apps, export) are printed exactly as the MCP server returns them — notice and <untrusted-ui-content> fence included; actions print one line; errors go to stderr with their name and code and exit 1 (2 for a usage mistake). .spart-desk-cli/ is the command line's project-local output folder: it is created with a .gitignore that ignores everything in it, its snapshot files carry the same fence as stdout, and it is safe to delete. The session host never writes into the project.

MCP server: spart-desk-mcp

For MCP clients (Claude Desktop, Claude Code, VS Code, …):

npm i -g @spartaficial/spart-desk-agent
spart-desk-mcp --allow-app notepad.exe --allow-app "C:\\Apps\\*"

Claude Code / Claude Desktop / VS Code (mcp.json):

{
  "mcpServers": {
    "spart-desk": {
      "command": "npx",
      "args": ["spart-desk-mcp", "--runtime", "local", "--allow-app", "MyApp.exe", "--log", "C:\\temp\\spart-desk-agent.jsonl"]
    }
  }
}

To run the applications in the Windows Sandbox (experimental), pass --runtime sandbox --sandbox-only instead of --runtime local.

Flag	Effect
--runtime local|sandbox	Where applications run (default local; sandbox, experimental, is the Windows Sandbox runtime).
--sandbox-only (experimental)	Refuse every action unless the runtime is the Windows Sandbox.
--allow-app <name|pattern>	Allow-list of executables (notepad.exe, C:\Apps\*); repeatable. Unset: any.
--restrict-key <chord>	Another restricted chord, repeatable; Alt+F4, Control+Alt+Delete, Control+Shift+Escape, Meta+L and Meta+* are always restricted.
--allow-mouse	Registers desktop_mouse (real mouse input at a point of a window, in the pixels of its desktop_screenshot: clicks, moves, presses, wheel turns, drags). Off by default.
--allow-clipboard	Registers desktop_clipboard (read the clipboard, or put text on it for the application to paste). Off by default: what is read may be anything the user copied.
--allow-screen	Registers desktop_screen (a picture of a whole display, or of every one: every window on it and the taskbar, not only the session's applications; the guest's desktop in the sandbox runtime). Off by default.
--allow-automation <progId|document>	Registers desktop_automation for that interface: an application's own automation interface by its ProgID (Excel.Application) or a document's path (a pattern with * too); repeatable. Off by default: a call reaches any member of the interface, which runs code inside the application.
--allow-support	Registers desktop_support_capture: a support bundle of an application of the session, pictures of its windows included, written to a folder on this machine, redacted unless asked otherwise; local runtime only.
--allow-run <program>	Registers desktop_run for that program (a name such as reg.exe, a path, or a pattern with *); repeatable. Off by default: running a program is running code, so allow only what the agent needs, or run the applications in the Windows Sandbox (--runtime sandbox, experimental).
--allow-attach	Registers desktop_attach (automate applications the agent did not launch; the allow-list applies to the process name). Off by default.
--allow-shell	Registers desktop_shell (the Windows shell: main taskbar, tray, notification centre, Start and Search while open). Off by default; the allow-list never opens it.
--confirm <family>	Actions of that family (launch, attach, shell, close, click, fill, press, scroll, mouse, clipboard, run) wait for desktop_confirm, which this registers; repeatable. Have your client ask you before each desktop_confirm call: that makes it your confirmation.
--timeout <ms>	Default auto-wait timeout.
--log <file>	Append one JSON line per tool call (action log).
--redact-pattern <re>	A regular expression (matched without regard to case) whose matches are replaced by [redacted] in every result and log line, beside the secrets always replaced; repeatable. A pattern that could take time without end (a repeated group holding a quantifier or an alternation, a backreference, more than 200 characters) is refused, as spart-desk support --redact-pattern refuses it.
--no-redact	Return and log everything as the application showed it: no secret replaced, a password field's value shown and what was filled into it exported.
--transport stdio|http	http serves MCP over loopback HTTP as a spart-desk-cli session host (needs --session <name>; --session-home <dir> relocates the session file). Default stdio.
--idle-timeout <minutes>	http hosts: exit after this long without a request (default 60; 0 = never).

Tools

Tool	Purpose
desktop_list_applications	Applications of this session with ids, targets, windows, the runtime, and the engine's capabilities map — what this session can do, named by ability (uia, nativeInput, screenshot, text, imageMatch, elements, grounding).
desktop_launch	Start an application (allow-list, confirmation policy); returns app1. processName names the process a launcher stub hands off to (Windows 11 Notepad); it is allow-listed like the executable. appId instead of executable starts a Store app or packaged application by its app ID, allow-listed as written; it takes args (what it is activated with) and windowTimeout beside it. uri instead of executable opens a URI by its protocol, its scheme allow-listed as ms-settings:; appId beside it names the packaged application to activate for it, processName or windowTitle the application to find when the shell names no process (each allow-listed too), joinExisting a running instance that took the URI.
desktop_close	Terminate an application.
desktop_snapshot	Controls of an application with refs (c17, a control keeps its ref while it is observed), one line per control indented by depth. maxControls (default 200, deepest dropped first), maxDepth, root (scope to a ref's subtree), query and role (only the controls whose name, value, automation id or cells hold the text, of that role, best first, at most limit — DEFAULT_SNAPSHOT_MATCHES, 20 — each with its ancestors; text included unless interactiveOnly; every other control keeps its ref; matched says how many matched), format: "json", enrich (structure — the default, no pixels — text, elements), changes: true renders only what was added or changed since the last snapshot, bounds: true restores coordinates for controls the accessibility tree named, locators: true appends each control's stable locator, full: true adds the complete view (every scoped control, text and structure included) as a second fenced content or as full in the JSON form. tables lays out tables the tree does not report as grids, from locators in the string grammar ({ root: "automationId=orders", rows: "className=OrderRow", columns: ["Number", "Customer"], window?: "Orders" }, window the title of the window the root is in, default the main window), so their rows show their cells; AgentSessionOptions.tables does it for every snapshot.
desktop_click	Click a ref (UIA invoke first, real input otherwise); button, clickCount, and modifiers (keys held: Control, Shift, Alt) or position (pixels from its top-left corner), which make it real input and export as the click's options.
desktop_fill	Replace a text control's value by ref.
desktop_set	Set a ref's state, only when it does not hold: checked, expanded, option (what a combo box or list shows), window (minimize, maximize, restore of a window or dialog) or bounds ({ x?, y?, width?, height? }: where a window or dialog's frame is to be, in physical screen pixels). Exports as check(), uncheck(), expand(), collapse(), selectOption(), the window call or setBounds(), where a click would not say what was meant.
desktop_press	Press a key/chord (key) or type text key by key (text) on a ref, or into a window of the app: window's by title, else the one in front, else its main window; restricted chords refused.
desktop_inspect	A control's own text and what is inside it, as its own snapshot: the other half of progressive disclosure. Reads only that control's area, and never invalidates references outside it. enrich, maxDepth, interactiveOnly, bounds, details (also what the control says of itself: help text, item status, keys, password, required, read-only, selected, range, MSAA), all (with details, which it implies: every property and pattern the control's provider says it has, by UI Automation's names), timeout.
desktop_read_table	A grid's or table's rows as data: its columns once, then one line per row with each cell's text in column order, offset/limit for a page (default 50 rows), scroll: true for the whole of a virtualised grid, read only as far as the page asked for; a later page of the same whole read is served from it until another tool is called, for 30 s at most (PAGED_READ_KEPT_MS). A table a snapshot or the session defined (tables) is read by its definition. Far fewer tokens than the rows' controls in a snapshot.
desktop_find_row	One row of a grid or table by its cells (where: { column: value }, each a whole text, a pattern written "/^Jan/i" or a number the cell shows; exact), waiting and paging a virtualised grid to it, strict: references to the row and to what it holds; strict: "all-pages" looks at every page for a second one. A defined table is searched by its definition.
desktop_sort_table	Sorts a grid by a column (column, descending), clicking its header with the real mouse until the rows shown are in that order, at most three times; authorised as a click, and recorded for the exported test.
desktop_read_list	A list's items as text, one line each, offset/limit for a page, scroll: true for the whole of a virtualised list, as desktop_read_table reads a grid.
desktop_find_item	One item of a list by its text (text, exact), waiting and paging a virtualised list to it, strict: references to the item and to what it holds.
desktop_read_tree	A tree's items as data, one line each, indented by level, open or closed, selected, with each item's reference; expand: true opens collapsed items top down as a user does, to maxDepth (default 3, DEFAULT_TREE_DEPTH) and maxNodes (default 200, DEFAULT_TREE_NODES), which acts on the application and is authorised as desktop_set with expanded.
desktop_candidates	Every locator that could address a referenced control, best first, as the engine weighs them: code, class, matches, resolvers, reasons, and the recommended one. Read-only.
desktop_scroll	Scroll a control, or the view it sits in, by direction, to top/bottom, or by a wheel amount (wheel: notches, positive scrolls down) — so a control a long list has not reached yet becomes addressable. to: "view" brings the control itself into view, exported as scrollIntoView().
desktop_hover	Bring the mouse over a ref and leave it there: tooltips, menus that open on hover, a row's buttons.
desktop_drag	Drag a ref onto another (to) with the real mouse; hold, steps.
desktop_wait_for	Wait for a ref or a semantic query (role+name, text, label, title, automationId, ocr — text read off the screen — or vision) to reach a state, or with rows until the table it holds shows that many rows (0: empty). A query looks in the application's main window and the dialogs it owns, or in the window window names by its title.
desktop_screenshot	PNG of one window of the application, as image content with its size: window's by title, else the one in front, else its main window. desktop_mouse takes points in the same pixels.
desktop_batch	Up to 20 steps on one application in one call (click, fill, set, press, type, scroll, hover, each with its own tool's fields, and expect: a check — a ref, or role/name, text, label with application — of a state (visible, hidden, enabled, disabled, checked, unchecked), a value held whole or text it contains, which ends the batch with code: "assertion" when it does not hold; click, fill and press take until, the same check, and are sent again, up to sends times, while it fails), then one snapshot of what changed. Every step is checked before the first runs (references, arguments, restricted keys, confirmation), so a refused step runs nothing; the first that fails ends the batch, and the result, an error then, says which and why. Each step is logged and recorded as a call of its own.
desktop_export_test	The application's recorded flow as a runnable TypeScript test: launch (or attach) in a before hook, every successful action as locator code, every desktop_inspect as expectDesk(...).toHaveText(...), dispose in an after hook. title, runner: "vitest" | "node" | "playwright", mode: "page" for the controls as getters of a page class, format: "json" for { title, runner, steps, source }. Works after desktop_close.
desktop_attach	Advanced, off by default: attach to a running application by process name / window title / pid / window handle; windowTitlePattern instead of windowTitle, a regular expression (at most MAX_TITLE_PATTERN characters, refused when it could take long to match), only beside processName or pid; logged.
desktop_shell	Advanced, off by default: the Windows shell as an application (a display's taskbar, tray, notification centre, Start and Search while open; display: 0 primary, 1… another, "all"); closing it stops nothing; logged.
desktop_mouse	Advanced, off by default: real mouse input at a point of a window (window, default as desktop_screenshot), in physical pixels from its top-left corner; action is click (default), move, down, up, wheel (deltaY, positive down), drag (to toX,toY), a finger's tap (count: 2 for a double tap), a pen's pen-tap, or pen-stroke from x,y through points (at most MAX_STROKE_POINTS; pressure from 0 to 1 for either pen action); every use is logged.
desktop_clipboard	Advanced, off by default: the clipboard's text, fenced as untrusted content, or with write text put on it; a write is logged by its length and exported as app.clipboard.writeText(...).
desktop_screen	Advanced, off by default: a PNG of a whole display (display: 0 the primary one, 1… another, "all", the default, every one), as image content with its size; logged. desktop_screenshot pictures a window of an application.
desktop_automation	Advanced, off by default: gets, sets or calls a member of an application's own automation interface the operator allowed with --allow-automation: application, progId or document, op (get, set, call), path (members joined by dots, Workbooks(1).Sheets("Orders").Range("A1").Value), value (for a set; a block of cells as rows of values) or args (for a call). Answers the value read, fenced as untrusted content. Logged by the interface, the operation and the path, never a value or an argument.
desktop_support_capture	Advanced, off by default: writes a support bundle of an application, as npx spart-desk support capture --app <pid> writes one: application, out (the folder; a new one in the temporary folder by default), redact (default true), notes. Answers the folder, the windows captured and the notes. Local runtime only.
desktop_run	Advanced, off by default: runs a program the operator allowed with --allow-run to its end where the applications run (the guest, in the Windows Sandbox runtime, experimental), to arrange or clean up: command, args (each passed as one; no shell), cwd, env, input, timeout. Answers the exit code (non-zero is a result, not an error) and the output, each stream cut at MAX_RUN_OUTPUT_CHARS characters (truncated says which), fenced as untrusted content. Logged by the program's name and how many arguments it had, never the arguments; at most MAX_RUN_ARGS arguments.
desktop_confirm	Only with --confirm: runs an action refused with ConfirmationRequiredError, exactly as it was asked, once, within two minutes.

Every tool declares all four MCP annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint).

Untrusted UI content

Control names, values, captions and window titles come from the application — or from whatever it displays. Results that carry such text (desktop_snapshot, desktop_inspect, desktop_list_applications, the snapshot after a desktop_batch, an error that names what would wait for confirmation) start with a notice and wrap the payload in <untrusted-ui-content> markers so the agent treats it as data, never as instructions.

Snapshot shape

Default (compact text; a fraction of the tokens of JSON):

snapshot s1 of app1: 3 controls; references stay valid while the control is still there
c1 window "Editor"
  c2 button "Save" id=btnSave
  c3 textbox "Name" value="Ada"

Coordinates are off by default for controls the accessibility tree named: an agent acts by reference, so they were noise on every line. They stay for controls only the screen saw, where the coordinates are the identity, and bounds: true brings them back:

c1 window "Editor" @0,0 800x600 locator=getByRole("window", { name: "Editor" })
  c2 button "Save" id=btnSave @100,200 80x32 locator=getByRole("button", { name: "Save" })
  c3 textbox "Name" value="Ada" @100,240 200x24 locator=getByRole("textbox", { name: "Name" })

With locators: true each line ends in the stable locator the engine suggests for the control (locator=getByRole("button", { name: "Save" })); format: "json" always carries it:

{ "id": "s3", "application": "app1", "takenAt": 1758553200000,
  "enrichment": { "requested": "text", "applied": "text" },
  "total": 3, "truncated": false, "omitted": 0, "compared": true, "controls": [
  { "ref": "c17", "role": "button", "name": "Save", "depth": 1, "source": "uia", "key": "/window/button:Save", "locator": "getByRole(\"button\", { name: \"Save\" })" },
  { "ref": "c18", "role": "text", "name": "Total: 42.00", "depth": 1, "bounds": { "x": 300, "y": 200, "width": 96, "height": 18 }, "source": "ocr", "confidence": 0.94, "changed": "changed", "locator": "getByText(\"Total: 42.00\")" }
] }

Large trees are capped: when more than maxControls controls remain after scoping and filtering, the deepest are dropped first (document order is kept), truncated/omitted say so, and the agent should scope with root or maxDepth rather than raise the cap. The cap only shapes what is shown: every scoped control gets a ref, and full: true returns all of them (text and structure included, each with its locator) — the command line writes that view to .spart-desk-cli/ on every snapshot so an agent greps it instead of reading it.

UIA and vision controls share one shape; source/sources keep the provenance. A reference belongs to a control, not to a snapshot: the engine derives a key from what a control is, so the same button keeps the same ref however often the application is observed again, and a snapshot can show only what changed. Every action re-resolves the ref against the live UI (through the engine's controlId selector). That includes controls only the screen saw: text read off a window whose tree names nothing is addressed by what it reads and where it sits (ocr:Export@canvas#2), so a reference to it is acted on again even after the window moved. A ref whose control is no longer observed fails with StaleReferenceError, which names what it used to be and tells the agent to take a new snapshot; a ref is never handed to another control.

From exploration to a test

The session records every successful action per application (launch or attach, click, fill, press, text reads, waits, raw clicks) together with the stable locator of the ref it targeted. desktop_export_test / spart-desk-cli export render that recording as the test spart-desk init would have written:

beforeAll(async () => {
  desktop = await Desktop.create(); // reads spart-desk.config.ts
  app = await desktop.launch({ executable: "charmap.exe" });
  const main = app.window();
});

afterAll(() => desktop.dispose());

test("automates charmap.exe", async () => {
  await main.getByRole("checkbox", { name: "Advanced view" }).click();
  await main.getByRole("textbox", { name: "Characters to copy :" }).fill("Hello");
  await expectDesk(main.getByRole("textbox", { name: "Characters to copy :" })).toHaveText("Hello");
});

Failed, stale and refused actions are never recorded; a control without a stable locator becomes a comment to complete by hand. The recording survives close (close, then export) and is dropped when the session ends.

Safety controls (all deterministic, no LLM provider involved)

AgentPolicy — application allow-list, sandboxOnly (experimental), restrictedKeys (added to the chords always refused), allowMouse, allowClipboard, allowScreen, allowAttach, allowShell, allowRun (the programs desktop_run may start), allowAutomation (the interfaces desktop_automation may reach), allowSupport (desktop_support_capture), sensitiveActions + confirm callback, beforeAction hook (throw/return false to block). With confirmation: "pending" (--confirm <family>), a sensitive action is refused with ConfirmationRequiredError and waits under its id for desktop_confirm, for two minutes; a client that asks its user before each desktop_confirm call makes that the user's confirmation. It is a speed bump the client enforces; the confirm callback stays the hard gate. A batch is confirmed as a whole, before any step runs. A sensitive action names a family, and the request's operation names the gesture in it: click covers hover and drag by reference, mouse every gesture at coordinates (click, move, down, up, wheel, drag), clipboard a read and a write. The agent caps a drag's hold at 60 s; the engine itself does not. Every tool call is logged with runtime identity, target, outcome, duration and the engine's operation record (resolver, backend, strategy); typed text is logged by length only. Typed values are part of the recording, though, and appear in an exported test, except what went into a password field.

Secrets are kept out of what the agent returns (redact, on by default). Every text a tool returns, an error included, and every log entry has the shapes of well-known secrets replaced by [redacted]: a PEM private key block, an AWS access key id, a GitHub token, an sk- API key, a JSON Web Token, the token after Bearer. A text box is asked once, before the first fill or typing into it, whether it is a password field; one that is shows [redacted] as its value and text from then on (snapshots, desktop_inspect, its Value properties), what went into it exports as a placeholder to supply from a secret store, and its text is never recorded as an assertion. redact: { patterns } (--redact-pattern <re>) looks for the operator's own patterns too; redact: false (--no-redact) turns it all off. Pictures are not painted over, and the session's methods hand programmatic callers what the engine read: it is what the MCP server, and so the command line, returns that is redacted.

Programmatic use

import { Desktop } from "@spartaficial/spart-desk";
import { AgentSession, createSpartDeskMcpServer } from "@spartaficial/spart-desk-agent";

const desktop = await Desktop.create({ runtime: "local" });
const session = new AgentSession({ desktop, policy: { allowedApplications: ["MyApp.exe"] } });
const server = createSpartDeskMcpServer({ session });
// connect `server` to any MCP transport, or drive `session` directly from your own agent loop

For the Windows Sandbox (experimental), create the desktop with runtime: "sandbox" and add sandboxOnly: true to the policy.

AgentSession is usable without MCP at all — it is the engine-facing core the tools delegate to.

Library API

Area	Exports
The session	AgentSession and AgentSessionOptions; one method per tool, each taking one request object and answering with a result: LaunchRequest, AttachRequest (a title pattern of at most MAX_TITLE_PATTERN characters), ShellRequest, CloseRequest, AgentApplication; SnapshotRequest, AgentSnapshot, AgentControl, RemovedControl; ClickRequest, FillRequest (each with UntilRequest's until and sends), SetRequest, HoverRequest, DragRequest, ScrollRequest, ActionResult; PressRequest, TypeRequest (a KeyTarget: a ref, or an application and the window its keys go to), ApplicationInputResult (with the window they went to); InspectRequest, InspectResult; ReadTableRequest, ReadTableResult, AgentTableDefinition, AgentColumnDefinition (a column by place, by a locator in the row or by its header, with value); FindRowRequest, FindRowResult (findRow); SortTableRequest, SortTableResult (sortTable); ReadListRequest, ReadListResult (readList), ReadTreeRequest, ReadTreeResult, TreeNodeRead (readTree); FindItemRequest, FindItemResult (findItem); CandidatesRequest, CandidatesResult (candidates); WaitForRequest, WaitForResult; ScreenshotRequest, ScreenshotResult; ScreenRequest (screen, a whole display); MouseRequest, MouseResult (mouse, a pen stroke through at most MAX_STROKE_POINTS points); ClipboardRequest, ClipboardResult; RunCommandRequest, RunCommandResult (runCommand, at most MAX_RUN_ARGS arguments, each stream cut at MAX_RUN_OUTPUT_CHARS characters); AutomationRequest (automation: an application's own automation interface); SupportCaptureRequest, SupportCaptureResult (supportCapture: a support bundle of an application); BatchRequest, BatchStep, BatchResult (batch, at most MAX_BATCH_ACTIONS steps); CheckRequest, CheckResult (expect: a batch's check, as a method of its own); ConfirmRequest, ConfirmResult (confirm: what waits for confirmation, at most MAX_PENDING_CONFIRMATIONS at once, each for CONFIRMATION_TTL_MS); ExportTestRequest, ExportTestResult, AgentRecording; AgentLogEntry (one line of the action log). The session runs its calls one at a time, in order: session.run({ signal }, work) opens a call with a signal that cancels it, before it starts or while the engine waits (the MCP server does this for every tool call); pending and busy say what is in hand; more than MAX_PENDING_CALLS waiting calls are refused as busy; session.log keeps its latest AGENT_LOG_LIMIT entries. A snapshot or inspection returns DEFAULT_MAX_CONTROLS controls unless asked for more, and at most MAX_CONTROLS (the complete view too); a table page returns DEFAULT_TABLE_ROWS rows unless asked, and at most MAX_TABLE_ROWS.
The MCP server	createSpartDeskMcpServer, SpartDeskMcpServerOptions (its version defaults to this package's); TOOL_NAMES (always registered) and the gated ATTACH_TOOL, SHELL_TOOL, MOUSE_TOOL, CLIPBOARD_TOOL, SCREEN_TOOL, RUN_TOOL, AUTOMATION_TOOL, SUPPORT_TOOL, CONFIRM_TOOL (registered for confirmation: "pending").
Policy	AgentPolicy, PolicyEnforcer, AgentActionRequest (what a confirm or beforeAction hook sees), SensitiveAction, DEFAULT_RESTRICTED_KEYS (always refused), isRestrictedKey, normalizeChord, isAllowedApplication. Redaction: RedactionSetting (what redact takes), SecretRedactor (session.redactor: text(), value(), enabled), SECRET_PATTERNS (the shapes always looked for), REDACTED (what replaces them).
Snapshots	renderSnapshot and its RenderOptions (the compact text a model reads), describeAgentControl, locatorCode.
Untrusted content	fence, unfence, UNTRUSTED_CONTENT_NOTICE: the markers and the notice around everything an application put on screen.
Errors	AgentError with a stable code (AgentErrorCode: policy:<violation>, stale-reference, unknown-application, invalid-arguments, busy, row-not-referenced, no-snapshot), PolicyViolationError and its PolicyViolationCode, ConfirmationRequiredError (policy:confirmation-required: the action waits under its id), StaleReferenceError, UnknownApplicationError.

Test export goes through the engine's own renderTest and defaultTitle (@spartaficial/spart-desk), which render an AgentRecording as it renders any Recording.

Like the engine, the package ships an ES module and a CommonJS build, so require("@spartaficial/spart-desk-agent") works as well. The two executables are ES modules only.