Agentic automation: MCP and command line

@spartaficial/spart-desk-agent (package README) exposes the SPART-Desk engine to AI agents in two ways: the spart-desk-cli command line for agents that live in a shell, and the spart-desk-mcp server for clients that speak the Model Context Protocol. It is an optional, separate package: the engine does not depend on it, and it depends only on the engine's public API — never on the helper executable, a screen provider or native input.

spart-desk-cli <command>                 Claude Code / Claude Desktop / VS Code agent / any MCP client
  (shell agents; one process per command)        │  MCP (stdio)
        │  MCP (loopback HTTP, bearer token)      │
        ▼                                         ▼
spart-desk-mcp --transport http            spart-desk-mcp
  session host: one AgentSession, started     one process per client
  on demand, stopped on `stop` or when idle
        └──────────────────┬──────────────────────┘
                           ▼
@spartaficial/spart-desk-agent
   createSpartDeskMcpServer  — tool definitions (zod schemas) → AgentSession
   AgentSession              — apps, snapshots, refs, policy, action log
        │  public engine API only
        ▼
@spartaficial/spart-desk
   Desktop → Application → DesktopLocator → ResolverPipeline (the resolver the chain names) → DesktopDriver (strategies) → trace/actions

Same engine for humans and agents

Human	Agent
await app.window().getByRole("button", { name: "Save" }).click()	desktop_snapshot → c17 = button "Save" → desktop_click({ ref: "c17" })

A ref is turned back into a locator (app.window().locator({ type: "controlId", id })), so the click above runs the exact same auto-waiting, strictness, UIA-first strategy selection and diagnostics as the human call. The engine's ActionRecord (resolvedBy, backend, strategy, timings) is attached to the agent log entry.

Command line transport

spart-desk-cli is the same agent layer for agents that live in a shell (Claude Code, Copilot CLI, Codex, Cursor, …): no client configuration, the commands are discovered through --help or the agent skill the package ships (spart-desk-cli skill --install writes it into the project's skill folders), results can be filtered before they enter the context window, and commands compose with && and pipes.

A command line cannot be stateless — launched applications, refs, the Windows Sandbox session (experimental) and the screen reader must outlive one process — so every command is a short-lived MCP client of a session host: spart-desk-mcp --transport http --session <name>, the MCP server serving Streamable HTTP on 127.0.0.1 with a random port. Once Desktop.create succeeded and it listens, the host writes { pid, port, token, runtime, startedAt } to %TEMP%\spart-desk-cli\<name>.json (SPART_DESK_CLI_HOME relocates it); every request must carry the token, and non-loopback Host headers are refused. The CLI starts the host on demand (detached, stderr in <name>.log next to the session file), start starts it with a policy (the spart-desk-mcp flags), stop asks it over HTTP to close its applications and exit — Windows delivers no signals to a detached process — and the host exits by itself after --idle-timeout minutes (default 60). -s <name> / SPART_DESK_CLI_SESSION keep several sessions apart.

Commands map one-to-one onto the tools (launch → desktop_launch, snapshot app1 → desktop_snapshot, click c17 → desktop_click, press app1 Control+S → desktop_press into the window in front (else the main window; --window <title> names one), type app1 "Hi!" → desktop_press with text, export app1 → desktop_export_test, …), and a tool's arguments onto flags (wait --ocr <text> → ocr, set --bounds <x>,<y>,<width>,<height> → bounds, _ or nothing keeping a part, inspect --details → details, inspect --properties → all, since --all already means what it means for snapshot); attach, shell, mouse, clipboard, screenshot --screen and run exist only on a host started with --allow-attach / --allow-shell / --allow-mouse / --allow-clipboard / --allow-screen / --allow-run <program>. Fenced results are printed verbatim, notice and markers included; actions print one line; tool errors go to stderr with their name and code and exit 1. The policy and the action log are the host's, so the transport changes nothing about what an agent may do. The one thing the command line adds is a project-local output folder, .spart-desk-cli/: every snapshot saves the complete view there (every control with its ref and locator, fenced like stdout) and screenshot defaults into it, so an agent greps and diffs files instead of reading whole trees; the folder ignores itself in git. The host, a detached process with its own working directory, never writes into the project. The package README has the command table.

Tools

desktop_list_applications, desktop_launch, desktop_close, desktop_snapshot, desktop_click, desktop_fill, desktop_set, desktop_press, desktop_inspect, desktop_read_table, desktop_find_row, desktop_sort_table, desktop_read_list, desktop_find_item, desktop_read_tree, desktop_candidates, desktop_scroll, desktop_hover, desktop_drag, desktop_wait_for, desktop_screenshot, desktop_export_test, desktop_batch, and — only when enabled — desktop_attach, desktop_shell, desktop_mouse, desktop_clipboard, desktop_screen, desktop_run and desktop_confirm. Each new ability widens a tool rather than adding one where it can, since every tool's definition is paid for on every turn: desktop_mouse takes an action (move, down, up, wheel, drag) beside its click, at a point of a window in the pixels of its desktop_screenshot; desktop_wait_for, desktop_press, desktop_screenshot and desktop_mouse take a window (a title) beside the main window or the one in front they use otherwise; desktop_wait_for takes rows beside its states and ocr (text read off the screen, getByOcr) beside text (what the accessibility tree reports; an ocr or vision (experimental) target holds no rows and is visible or hidden, never attached or detached), and desktop_scroll takes to: "view" to bring the control itself into view. desktop_set sets one state of a control, and does nothing when it already holds: checked, expanded, option (what a combo box or list shows), window (minimize, maximize, restore) or bounds (where a window's frame is to be). An exported test then says what was meant (check(), expand(), selectOption("Green"), minimize(), setBounds(…)), where a click on a toggle would toggle it whatever its state. desktop_shell opens the Windows shell (a display's taskbar, tray, notification centre, Start and Search while open) as an application with an id, snapshotted and acted on like any other; display picks another display's taskbar or "all", and desktop_close of it stops nothing. No tool duplicates another: waiting and assertions-as-waits go through desktop_wait_for (a table's row count included), reading a control and looking inside it through desktop_inspect (with details: true, also what the control says of itself: help text, item status, keys, password, required, read-only, selected, range, MSAA; with all: true, which implies details, every property and pattern its provider says it has, by UI Automation's names), a grid's rows as data through desktop_read_table (its columns once, then one line per row, a page at a time with offset/limit, or the whole of a virtualised grid with scroll, read only as far as the page asked for; a later page of the same whole read is served from it until another tool is called, for 30 s at most), one row by its cells through desktop_find_row (where: { column: value }, a whole text, a pattern written "/^Jan/i" or a number, waiting and paging, strict, strict: "all-pages" for every page, returning references to the row and what it holds), a grid sorted by a column through desktop_sort_table (its header clicked as a user does, authorised as a click), a list's items through desktop_read_list and one item by its text through desktop_find_item (as for a grid's rows), a tree's items, indented by level, open or closed and selected, through desktop_read_tree (expand: true opens closed items to maxDepth, as a user does, authorised as desktop_set with expanded), the locators that could address a control through desktop_candidates (code, class, matches, resolvers, reasons, and the recommended one), and desktop_export_test touches no UI at all — it renders what the session recorded. desktop_batch runs up to 20 steps on one application in one call (click, fill, set, press, type, scroll, hover, each with the fields its own tool takes, and expect: a check of a state, a value or text it contains, which ends the batch with code: "assertion" when it does not hold; click, fill and press take until, the same check, and are sent again, up to sends times, while it fails), then returns one snapshot of what changed. Every step is checked before the first runs: its reference, its arguments, its restricted keys, the sandbox policy (experimental) and its confirmation. A refusal of any step runs nothing; the first step that fails while running ends the batch, and the result, an error then, says which step and why. Each step is still a call of its own in the log and in the recorded flow. Every tool declares all four MCP annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint).

Tables the agent was told about, and trees

A table the tree does not report as a grid is defined in the locator string grammar, for the session (AgentSession({ tables })) or one snapshot (desktop_snapshot { tables }): root, rows, cells, header, pageDown, and columns, as names by place or as columns found by index, locator or header, each with an optional value (checked for a column of check boxes). Once a snapshot laid such a table out, desktop_read_table and desktop_find_row on its root's reference read and search it by its definition.

A tree item by its path is written in the grammar too: role=tree[name=Folders] >> role=treeitem[name="Customers"][outermost=true] >> role=treeitem[name="Active"][outermost=true]. The candidates of a tree item offer such a path, and an exported test writes it as app.window().tree(…).node(…).

Snapshot and reference model

desktop_snapshot returns the application's controls (interactive ones, windows and whatever only the screen saw by default; interactiveOnly: false adds text and structure from the tree) in one normalized shape whatever observed them, rendered as one indented line per control by default (format: "json" for the object form):

{ "ref": "c17", "role": "button", "name": "Save", "bounds": {...}, "source": "uia", "locator": "getByRole(\"button\", { name: \"Save\" })" }
{ "ref": "c18", "role": "text", "name": "Total: 42.00", "bounds": {...}, "source": "ocr", "locator": "getByOcr(\"Total: 42.00\")" }
{ "ref": "c19", "role": "button", "name": "Open", "source": "uia", "sources": ["uia", "ocr"], "locator": "read().getByRole(\"button\", { name: \"Open\" })" }

A control only the screen saw, or one the tree reports but reading named, gets a locator that says so (getByOcr, read()): what the exported test runs is what found the control.

• A grid is its columns and one line per row. A grid or table shows its column headers (columns="Order | Customer | Date"), and each row the row's cells in column order (c42 row "1042 | Acme Ltd | 2026-01-12"), in place of a line per cell; the row keeps its reference. A text inside a control that says what the control is called (a WPF button's caption, a cell's value) is left out. The JSON form carries cells and columns.
• A reference belongs to the control, not to the snapshot. The engine derives a key from what a control is rather than from how the backend addresses it, so the same button keeps the same reference however many times the application is re-observed. That is what makes a snapshot worth diffing: an agent can be shown only what changed, because everything it already knows still resolves.
• A reference goes stale when its control is no longer observed. It is remembered rather than forgotten, so the refusal names what it used to be (c18 (button "Export") is gone) instead of only saying it is old.
• Every action re-resolves twice before giving up. The control id is how the backend addressed the control when it was observed, and it stops being valid the moment the UI is rebuilt — a dialog closed and reopened, a list redrawn. The suggested locator describes the control by what it is, so it finds the new one. Trying the id first keeps the common case a single cheap call; falling back to the locator is what makes a reference survive a redraw instead of forcing a fresh snapshot.
• A control that is merely not ready yet is waited for like any locator.
• A control whose name changes and that carries no AutomationId is a new control as far as the key is concerned, and gets a new reference. With an AutomationId the identity is the id the developer set, and a rename is reported as a change.
• Every control carries locator: the stable locator the engine's recommendLocators derives for it (a field's label, role + name, exact when ambiguous, AutomationId, an ancestor scope or the row that a text tells from its siblings, nth last). locators: true renders it inline; it is what an exported test uses, and the second way an action finds the control when its id has gone stale.
• Closing an application forgets its refs.

Token economy

Narrow a snapshot to what you are looking for with query (and role): only the controls whose name, value, automation id or cells hold the text, best first, at most limit (20), each with its ancestors for context: a narrowed snapshot costs a fraction of a whole one. Every control keeps its reference, whatever the narrowed view shows. A batch's expect steps check what the steps did in the same call, where a batch and a desktop_wait_for took two.

A snapshot is capped at maxControls (default 200). Over the cap, the deepest controls are dropped first so windows and top-level controls always survive, and the result reports truncated and omitted. maxDepth limits the depth up front and root (a reference) scopes the snapshot to one subtree, with depths relative to that root. When the screen is read, root limits the reading to that control's area. With the Windows backend it also limits the observation itself to that subtree: the control is read alone, inspectDepth levels below it, where an earlier observation saw it. The suggested locators are then the root's own locator followed by one that is unique inside it. Runs of siblings that say nothing a sibling of the same role would not are collapsed into a count, and coordinates are left out for controls the accessibility tree named, since the agent acts by reference. The compact text form costs a fraction of the JSON form.

Round trips cost more than any one result. Every call is a model turn that reads the whole context again: the tool definitions and everything said so far. A form of a few fields and a button filled one step at a time takes a call per action and a snapshot to see the outcome; as one desktop_batch it is one call carrying about the same characters: the saving is the turns, not the payload. Use it whenever one snapshot tells the agent the next steps.

The largest saving in a result is changes: true. Because a reference belongs to the control rather than to the snapshot, a second observation can show only what was added (+) or changed (~) and count the rest: everything omitted is still addressable, because the agent already holds its references. What is gone gets one line per part that went, so a closed dialog costs one line: - c2 window "Tip of the Day" (+4 inside). That reference is retired.

The cap shapes what is shown, not what can be addressed: every scoped control gets a ref, and full: true returns the complete view (text and structure included, each control with its locator) as a second fenced content — or as full in the JSON form — for clients that write it to a file. spart-desk-cli does exactly that on every snapshot (.spart-desk-cli/<app>-<snapshot>.txt; -q prints only the header and the file name), so a full inspection of a large application costs the context window one line plus whatever the agent greps.

Untrusted UI content

Text read from the UI (names, values, captions, window titles) is data the application produced. Results that contain it (desktop_snapshot, desktop_inspect, desktop_list_applications, the snapshot after a desktop_batch and its summary when a step failed, an error that says what would wait for confirmation) are prefixed with a notice and wrapped in <untrusted-ui-content> markers so an agent never mistakes on-screen text for instructions.

What a window shows can be a secret nobody meant to hand a model: a private key pasted into a text box, a token in a log viewer. Every text a tool returns, errors included, and every line of the action log has the shapes of well-known secrets replaced by [redacted] (a PEM private key block, an AWS access key id, a GitHub token, an sk- API key, a JSON Web Token, the token after Bearer), and a password field's value is never shown: it is replaced inside the fence, string by string in a JSON result, at the one place every result leaves the server (redact below).

Agents are expected to snapshot, act, and snapshot again after actions that change the screen. Raw coordinates are not part of the normal interface.

Safety controls

All controls live in AgentPolicy and are deterministic; no LLM provider is involved anywhere.

Control	Behaviour
allowedApplications	Executables an agent may launch (base name or path, * wildcards, RegExp). A launcher stub's processName must be allowed as well, since that is the process automated and terminated.
sandboxOnly (experimental)	Every action requires desktop.runtimeName === "sandbox".
restrictedKeys	Chords that desktop_press refuses, added to the ones it always refuses: Alt+F4, Control+Alt+Delete, Control+Shift+Escape, Meta+L, Meta+*. A named key matches exactly; a * blankets a modifier family, so Meta+* covers Meta+Shift+S too.
allowMouse	Registers desktop_mouse (--allow-mouse: clicks, moves, presses, wheel turns and drags at a point); off by default. Its use is logged with raw: true and its action.
allowClipboard	Registers desktop_clipboard (--allow-clipboard); off by default. A read hands back whatever the user copied, fenced as untrusted content; a write is logged by its length only. sensitiveActions: ["clipboard"] asks for confirmation first.
allowScreen	Registers desktop_screen (--allow-screen); off by default. It pictures a whole display, or every one (display), with every window on it and the taskbar, not only the session's applications; in the sandbox runtime, the guest's desktop. The request a hook sees is screenshot with operation: "screen"; sandboxOnly applies.
allowRun	Registers desktop_run (--allow-run <program>, repeatable) for the programs it names, matched as allowedApplications are; off by default. A command runs where the applications run (the guest, in the sandbox runtime), as their user; it is logged by its program and how many arguments it had, never the arguments, and its output comes back fenced as untrusted content, each stream cut at 20 000 characters. Running a program is running code: name only what the agent needs, prefer sandboxOnly, and sensitiveActions: ["run"] asks for confirmation first.
allowAttach	Registers desktop_attach; off by default. allowedApplications applies to the process name, and a target without one is refused while an allow-list is set. A windowTitlePattern (a regular expression, ignoring case) is taken only beside processName or pid, so a pattern never picks an application by its title alone, and is refused when longer than 200 characters or when it could take long to match.
allowShell	Registers desktop_shell (--allow-shell); off by default. The shell is no application an allow-list names, so allowing Explorer.EXE does not open it; sensitiveActions: ["shell"] asks for confirmation first. Restricted keys apply as everywhere.
sensitiveActions + confirm	Actions that must be confirmed by a callback before they run; without a callback they are denied. In a desktop_batch, confirm is asked about every sensitive step before the first step runs, so a denial runs nothing. Each names a family, and the request's operation says which gesture in it is asked for: click covers hover and drag by reference; mouse covers click, move, down, up, wheel and drag at coordinates (to holds a drag's end as "x,y"); clipboard covers read and write. A request that cannot be carried out (a wheel turn of 0 notches, a drag at coordinates without its end) is refused with invalid-arguments before confirm or beforeAction is asked.
confirmation: "pending"	(--confirm <family>, repeatable, which also names the families.) A sensitive action is refused with ConfirmationRequiredError (policy:confirmation-required), nothing run, and waits under its id for desktop_confirm, registered only then. That runs it once, exactly as it was asked, within two minutes; at most 16 wait at once, and closing the application ends their wait. Its references must still resolve, and the sandbox check and beforeAction run again. Configure your client to ask you before each desktop_confirm call: that makes it your confirmation. It is a speed bump the client enforces, not a gate this package enforces; the confirm callback stays the hard gate.
beforeAction	Hook for destructive-action policies: throw or return false to block. In a batch it runs before each step as that step runs, so it can stop a batch midway.
Action log	Every tool call with runtime identity, application, target, outcome, duration and the engine record; typed text logged by length only. --log <file> appends JSON lines.
redact	On by default. Every result's texts and every log entry have SECRET_PATTERNS replaced by [redacted]. A text box is asked once, before the first fill or typing into it, whether it is a password field (details().password); one that is shows [redacted] as its value and text from then on, its Value properties too, what went into it exports as a placeholder to supply from a secret store, and its text is never recorded as an assertion. { patterns } (--redact-pattern <re>, repeatable) adds the operator's own regular expressions, matched without regard to case and refused when they could take time without end, as the engine refuses a support bundle's (unsafePattern); false (--no-redact) turns it all off. Pictures (desktop_screenshot) are not painted over.

Errors are typed and carry codes: PolicyViolationError (policy:application-not-allowed, policy:sandbox-required, policy:restricted-key, policy:mouse-disabled, policy:attach-disabled, policy:shell-disabled, policy:clipboard-disabled, policy:screen-disabled, policy:run-disabled, policy:command-not-allowed, policy:automation-disabled, policy:automation-not-allowed, policy:support-disabled, policy:confirmation-denied, policy:blocked-by-hook), ConfirmationRequiredError (policy:confirmation-required, with its id, summary and expiresAt), StaleReferenceError (stale-reference), UnknownApplicationError (unknown-application), and AgentError: invalid-arguments for a request the tool cannot carry out as given, busy when too many calls already wait their turn, row-not-referenced when a row desktop_find_row found is in no snapshot the session can refer to, no-snapshot when a reference is in no whole snapshot of its application, and closed for a call made after the session was disposed. Engine errors (TimeoutError, StrictModeViolationError, BackendError, ScreenError, CapabilityUnavailableError, …) pass through unchanged and are returned to the client as tool errors with their name, code and full message (including resolver and attachments).

A window above the session's integrity level (an application run as administrator, Windows' credential prompt) comes back as BackendError elevation_required, and its snapshot says why it is empty. The agent host never elevates itself. To drive such windows, start the MCP client, or spart-desk-cli start, elevated, and keep the policy narrow (elevated.md).

Agentic testing philosophy

Agents are best at exploration, discovery and test generation: snapshot, try, observe, refine. The output of that work should be deterministic SPART-Desk scripts (getByRole(...).click(), expectDesk(...)) that run as the regression suite without an agent in the loop. The shared engine makes that hand-over lossless, and the session closes the loop: it records every successful action per application with the stable locator of its ref, and desktop_export_test / spart-desk-cli export render the recording as the test spart-desk init would have written — launch (or attach) in a before hook, the steps as locator code, every desktop_inspect as an expectDesk(...).toHaveText(...) assertion, dispose in an after hook. Failed, stale and refused actions are never recorded; a control without a stable locator becomes a // TODO line; the recording survives desktop_close and ends with the session. The full autonomous loop an agent runs (npx spart-desk doctor, npx spart-desk init --yes, launch, inspect, act, read, export, close, run the test) is in the agent skill the package ships, which spart-desk-cli skill --install writes into the project. A host started in a project folder also reads that project's spart-desk.config.ts (timeouts, reading, images, diagnostics; the runtime stays what --runtime says), so agent sessions and scripts share the same defaults (configuration.md).

What a session can do is never a guess for the agent, and never named after whatever provides it. desktop_list_applications returns the engine's capabilities map — uia, nativeInput, screenshot, text, imageMatch, elements, grounding (experimental) — each either available or carrying the reason it is not. Tool descriptions are written from that map, so enrich says what it would cost when reading is possible and why it is unavailable when it is not. An ability the engine does not have raises the engine's own CapabilityUnavailableError, which names the ability, the reason and what would enable it; the agent layer adds nothing by restating it. No provider name appears in any description, schema or error.