Skip to main content

SPART-Desk

SPART-Desk is a Windows desktop automation engine for TypeScript: lazy locators, auto-waiting actions and retrying assertions against native Windows applications. This page shows what it does and where to read on; getting-started.md takes you through a first script.

import { Desktop } from "@spartaficial/spart-desk";

const desktop = await Desktop.create();
const app = await desktop.launch({ executable: "C:\\Windows\\System32\\charmap.exe" });
const main = app.window();

await main.getByLabel("Characters to copy").fill("Hello world");
await main.getByRole("checkbox", { name: "Advanced view" }).click();
console.log(await main.getByLabel("Characters to copy").textContent());

await desktop.dispose();

The Windows backend is SPART-Desk's own helper executable, which ships inside the package and speaks Windows UI Automation directly. Nothing is installed beside the package; npx spart-desk doctor reports whether the helper runs and what the machine can do.

Core ideas​

  • Locators are lazy. app.window().getByRole("button", { name: "Save" }) builds an immutable description and never talks to the backend. Only operations such as click(), textContent() or isVisible() resolve it, and every operation resolves anew: nothing is cached inside a locator.
  • Strict by default. Operations that need one control fail with a StrictModeViolationError that lists every candidate when more than one control matches.
  • Every action waits. An action resolves its locator, waits until the control exists, is unique, visible, enabled and stable, then acts. No sleeps are needed, and a failure lists what was observed.
  • The accessibility tree first. A locator reads the UI Automation tree unless one of its steps names the screen, so a control the tree cannot see is reached where the script says so: by the text painted on it, by a picture of it, or by a grounding provider you supply.
  • Backend-agnostic core. Desktop, Application, Window and DesktopLocator depend only on the DesktopDriver and DesktopRuntime interfaces, so your own automation code can be tested without Windows.

Locators and actions​

const main = app.window();
const dialog = main.getByRole("dialog", { name: "Customer" });

await dialog.getByLabel("First name").fill("Alex");
await dialog.getByAutomationId("chkActive").click();
await dialog.getByRole("button", { name: "Save", exact: true }).click();
await dialog.waitFor({ state: "detached" });

await main.locator('role=menuitem[name="File"] >> text=Exit').click();

Every locator starts from a window; app.window() is the main one with the dialogs it owns. getByRole, getByText, getByLabel, getByAutomationId, getByTitle and locator chain and filter, and first(), last() and nth() pick among several matches. See locators.md and actions.md; the full reference is api.md.

UIA first, real input as fallback​

const main = app.window();
await main.getByRole("checkbox", { name: "Advanced view" }).click(); // uia.invoke: no foreground needed
await main.getByRole("pane", { name: "Character Grid" }).click();    // input.mouse.click: no InvokePattern
await main.getByLabel("Notes").fill("text");                          // uia.set-value, or focus + Ctrl+A + typing

await main.mouse.drag({ x: 110, y: 90 }, { x: 300, y: 90 });             // points in the window: its screenshot's pixels
await main.keyboard.press("Control+S");

console.log(desktop.actions.records.map((r) => `${r.action}: ${r.strategy}`));

An action prefers a UI Automation pattern, which works on a window in the background, and falls back to real mouse, keyboard, touch or pen input when the control offers none. Real input needs the target window in the foreground: when no way in the background is left, SPART-Desk brings the window to the front for it (foreground: "require" turns that off) and reports foreground_required when Windows keeps the window behind. Details in input-engine.md.

Auto-waiting​

const main = app.window();
const desktop = await Desktop.create({ timeout: 10_000, intervals: [100, 250, 500, 1000] });

// Resolves, waits until the button exists, is unique, visible, enabled and
// stable, then clicks. No sleeps required.
await main.getByRole("button", { name: "Save" }).click();

// Per-call overrides and cancellation.
const controller = new AbortController();
await main.getByRole("dialog", { name: "Done" }).waitFor({ timeout: 30_000, signal: controller.signal });

Failures explain themselves:

TimeoutError: Timeout 5000ms exceeded waiting for getByRole("button", { name: "Save" }) to be visible, enabled, stable.
Last observed state: disabled
  1) button "Save" [automationId=btnSave] (615,591 113x33) disabled
Application: pid=39144 process="charmap" · windows: "Character Map" (hwnd 3935002)
Backend: windows

What each operation waits for, and how timeouts, intervals and cancellation work, is in auto-waiting.md.

Assertions and diagnostics​

import { Desktop, expectDesk } from "@spartaficial/spart-desk";
const main = app.window();

await expectDesk(main.getByText("Completed")).toBeVisible();          // retries with the same loop actions use
await expectDesk(main.getByLabel("Name")).toHaveValue("Alex Example", { timeout: 10_000 });
await expectDesk(main.getByRole("button", { name: "Save" })).not.toBeEnabled();
const rows = await expectDesk.poll(() => countRows(app)).toBeGreaterThan(0); // poll and toPass return the value
// AssertionError: expected vs actual, timeout, last candidates, Resolver: windows/uia, screenshot + UI tree attached

desktop.trace.pipe({ write: (entry) => log.write(JSON.stringify(entry) + "\n") }); // operations, snapshots, screenshots, errors

expectDesk is independent of the test runner and retries until the assertion holds or its timeout passes; .not, poll and toPass hand back their value. Every operation is recorded with its resolver, backend, strategy and timings in desktop.actions, and desktop.trace is pluggable, with screenshots and UI trees attached to failures. Details in assertions.md and assertions-diagnostics.md.

Grids, lists and combo boxes​

const main = app.window();
const orders = main.getByRole("grid", { name: "Orders" });
const row = orders.getByRole("row").filter({ hasText: "Customer 120", exact: true });
await row.getByRole("cell", { column: "Email" }).click();     // pages a virtualised grid to the row first
await expectDesk(row.getByRole("cell", { column: "Total" })).toHaveText("840.50");
await main.getByLabel("Colour").selectOption("Green");          // opens, selects, closes
await orders.getByRole("row").last().scrollIntoView();

WPF and Windows Forms grids get the same roles (row, cell, columnheader); a cell is found by the header of its column. Actions bring their control on screen: rows off screen are scrolled into view, and rows a virtualised WPF list has not created yet are paged to. Rows are also found by what their cells hold, and whole tables are read and asserted on; see tables.md and dotnet-apps.md.

When the accessibility tree is not enough​

const main = app.window();
await main.getByRole("button", { name: "Save" }).click();   // the accessibility tree
await main.getByOcr("Save").click();                         // the text painted on screen
await main.read().getByLabel("Name").fill("Ada");            // a field named by the label painted beside it
await main.getByImage("save.png").click();                   // wherever this picture appears
await main.getByRole("textbox").rightOf(main.getByOcr("Customer number")).first().fill("42");
await main.getByVision("blue floppy disk save icon").click(); // experimental: a grounding provider you supply
const observed = await app.snapshot({ enrich: "text" });     // structure plus what is painted

A locator names what answers it, so getByText never takes a screenshot, getByOcr always reads one, and getByImage never queries the tree for the picture. Reading text needs nothing installed on Windows; matching a picture is the engine's own correlation. Strictness, auto-waiting and diagnostics are the same whichever answered, and every failure lists what ran. For applications the Windows text recognizer misreads, the optional @spartaficial/spart-desk-ocr-paddle reads with PaddleOCR. See resolution.md.

Configuration​

// spart-desk.config.ts: what `npx spart-desk init` writes; Desktop.create() reads it from the working directory
import { defineConfig } from "@spartaficial/spart-desk";

export default defineConfig({
  runtime: "local",
  timeout: 10_000,
  expect: { timeout: 15_000 },
  diagnostics: { directory: "test-results/spart-desk" },
});

Options passed to Desktop.create win over the file. Details in configuration.md.

Agents​

Human:  await app.window().getByRole("button", { name: "Save" }).click();
Agent:  desktop_snapshot → c17 = button "Save" → desktop_click({ ref: "c17" })

The separate package @spartaficial/spart-desk-agent exposes the same engine to AI agents. npx spart-desk-mcp --runtime local --allow-app MyApp.exe gives Claude Code, Claude Desktop, VS Code agents and other MCP clients tools such as desktop_snapshot, desktop_click, desktop_fill, desktop_read_table, desktop_find_row, desktop_wait_for, desktop_screenshot and desktop_export_test, with semantic references that stay with their control, compact capped snapshots, fencing of untrusted UI content, an application allow-list, restricted key chords, confirmation hooks and a full action log. Agents that live in a shell use spart-desk-cli from the same package (launch charmap.exe, snapshot app1, click c17) with the same engine and policy. See agentic-mcp.md.

Tools​

npx spart-desk studio opens the Inspector, which shows the controls of a running application with the locator that finds each one, and the Recorder, which turns your clicks and keystrokes into a script. npx spart-desk doctor reports what the machine can do, and npx spart-desk support capture writes a redacted support bundle. See studio.md.

Testing your own automation code​

import { Desktop } from "@spartaficial/spart-desk";
import { FakeDriver, FakeRuntime } from "@spartaficial/spart-desk/testing";

const driver = new FakeDriver([{ role: "window", name: "Demo", children: [{ role: "button", name: "OK" }] }]);
const desktop = await Desktop.create({ runtime: new FakeRuntime(driver) });
const app = await desktop.launch({ executable: "demo.exe" });
const main = app.window();
await main.getByRole("button", { name: "OK" }).click();
console.log(driver.performed); // [{ action: { type: "click", ... }, controlId: "0.0" }]

The /testing entry point runs your page objects and helpers on any operating system, without Windows and without an application.

Requirements​

  • Windows 10 1809 (build 17763) or later, Windows 11, or Windows Server 2025 with the Desktop Experience, for running automation. The package installs and type-checks anywhere.
  • Node.js 24 or newer, ESM only.
  • Nothing else. The Windows backend is SPART-Desk's own helper executable, spart-desk-host.exe, signed by Spartaficial, which ships inside the package: it reads every window through UI Automation, sends real input, takes pictures, reads the process table, screen text and Java windows, and never needs PowerShell. backend-windows.md lists its prerequisites, such as an unlocked desktop for real mouse and keyboard input. Reading text off the screen uses the recognizer built into Windows and matching a picture is the engine's own correlation, so neither adds a download, a model or a cache entry (distribution.md).
  • The Windows Sandbox runtime (experimental) needs Windows 11 24H2; Windows Server has none. See runtime-windows-sandbox.md.

Installation​

npm install @spartaficial/spart-desk
npx spart-desk init               # spart-desk.config.ts and an example test (asks; --yes for defaults)
npx spart-desk doctor             # platform, the helper, Java: what this machine can do, capability by capability

Air-gapped machines install the same npm package from a registry mirror or a tarball (npm pack @spartaficial/spart-desk writes one); there is no artifact to download, cache or bundle. On machines with application control (WDAC, AppLocker, Smart App Control), allow the signed spart-desk-host.exe by publisher; doctor says whether it runs and whether it is signed. What each of Windows' gates does with it is in windows-security-gates.md; everything else in distribution.md and, for pipelines, ci.md.

The package ships both an ES module and a CommonJS build, so import and require() work alike, including in test projects that compile their files to CommonJS. Load it through one of the two in any one process: each build is its own copy of the engine, and instanceof does not hold across them.

Where to go next​