Windows backend: the helper

SPART-Desk drives Windows applications through its own helper executable, spart-desk-host.exe, which ships inside the npm package. This page is for anyone who tunes the Windows backend or reads its errors: what the helper is, what it needs from the machine, how it treats windows of other processes, Store apps and Java windows, the options WindowsDriver takes and the error codes it reports.

The helper reads every window through Windows UI Automation, acts through the patterns and the windows' own messages, posts keys to the control with the keyboard focus, sends real mouse and keyboard input, and takes pictures. The driver over it is WindowsDriver; the backend's name, as errors, action records and desktop.backendName give it, is windows (windows@sandbox inside the Windows Sandbox guest (experimental), where the same helper runs, runtime-windows-sandbox.md).

Nothing to install. The helper is SPART-Desk's own executable, built from SPART-Desk's own sources: no third-party automation binary, no server to start and no port to open. It runs on the .NET Framework that every supported Windows has, so nothing is installed beside the package.

What the helper is

One small process on the .NET Framework that every Windows 10 and 11 installation has. The runtime keeps one helper running for its whole life and asks it over its standard input and output: which windows belong to an application (and which are the shell's, and which is in the foreground), each window's UI Automation tree, a control's states, ancestors, focus and value, pattern actions, the rows of Windows Forms grids and list views, Store apps' frames, and what an application announces. What goes to one window rather than through UI Automation — keys and text posted to the control with the focus, messages to a Win32 or Windows Forms control's own window (a menu, a combo box's choice, a list's selection, a date), real input, a window's scale and its picture — runs as a short-lived helper process per request. Inside the Windows Sandbox guest (experimental) the same helper runs as an agent that runs these jobs there.

A helper that stops on its own (ended by an antivirus, or by an application's UI Automation provider that makes it fail) is started again at the next request, at once while someone listens for announcements, and subscribed again. A request it was answering is asked again of the new process when sending it twice changes nothing (a read, a focus, a page turn); an action fails as delivery_unknown and is not sent again. After three restarts within five minutes, the fourth stop ends it for the desktop (helper_lost). Each restart is a helper entry in the trace.

The helper also serves the rest of the engine: the process table, the session (whether it is locked, remote or not interactive), the recognizer built into Windows, the Java Access Bridge, starting programs and Store apps, and Studio's picker and recorder. A helper of another build than the engine's is refused before its first request (helper_protocol).

Prerequisites

• Windows 10 1809 (build 17763) or later, Windows 11, or Windows Server 2025 with the Desktop Experience (x64 or arm64); Node.js 24+. On a server reached over Remote Desktop, keep the session drawn (ci.md).
• The helper runs: npx spart-desk doctor starts it once and says whether it does, and whether it is signed; it also reports the helper's path and the session as it reads it. Nothing else is installed (distribution.md). An antivirus may quarantine the helper, and an application control policy (WDAC, AppLocker, Smart App Control) may block an unsigned build; releases ship it signed, to allow by publisher (windows-security-gates.md).
• For operations through UI Automation and window messages (inspect, invoke, set-value, reading a value, posted keys, pictures) the target window does not need to be in the foreground.
• For real input (click with the mouse, press with modifier chords and the Windows key, dblclick, right-click, hover, the wheel) the session must be interactive and unlocked, and the window in front; SPART-Desk brings it there (input-engine.md).
• The application under test must expose UI Automation (WPF, Windows Forms, Win32, WinUI 3, Electron, Qt, Java through the bridge and UWP apps do); what the tree lacks is read off the screen (resolution.md).

Windows of other processes, Store apps, Java windows

• The helper tells which windows belong to an application without being its process's own (an out-of-process dialog, a credential prompt, a window of a process it started), and which of the shell's surfaces are showing; app.windows() lists them all, and desktop.shell() is observed window by window.
• Calculator, Settings and other UWP applications draw into a CoreWindow that ApplicationFrameHost.exe holds inside a frame of its own. The helper tells which application each frame holds, so app.windows() reports the frame under the application's own process (processName: "CalculatorApp"), close() stops that process and never ApplicationFrameHost, and the frame's title bar and the CoreWindow's controls are read as one window. A minimized Store app's frame is empty; restore() brings it back as its taskbar button does. Keys to XAML controls go as real input, since XAML ignores many posted keys. desktop.launch({ appId }) starts a Store app the way the Start menu does.
• Swing and AWT windows show UI Automation only their frame. The local runtime wraps this backend so that what the Java Access Bridge reports inside such a window is grafted under it, with source: "java" (java-apps.md). A driver passed to createLocalRuntime({ driver }) is used as it is.

Options

import { Desktop, createLocalRuntime, WindowsDriver } from "@spartaficial/spart-desk";

const desktop = await Desktop.create({
  runtime: createLocalRuntime({
    driver: new WindowsDriver({
      inspectDepth: 12,          // default 20: how deep the tree the engine matches goes; the read is not shorter for it
      keyboard: "send-input",    // default "auto"
      captureMode: "screen",     // default "window"
    }),
  }),
});

WindowsDriverOptions:

Option	Default	Meaning
execution	"host"	Where the backend acts: this machine's session, or the Windows Sandbox guest ("sandbox", experimental, which createSandboxRuntime sets). A driver is bound to one target for its whole life, so host and guest state can never be mixed.
inspectDepth	20	How deep the tree the engine matches goes. The helper reads a window whole in one call whatever the depth; the depth cuts the tree after the read, so it bounds what is matched and reported, not what the read costs, and the observation says when the cut was reached.
keyboard	"auto"	Transport for press: "auto" posts what can be posted (a character, a named key, a lock or function key, a chord of Ctrl, Shift and Alt with any of those or a letter or digit) and uses real input ("send-input") for the rest: a chord with the Windows key, a key that depends on the layout (punctuation), a control that takes a chord only as real input, Store apps hosted by ApplicationFrameHost, whose XAML controls ignore posted keys, and keys into a WebView2's page, whose field is first given the focus by a click past its text; "send-input" sends every key and chord as real input.
captureMode	"window"	How pictures for the screen readers are taken: "window" has the window paint itself, which works when it is covered; "screen" copies the window's screen rectangle as it is (overlays included, but also anything covering the window).

The other members of WindowsDriverOptions are internal: the runtimes use them to hand the driver the helper they keep running.

Performance envelope

A question to the helper costs a request to a process that is already running, not a process start; what goes to one window (posted keys, real input, a picture) starts a short-lived helper process per request. Each window is read whole in one call, so an observation costs about what the application's windows hold, whatever inspectDepth is. Consequences:

• inspectDepth bounds what is matched and reported, not what the read costs. A control named by its id (an agent's reference, a snapshot's control) is read alone.
• A whole observation is what an attempt costs; the auto-waiting loop samples at that rate plus the current entry of intervals. Raise timeout for slow applications rather than shortening the intervals.
• Real input (input.* strategies) needs the target window in the foreground; UI Automation patterns and posted keys work on background windows. Every way that works in the background runs first; an action with none left has its window brought to the front and runs once more. While real input is sent, the helper reports what else happened on the desktop (a person's own input, another window coming to the front), which the action's detail says.
• The helper answers an application's questions on one worker, in order; concurrent locator operations on one application queue there and gain nothing (parallelism.md).

Errors

Failures surface as BackendError with backend: "windows" (or "windows@sandbox", experimental), a backend-neutral code and a retryable flag; the codes the engine acts on come from the helper's answers:

code	Meaning	retryable
application_not_found	No process or window matched the target (the application may still be starting).	yes
element_not_found	The control is no longer there (the UI changed between resolve and act).	yes
not_supported	The control lacks the pattern for the action, or cannot take the keyboard focus an action needs: its provider says so, and one try gave it none.	no
foreground_required	Real input was refused because the window is not in front.	yes
focus_not_moved	Another control kept the keyboard focus, so no key was sent.	yes
delivery_unknown	A call that delivers something was stopped before it answered, or its helper stopped while sending it; it is never sent again.	no
command_timeout	A one-shot request to the helper did not finish in time.	yes
invalid_output	The helper answered a shape the engine does not expect.	yes
helper_unavailable, helper_protocol	The helper is missing, blocked, or from another build; the first action fails at once.	no
helper_exited	A question stopped the helper twice: asked, then asked again of the restarted helper; it is not asked a third time.	no
helper_lost	The helper stopped a fourth time within five minutes and is not started again for this desktop.	no
cancelled	The AbortSignal fired.	no

The complete list, with the session and occlusion codes, is under BackendError in api.md.