Running SPART-Desk in CI

SPART-Desk automates real Windows applications, so a CI job needs a Windows machine with the package installed and — for real input — an interactive desktop. This page lists what each kind of job needs and how to provision it, online or offline.

Runner requirements

Requirement	Why
Windows 10 1809 or later, Windows 11, or Windows Server 2025 with the Desktop Experience (x64 or arm64)	UI Automation and the helper are Windows-only. core-tier tests (fake backend) run anywhere.
Node.js 24+	Engine requirement (ESM).
The helper runs	spart-desk-host.exe ships inside the package, signed in releases; nothing is installed beside it (distribution.md). npx spart-desk doctor starts it once and says whether it runs and whether it is signed; an application control policy (WDAC, AppLocker, Smart App Control) must allow it, by publisher (windows-security-gates.md).
An interactive session for real input	Mouse/keyboard injection needs an unlocked, logged-on desktop in the foreground; UIA pattern actions do not. See below.
Optional: a grounding provider (experimental)	Only for getByVision; SPART-Desk ships none, so it is whatever you configure. Reading text off the screen and matching a picture need nothing installed.
Optional: Windows Sandbox (experimental)	runtime: "sandbox" needs Windows 11 24H2 or later, the Windows Sandbox feature and wsb.exe; see runtime-windows-sandbox.md.

Gate every job on npx spart-desk doctor: it exits non-zero when the tier is core and prints why. --json gives a machine-readable report.

Local Windows execution

The simplest setup is a self-hosted runner on a Windows VM:

1. Install Node 24 and check out your project.
2. Install its packages as usual (npm ci); the helper comes with @spartaficial/spart-desk. npx spart-desk doctor checks that it runs on this runner.
3. Run the runner agent as an interactive process, not as a service, when jobs need real input (see the next section). Use auto-logon for the runner account and disable the lock screen / screen saver via policy. A screen saver that starts during a run fails real input with session_locked, the message saying the session is behind a running screen saver.
4. Put the suites that drive real applications in projects of their own (parallelism.md), chosen by the command that runs them; keep unit tests (fake backend) unconditional.

Hosted runners (GitHub windows-latest, Azure Pipelines Windows images) run jobs in a service session without an interactive desktop: UIA-only operations and screenshots work, real input does not (session_non_interactive where SPART-Desk's helper can read the session, foreground_required otherwise; npx spart-desk doctor names the session). Use them for core and UIA-only suites; use a self-hosted VM for input.

Windows Sandbox execution

Experimental

The Windows Sandbox runtime may change in minor versions until its prerequisites are validated against a live Windows Sandbox.

Desktop.create({ runtime: "sandbox" }) runs the application inside a disposable Windows Sandbox guest with the same locators and actions. In CI:

• the host needs the Windows Sandbox optional feature, virtualisation enabled in the VM (nested virtualisation for cloud VMs) and wsb.exe on PATH;
• the host session must be interactive: the sandbox is a window on the host's desktop, so real input inside the guest has the same foreground requirement as on the host;
• Windows Sandbox must have been opened once in that user's session, which installs the app that brings wsb.exe;
• the helper runs inside the guest as the engine's agent. A guest enforces Smart App Control, which the signed helper passes; for an application the policy refuses, createSandboxRuntime({ guestAppControl: "off" }) turns it off there for that boot;
• the application's folder is copied into the guest, so the application must bring what it runs on (a Java runtime beside it, as a jpackage image has); a shortcut, a batch file or a ClickOnce reference names programs of the host and is refused. A packaged application starts by its app ID once it is registered in the guest; the Windows Sandbox image has no Store apps;
• SandboxUnavailableError (wsb-missing, start-failed, not-ready) tells you which prerequisite is missing, and SPART-Desk never falls back to the host session.

Interactive desktop requirements for real input

Windows only delivers injected input to the foreground window of an interactive, unlocked session. That affects:

Needs an interactive desktop	Works in a service/background session
input.mouse.* strategies: click() without an Invoke pattern, dblclick(), right-click, hover(), scroll({ wheel }), app.window().mouse.*	uia.invoke (click() on buttons, check boxes, menu items with a pattern)
input.keyboard.*: press() with modifier chords, fill() on controls without a Value pattern, app.window().keyboard.*	uia.set-value (fill() on text boxes), press() with plain keys via posted messages
Actions on controls found by getByVision (experimental): they are real clicks at coordinates	textContent(), isVisible(), isEnabled(), waitFor(), expectDesk(...), snapshot(), controls(), screenshot()

SPART-Desk brings the target's window to the front when an action needs real input, and reports an injection Windows still refuses as BackendError with code: "foreground_required" and a hint. Where the session itself takes no real input, the code says why: session_locked, session_disconnected (a Remote Desktop session whose client disconnected) or session_non_interactive (session 0, or a window station no display shows, as a scheduled task run whether the user is logged on or not has). Every ActionRecord.strategy tells you which category an action used, so a suite can be split into "UIA-only" (hosted runners) and "input" (interactive VM) jobs from the action log. Running those jobs in parallel, and why the input job gets one worker per interactive session, is covered in parallelism.md.

Practical settings for an interactive runner VM: auto-logon, no lock screen (Interactive logon: Machine inactivity limit = 0, screen saver off), a fixed resolution and 100 % DPI (or let CoordinateMapper handle scaling — it does, but consistent screenshots are easier to compare), and no RDP session left disconnected (a disconnected RDP session has no visible desktop; use tscon to hand the session back to the console, or run the runner from the console).

A runner reached over Remote Desktop

A Remote Desktop session is drawn only while its window on the client is shown. Once the client's window is minimised, Windows stops drawing the session: every picture taken there is blank, and anything that reads the screen or matches a picture finds nothing. In a Remote Desktop session, the blank picture's note says so. On the machine that runs the Remote Desktop client (not the runner), keep the session drawn while minimised:

# the client machine; 2 keeps a minimised session drawn
$client = "HKCU:\Software\Microsoft\Terminal Server Client"
# The key exists only once the Remote Desktop client has run.
if (-not (Test-Path $client)) { New-Item $client | Out-Null }
New-ItemProperty -Path $client -Name RemoteDesktop_SuppressWhenMinimized -PropertyType DWord -Value 2 -Force

The same value under HKLM:\Software\Microsoft\Terminal Server Client applies to every user of the client machine. A runner nobody watches is better started at the console, or handed back to it with tscon before the client disconnects.

Keep an unattended runner on one virtual desktop as well: a window left on another one is cloaked, and UI Automation shows nothing of it. A locator that finds nothing there says where the window is.

A Windows Server runner

On a Windows Server 2025 machine (Desktop Experience) that people also manage over Remote Desktop, run the jobs in the console session and keep Remote Desktop for the people:

• A dedicated runner account, not an administrator, signed in at the console by auto-logon, with Server Manager at logon, the lock screen and the idle timeouts turned off. An administrator's processes run elevated on Server, and an application started elevated is not what a user runs.

• The agent started at that account's logon, as a process in its session: GitHub Actions' run.cmd rather than its service, Azure Pipelines' agent configured with --runAsAutoLogon, or a scheduled task At log on with Run only when user is logged on. As a service, it runs in session 0, where real input fails with session_non_interactive.

• Nobody signs in as the runner account over Remote Desktop. Windows Server gives a user one session, so that sign-in takes the console session over, and closing the client then leaves it disconnected: every job fails with session_disconnected until someone reconnects. Administrators sign in with their own accounts. Windows Server allows two Remote Desktop sessions for administration beside the console without licences, and each session has its own desktop and foreground, so it does not disturb the runner's.

• If you did sign in as the runner account, hand its session back to the console before you leave, rather than closing the client. In that session, elevated:

  tscon ((Get-Process -Id $PID).SessionId) /dest:console

  The client disconnects, and the session goes on at the console, unlocked.

• One job at a time on that account: the jobs share its desktop, mouse and keyboard. More at once need a session each (parallelism.md).

Reading the screen on a runner

• Reading text needs nothing installed: Windows recognizes text in its own screenshots. A runner needs no model, no download and no cache entry for it, and doctor reports the text capability wherever the backend works.
• It costs one screenshot and the read, and only when the accessibility tree found nothing — so a suite that resolves through the tree barely pays for it. Reading is throttled to resolution.screen.interval (1 s) while a locator waits.
• A hardened runner can refuse it: an application control policy (WDAC, AppLocker) that does not allow the signed spart-desk-host.exe blocks the reader. doctor reports it, the failure is reported as it happens, and ocr: { provider } accepts a reader of your own. PowerShell policies (Constrained Language Mode, execution policy) do not matter: nothing runs through PowerShell.
• Matching a picture (getByImage) is the engine's own correlation, again with nothing installed. Keep the template folder in the repository beside the tests, with its typed index (index.mjs, index.d.mts). npx spart-desk images --check exits 1 when the index is not the one the pictures make, so a picture added by hand without npx spart-desk images fails the build rather than a test.
• Grounding (getByVision, experimental) has no bundled provider, so a runner only needs what your own provider needs.

Golden pictures and trees on a runner

toHaveScreenshot and toMatchTreeSnapshot write a golden file that is missing, except where the CI environment variable is set: there a missing golden file fails the assertion at once, so a run never passes on a file it just wrote. Write them on a developer machine (SPART_DESK_UPDATE_SNAPSHOTS=missing, the default there) and commit them beside the tests; SPART_DESK_UPDATE_SNAPSHOTS=all writes every one anew after an intended change. A runner whose display scale differs from the machine that wrote them keeps a golden picture per scale (<name>-125.png), which must be written there once.

Offline setup

Air-gapped runners install the npm package and nothing else: from the registry mirror the rest of the project's dependencies come from (Artifactory, Nexus, Verdaccio), or from a tarball (npm pack @spartaficial/spart-desk writes one on a connected machine). There is no artifact to download, no bundle to transfer and no mirror or proxy setting of SPART-Desk's own; pin the package through your lockfile as usual, and run npx spart-desk doctor once on the runner (distribution.md).

Example pipelines

GitHub Actions, self-hosted interactive Windows runner:

jobs:
  desktop:
    runs-on: [self-hosted, windows, interactive]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 24 }
      - run: npm ci
      - run: npx spart-desk doctor             # the helper runs, the session is interactive
      - run: npm test                          # fake backend, any machine
      - run: npm run test:integration          # real applications: your integration config, its own projects
      - if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: spart-desk-diagnostics
          path: ${{ runner.temp }}/spart-desk/**

Azure Pipelines, offline runner:

pool: { name: windows-desktop-interactive }
steps:
  - task: NodeTool@0
    inputs: { versionSpec: "24.x" }
  - script: npm ci --offline
  - script: npx spart-desk doctor --json
  - script: npm test && npm run test:integration
  - task: PublishBuildArtifacts@1
    condition: failed()
    inputs: { pathToPublish: $(Agent.TempDirectory)/spart-desk, artifactName: spart-desk-diagnostics }

Failure evidence (screenshots, UI trees) is written to diagnostics.directory (default %TEMP%\spart-desk); pipe desktop.trace to a file sink to keep the full operation trace with the job's artifacts (assertions-diagnostics.md). With diagnostics: { traceFile: "test-results/trace.jsonl", steps: "on-failure" }, a failed job also leaves the pictures of the steps that led up to its failure beside the trace, and npx spart-desk report test-results/trace.jsonl turns them into one HTML file to publish with the artifacts. Where those artifacts leave the machine, diagnostics: { redact: true } writes them redacted as a support bundle is: the data texts tokenised, their places painted over in every picture.

With diagnostics: { trace: "retain-on-failure" }, a failed job leaves a trace.zip for each failed trace under test-results/spart-desk/. Each one holds every step with its pictures before and after, its tree and its waits. The page that opens them, trace-viewer.html, is written at the top of that folder, so uploading test-results/spart-desk/ carries it. Open it and choose the folder it is in to list every trace.