Native input engine

Some controls expose no UI Automation pattern for the thing you want to do: a custom-drawn canvas without InvokePattern, a rich-text box that rejects ValuePattern, a list that only scrolls with the wheel. SPART-Desk keeps the locator API unchanged and lets the backend choose: UIA pattern first, real input as fallback. Which one ran is recorded for every action.

Strategy order

DesktopLocator action	Preferred (UIA)	Fallback (real input)
click()	uia.invoke — Invoke / Toggle / SelectionItem / ExpandCollapse pattern; needs no foreground window	input.mouse.click on the control (or at the centre of its bounds for a top-level window)
dblclick(), click({ button: "right" })	—	input.mouse.dblclick / input.mouse.click
fill(value)	uia.set-value — ValuePattern / RangeValuePattern / LegacyIAccessible	input.keyboard.type — focus, Ctrl+A, type the text (Delete for an empty value)
press(key)	window.key — a named key (End, PageDown, Enter, arrows, F-keys) posted to the control's own window, without moving the focus (Win32, Windows Forms, Delphi); otherwise the control focused through UI Automation by the helper and the key posted (WPF); both in the background	input.keyboard.press — ui focus (which brings the window to the front) and real input, for modifier chords and Store apps
pressSequentially(text)	—	uia.focus, then input.keyboard.type — the text verbatim, nothing cleared first
focus()	uia.focus — SetFocus; needs no foreground window	a click at the centre of the bounds (vision controls only)
check() / uncheck(), expand() / collapse()	the click() strategies above, at most once, only when the observed state differs
hover()	—	input.mouse.hover
tap()	—	input.touch.gesture: a finger at the control, from a synthetic touch device
scroll({ direction | to })	uia.scroll — ScrollPattern	input.mouse.wheel
paging a virtualised list or grid (an action's locator scoping into it)	uia.scroll-page — one page with the scroll pattern, through the helper, in the background; the mouse wheel in the foreground	PageDown on the last item shown, for a container without a scroll pattern
scroll({ wheel: n }) (whole notches from -1000 to 1000 other than 0; positive scrolls down, as mouse.wheel's deltaY)	—	input.mouse.wheel

The fallback is taken only when the backend reports the pattern as unsupported (BackendError.code === "not_supported"). Any other failure (ambiguous selector, application gone, foreground refused) is thrown as-is.

Every action becomes an ActionRecord in desktop.actions:

const main = app.window();
await main.getByText("Canvas").click();
console.log(desktop.actions.records.at(-1));
// { action: "click", strategy: "input.mouse.click", fallbackFrom: "uia.invoke",
//   locator: 'getByText("Canvas")', control: { id: "…", role: "custom", … },
//   application: { pid: 4212 }, startedAt: …, durationMs: 180, ok: true }
desktop.actions.subscribe((record) => trace.push(record)); // every record as it happens, to a sink of your own

Strategies: uia.invoke, uia.set-value, uia.scroll, uia.scroll-page, uia.focus, window.key, input.mouse.click, input.mouse.dblclick, input.mouse.drag, input.mouse.hover, input.mouse.wheel, input.keyboard.type, input.keyboard.press, input.touch.gesture, input.pen.gesture. Failed actions are recorded with ok: false, strategy: "unknown" and the error message. The log is not coupled to any reporter.

Explicit primitives

When no control can be addressed at all, a window takes real input. A point is in the window's physical pixels, from its top-left corner where Windows draws it now: a pixel of window.screenshot(). A place in a control is better given to the control: locator.click({ position }).

const main = app.window();
await main.getByRole("custom", { name: "Canvas" }).click({ position: { x: 40, y: 25 } }); // a place in a control
await main.mouse.click({ x: 140, y: 85 });                         // press/release at a point in the window
await main.mouse.click(point, { button: "right" });
await main.mouse.click(point, { clickCount: 2 });                 // both clicks at once
await main.mouse.drag({ x: 100, y: 100 }, { x: 300, y: 120 });     // press, moves, release
await main.mouse.drag(from, to, { hold: 500, steps: 20 });         // held half a second first
await main.mouse.move(point);                                      // the pointer arrives (hover menus)
await main.mouse.down(point); await main.mouse.up(other);           // a gesture of your own
await main.mouse.wheel(point, { deltaY: 3 });                      // three notches down
await main.keyboard.type("literal text");                          // no key-name interpretation
await main.keyboard.press("Control+Shift+S");
await main.getByRole("listitem", { name: "report.pdf" }).dragTo(main.getByRole("list", { name: "Archive" }));
await app.clipboard.writeText("Ada"); await app.clipboard.readText();
const copied = await app.clipboard.waitForText(() => main.getByRole("button", { name: "Copy" }).click());

They are recorded as mouse.click, mouse.drag, mouse.move, mouse.down, mouse.up, mouse.wheel, keyboard.type, keyboard.press, dragTo, clipboard.read, clipboard.write and clipboard.waitForText. Moves, presses, releases, wheel turns and drags go through SPART-Desk's helper executable as one request each: a drag's press, hold, moves and release keep their timing. A request that fails half way (a later point on no display, input Windows refused, the helper stopped) releases the buttons it pressed, so no button stays down for a retry or the user's own hand; a request is given the time its pauses take on top of the usual 15 seconds.

Real input is paced as a hand gives it, since toolkits listen for that:

• A double click is two clicks a moment apart (60 ms, well within Windows' double-click time), each sent only while the window is still in front at the point: Qt and Flutter take no double click from two sent at once.
• Keys held for a click (modifiers) go down, rest 30 ms, and come up after it, so that the application reads them held when the button arrives. Keys held through a drag, a wheel turn or a move do the same around the whole gesture, inside the one request to the helper. A request given up on (cancelled, timed out, the helper gone) is followed by a release of the keys Windows still says are down, as of the buttons. Alt held through a wheel turn or a move is released after an unassigned key (0xE8): a lone Alt let go of after mouse input without a button opens the window's menu, which then takes the next key.
• A move places the pointer by absolute coordinates over the whole virtual desktop, then checks it landed there.
• A hover comes onto the control, a little to the side within it, and back, resting between; a drag rests after it crosses Windows' drag threshold, so that an application's drag-and-drop loop can start, and rests on the target before it lets go.
• A wheel turns notch by notch, a moment apart, rather than in one turn of several notches.
• A pointer at rest where the desktop's own last click left it is moved a pixel aside and back, once, when the application looks busy by its cursor alone: a window that appeared under it (WinUI 3's context menu) shows a stale wait cursor until the pointer moves.

The primitives live on a window rather than on the application or the desktop because input always goes to a window: the keyboard brings its window to the front first, unless it or a dialog it owns is there already (broughtToFront on the record says when), and the mouse's points are the window's. With foreground: "require", nothing is brought forward. The clipboard is the session's, on the application.

Coordinate model

Bounds are physical pixels of the application's screen, the space UI Automation reports them in and the helper's real input takes: locator.boundingBox(), a snapshot's bounds. A window's mouse takes points in the window, which it adds to where Windows draws the window now. One component, CoordinateMapper, does every conversion:

Conversion	Method
control bounds → centre / offset	centerOf(bounds), within(bounds, offset)
window-relative ↔ screen	fromWindow(window, p), toWindow(window, p)
logical (96-dpi DIPs) ↔ physical	fromLogical(window, p), toLogical(window, p) using the window's scale (WindowInfo.scale, WindowInfo.dpi from inspection)
plain { x, y } → tagged ScreenPoint	screen(p)

Points are tagged with a CoordinateSpace: host-screen or sandbox-screen (the Windows Sandbox runtime, experimental). A driver bound to the sandbox refuses host points and vice versa (SpartDeskError), so bounds observed in one runtime can never be clicked in the other. Multi-monitor layouts need no extra handling: physical screen coordinates span the whole virtual desktop, negative values included.

The foreground requirement

Windows only lets a process inject input into the foreground window, and the helper checks it before sending: on a background window every real-input request fails with foreground_required, before anything is delivered.

Every way that works in the background runs first: UI Automation patterns, messages posted to the control's own window, the Java Access Bridge. Only when an action has none of those left does it need the foreground. Then the desktop brings the window to the front and runs the action once more:

• the window is the one the control lies in (a Store app's frame), the one under the point for window.mouse, or the window itself for window.keyboard; a minimized window is restored first;
• the record says so: broughtToFront: true, and the detail names the window;
• when Windows keeps the window behind, the action fails with BackendError code: "foreground_required", saying that it tried.

Windows lets a background process bring a window forward while nobody has used the foreground window since it came forward, and lets the process that sent the last input do it. The helper first presses and releases a key no keyboard has, which makes it that process without any application hearing a key. Where that is not enough, it shares the foreground thread's input for the moment, unless the window in front does not answer: Windows would hold the call until it did. Then it clicks the window's title bar. Bringing a window forward takes the keyboard from whoever is working in the window that was in front: keys typed at that moment reach the application under test. On a desktop someone is using, create the desktop with foreground: "require": such an action then fails with foreground_required and leaves every window where it is.

Consequences:

• UIA strategies (click() on invokable controls, fill() on value controls, textContent(), inspection, screenshots) work on background windows and are always preferred.
• Plain-key press() is posted to the control's own window, or sent after the helper gave the control the focus through UI Automation, and works in the background; modifier chords need real input and therefore the foreground. A WPF window may bring itself forward when focused or paged through UI Automation, where Windows lets it.
• Real input needs an interactive, unlocked session: a dedicated CI session, the Windows Sandbox client (experimental), or an operator's desktop. On a locked session nothing comes to the front (session_locked). In the sandbox the hint names the connected client window as the requirement.

A click is the click asked for

Two things Windows does with real mouse input could make a click something else:

• Two clicks become a double click. Windows takes a second press of a button as a double click when it comes within the double-click time (500 ms by default) and inside a small rectangle (4 × 4 pixels by default) around the first, so two mouse.click calls at one point in quick succession would open a list's item. A real press that would join the last one this desktop sent (a click, a drag's start, mouse.down) waits out the rest of that time first, and the action's detail says how long. A double click asked for (dblclick(), clickCount: 2) is unchanged, and the press after one is not held back.
• Swapped buttons. With the left-handed setting, real input names the physical buttons, and Windows hands the application the left button for the physical right one. The helper reads the setting with each real click and gesture, sends the other physical button, and the detail says so. The application hears the button asked for.

The helper reads Windows' double-click settings with each real click.

Keys held down before the input

A modifier key that is already down when SPART-Desk sends its input would join it. A person's hand, a key stuck after a window of a higher integrity level took its release, or another program can hold it. With Control held, "dog" typed would reach the application as Ctrl+D, Ctrl+O and Ctrl+G, and a click as a Ctrl+click.

Real input.

• The helper releases every modifier key that is down and that the request does not ask for (Control, Shift, Alt, the Windows keys, left or right).
• The releases go at the head of the same batch as the request's keys, an Alt or a Windows key behind a key no keyboard has, so that no menu opens.
• The keys are not pressed again afterwards.
• The action's detail names them: "released Control, held down before the input".

Posted keys. They are read without the held keys too:

• The window in front has the keys released as for real input.
• Another window has them cleared from its thread's key state.

Caps Lock. With Caps Lock on, real typing turns it off for the keys and on again after them, so the text arrives as written. Under Windows' setting "press the Shift key to turn Caps Lock off", Shift turns it off.

Touch and pen

A window taps as a finger does and draws as a pen does: window.touchscreen and window.pen, beside window.mouse, and locator.tap() for a finger on a control. Windows makes the touch screen or the pen for the call (a synthetic pointer device, Windows 10 1809 and later): no digitizer, driver or elevation is needed, and the application hears the pointer messages a real one sends, then the mouse messages Windows makes from them.

const main = app.window();
await main.getByRole("button", { name: "Open" }).tap();                       // a finger at the control's middle
await main.touchscreen.tap({ x: 120, y: 80 }, { hold: 800 });                 // a long press
await main.touchscreen.tap({ x: 120, y: 80 }, { count: 2 });                  // a double tap
await main.pen.stroke([{ x: 40, y: 100, pressure: 0.2 }, { x: 360, y: 100, pressure: 0.9 }], { tiltX: 30, twist: 90 });
await main.pen.tap({ x: 200, y: 150 }, { barrel: true });                     // the barrel button held
await main.pen.stroke(points, { eraser: true });                              // the pen turned over
await main.pen.hover([{ x: 60, y: 60 }, { x: 100, y: 60 }]);                  // in range, not touching

• A gesture is sent whole. The engine turns the call into frames, one every 16 ms, each saying where the finger or the pen is, whether it is down, and its pressure, size, tilt and twist; the helper sends them in one request, with their timing. The finger lifts and the pen leaves by the last frame, so nothing stays pressed for the next call or the user's hand; a gesture that fails half way lifts what is down, canceled. A gesture lasts 15 s at most.
• What can be said. A finger has a pressure from 0 to 1 (0.5 by default) and a size in pixels (8 by default). A pen has a pressure (0.5 by default, or each point's own), a tiltX and tiltY from -90 to 90 degrees, a twist from 0 to 359, its barrel button and its eraser; it comes into range before it touches and leaves after it lifts, as a hand's pen does. Anything out of range is a RangeError before anything is sent.
• Points are the window's, as for window.mouse. locator.tap() waits for the control as click() does, taps its middle or position, and checks that the control is what lies there.
• Real input. The window is brought to the front for the gesture (foreground_required when it cannot be), a locked session refuses it (session_locked), and a person's hand on the mouse or keyboard meanwhile is noted on the record, as for real mouse input.
• No swipes. A finger does not move while down, and there is one finger at a time. Windows pans a swiped list with its own inertia, so a swipe would scroll it a different distance each time, and three or more fingers go to Windows 11's own touch gestures. Scroll with scroll(), which the list does the same way each time.
• What a toolkit makes of them. A finger is touch to WPF, WinUI 3, Avalonia, Qt Widgets, Flutter, JavaFX and web pages, and the mouse to Swing, SWT, Tk and wxWidgets; a pen keeps its pressure and tilt in WPF, WinUI 3, Avalonia and web pages, its pressure in Flutter, neither in Qt Widgets, and is touch to JavaFX and the mouse to the rest. Each takes a tap as its click. Where a toolkit takes a finger or a pen as the mouse, a locator action or window.mouse gives the application the same click; prefer them.
• The mouse after a tap. A WPF window takes no mouse button after a finger's or a pen's tap until the mouse has left it and come back. Before the first real mouse input after a gesture, the pointer goes onto the touched window's title bar (else just outside it) for a moment, and the record says so.

Limitations

• mouse.click(point, { clickCount: 2 }) is one real double click through the helper executable, both clicks within the double-click time.
• button is "left", "right", "middle", "back" or "forward" (the side buttons, XBUTTON1 and XBUTTON2). The middle and side buttons are always real input: no pattern means them, and Windows' own controls take only the left and right buttons posted. Only left and right are swapped for a left-handed mouse.
• Touch and pen are whole gestures only (see Touch and pen): nothing stays pressed between calls, so there is no down and up of a finger or a pen of your own. A finger taps; there are no swipes, flicks, pinches or several fingers at once.
• Typing into Windows' own edits posts characters, not key presses: it is independent of the keyboard layout, and dead keys never compose. A field that listens only for key presses, and so shows nothing afterwards, is typed into again as the keys that make each character in its own layout; a character pressed in the background (press("5")) is pressed as its key (desktop-conditions.md).
• Real input on a locked or disconnected session is refused before it is sent (session_locked): Windows would deliver it to no application.
• Moves reach a raw-input reader as absolute positions.
  • Every move is an absolute one (MOUSEEVENTF_ABSOLUTE over the virtual desktop), and a click puts the cursor there first (SetCursorPos, which raw input never sees).
  • A view that reads raw mouse input (WM_INPUT) gets each move as a position with the absolute flag, and no relative delta.
  • So a view that turns or pans by relative deltas, as a game engine's camera does, hears no movement. One that works from positions, as Windows' own messages give them, hears all of it.
  • Relative moves are not sent: Windows' pointer speed and acceleration would scale them, so ordinary applications would see the cursor somewhere else for a moment, mid-gesture.
• The clipboard carries text and pictures (app.clipboard): a picture is read as PNG (an application's own PNG kept whole, else its bitmap encoded) and written as a PNG and a 32-bit bitmap with its alpha. Bitmaps of fewer than 24 bits per pixel, or compressed, are not read. Files and other formats are left to the application's own copy and paste.