Actions

Every action resolves its locator, waits until the control is ready for that action, performs it through the best available mechanism and records what happened. Signatures: api.md; the waiting rules: auto-waiting.md.

DesktopLocator actions

Action	Waits for	Preferred mechanism	Fallback
click({ button?, mouse?, modifiers?, position?, timeout?, signal? })	visible, enabled, stable, its window not blocked by a modal dialog (window_blocked at the timeout) nor the application busy (app_busy); for a real click, the control under its point, where it was observed and still enabled, the instant before the click (occluded at the timeout; a control that moved is waited for as moving, one disabled meanwhile observed again; a position is checked at that place)	uia.invoke (Invoke / Toggle / SelectionItem / ExpandCollapse pattern); an invoke whose handler shows a modal dialog returns once the dialog is open. Through the control's own window instead, in the background (win32-apps.md): a push button of Windows' own (Win32, MFC, Windows Forms), whose BN_CLICKED is posted to its parent, so the click runs after the action returned (window.press); an item of a native list box or list view (window.select, since its invoke is a double-click); a Win32 menu bar's item, whose menu is read and opened without being shown, and that menu's items (window.menu); a SysLink's link (window.link); an up-down control's or NumericUpDown's arrow (window.spin); an item of a context menu the application shows (window.menu). A Windows Forms grid's cell is selected through the grid's MSAA object (msaa.select)	input.mouse.click at the control's centre; behind another window, a click posted to the control's window (window.click), for toolkits that take one (not WPF, Java, Chromium, XAML, nor a list view); a right click there first asks the control for its context menu through UI Automation, where its toolkit answers that: XAML, File Explorer's items, web pages (uia.context-menu). An item of the shell's items view (File Explorer, the Open and Save dialogs) is selected by a click and opened by a double click, through its patterns (uia.invoke). mouse: true asks for a real click at the control's middle whatever else the control offers. modifiers: ["Control"] ("Shift", "Alt"; at least one, none twice) holds those keys while a real click is sent, through the helper: down before the click, in the keyboard layout of the window under the point, and up after it whatever the click did, so nothing stays held (Ctrl-click adds to a selection, Shift-click extends it). position: { x, y } clicks at that place from the control's top-left corner, in whole physical pixels, as dragTo's sourcePosition; the control under that point is checked as it is at the middle: what the control holds or draws there passes, a sibling or a cover refuses the click. Either makes the click real input, needing the window in front: a list view's item and a Windows Forms grid's cell are then clicked with the real mouse rather than selected through their windows, which would replace the selection, and a click with keys is never posted to a window behind another, since a posted message holds no key
dblclick({ modifiers?, position? })	visible, enabled, stable; the control under the point as for a real click	—	input.mouse.dblclick (the two clicks a moment apart, well within Windows' double-click time, each sent only while the window is still in front at the point, as a hand clicks: Qt, Flutter and others take no double click from two sent at once); behind another window, a double click posted to the control's window (window.dblclick), as for click(). A list view's item is double-clicked on its first column's text, where the list hits it. modifiers and position as for click()
check(), uncheck(), setChecked(v)	visible, enabled, stable, with a toggle state (a radio button's checked state is read through its window, or from IsSelected)	nothing when already in that state; else exactly the Toggle pattern's on or off (uia.invoke, invoke --action toggle-on/toggle-off), then a wait for the new state. A WPF check box, toggle button or radio button is given Space with its focus instead, in the background, which runs its Click handler and bound command; an Avalonia one hears Checked alone through the pattern, and its record says its Click handler and bound command did not run; check({ mouse: true }) clicks as a user does	one click (see above)
expand(), collapse()	visible, enabled, stable, with an expansion state	the same, on combo boxes, tree items, menus: exactly the ExpandCollapse pattern (uia.invoke, invoke --action expand/collapse), in the background	a control without that pattern: a tree item gets the Right / Left arrow and an open combo box F4, posted to its window or after the helper's UI Automation focus, in the background; else once the helper confirmed the focus on it. A tree item that reports no expansion state but holds an expander (a toggle button, Avalonia's) is expanded by toggling it (invoke --action toggle-on/toggle-off), and its state read from it
selectOption(text, { exact? })	visible, enabled; a combo box, a list, or a drop-down field that is a button named on two lines, its label and what it shows (Flutter's: pressed once, the item's button its menu shows is clicked, and the choice is taken once the field reads it)	opens a combo box when the item is not listed, selects the item (uia.invoke; window.select in a native list), waits until the combo box shows it, closes it (window.collapse for a native drop-down that stays open); see dotnet-apps.md	uia.set-value with the item's text when selecting did not change the combo box (Windows Forms), or typing it (an editable combo box). A native combo box (Win32, MFC, Windows Forms, Delphi VCL) is then told of the choice (window.notify, CBN_SELCHANGE), which closes it and runs the application's handler (win32-apps.md, vcl-apps.md)
fill(value)	visible, enabled, editable (a read-only field is waited for)	uia.set-value (Value pattern). Typed into instead, character by character, through messages to its own window (window.type), in the background: Windows' own edits in Win32 and Windows Forms (an editable combo box's, an up-down control's buddy, a NumericUpDown's, a masked field), a Scintilla editor (Notepad++; line ends as the Enter key) and a Delphi data-aware field (TDBEdit). A date picker takes 2026-12-24 (window.date); a Windows Forms grid's cell is set through the grid's MSAA object (msaa.set-value). A web page's rich-text editor (an editable region holding paragraphs: Slate, Lexical, ProseMirror) is never given a value, which its own model would not hear: its text selected through its Text pattern and the new text posted after UI Automation focus (input.keyboard.type)	input.keyboard.type after focusing
clear()	visible, enabled, editable	fill("")
pressSequentially(text)	visible, enabled	Windows' own edits: the characters posted to the edit's window at its caret (window.type), in the background	uia.focus, then input.keyboard.type key by key at the caret, without clearing (press("End") first to append after a fill)
press(key)	visible, enabled	posted key messages for plain keys; a chord a window's own menu lists as a shortcut sends that item's command (window.menu); any other chord is posted to the control's window with its modifiers held in the window's own key state (window.key; Win32, Windows Forms), in the background	input.keyboard.press for a chord with the Windows key, or on a control without a window of its own (WPF, a web page). A chord with punctuation is pressed as the keys of the window's keyboard layout (Control+/ is Control+Shift+7 under German), and numpad Enter and / with their extended flag, through the helper
focus()	visible	window.focus: the focus moved through the window's own thread when the window is its thread's active window, which activates nothing; else uia.focus, the helper's UI Automation focus, which activates the window where Windows lets it (the detail says so); a control its provider reports not keyboard-focusable is refused after that one try, naming the control that can take it; refused at once on a locked, disconnected or non-interactive session (session_locked and the others)	the window brought to the front and the control focused there, confirmed by the focus it reports, when it took none in the background, as press() and type() do; a click at the centre (vision controls)
hover()	visible, stable; the control under the point as for a real click	—	input.mouse.hover
tap({ position?, count?, hold?, pressure?, size? })	visible, enabled, stable, its window not blocked by a modal dialog nor the application busy; the control under the first contact as for a real click	—	input.touch.gesture: a finger at the control's middle (or position), sent whole from a synthetic touch device and checked to land on the control; count: 2 double-taps, a hold makes a long press
scroll({ direction?, to?, wheel?, modifiers? }) (wheel: whole notches, positive down; modifiers held while it turns, only with wheel)	visible, stable	uia.scroll (Scroll pattern)	input.mouse.wheel
scrollIntoView()	attached, or not created yet in a virtualised list	nothing when visible; uia.scroll-into-view (ScrollItem pattern), checked afterwards; a list or grid the locator scopes into is paged to a row it has not created; a Windows Forms grid's cell is paged into view with the grid's scroll bar, selecting nothing (msaa.scroll-page)	uia.invoke on a cell, row or item whose container ignores the request (selecting it brings it on screen); an item its list holds and keeps out of view after that (Flutter's, Qt Quick's) is paged toward with the list's scroll pattern or the wheel, as an action's control is
close(), minimize(), maximize(), restore() on a window locator	attached (a minimized window is not visible)	the window's own title-bar button over UI Automation, found by the AutomationId Windows gives it in every language (Close, Minimize-Restore, Maximize-Restore), or, in a 32-bit process whose title bar reports none, by its place (Close last); works in the background	a title bar the toolkit draws, with no buttons UI Automation can press (Electron, Krypton, Material Design): Windows' own system command posted to the window, as its system menu sends it, and its state read back (window.system-command)
textContent(), boundingBox(), isEnabled(), waitFor()	attached (or the requested state)	observation only	—
isVisible(), count(), all(), allTextContents()	do not wait	observation only	—

check() and friends are idempotent: a control that already is in the wanted state is left alone (the record says observe, already checked), one that is not gets exactly one click and the wait continues until the backend reports the new state. A control that reports no such state fails with BackendError not_supported rather than being clicked blindly.

Key names: Enter, Tab, Escape, ArrowDown, F5, single characters, and chords joined with + such as Control+A or Shift+Alt+F.

A key goes to whichever control has the keyboard focus. So before press(), pressSequentially(), the keys of expand() and collapse(), or typing in fill(), SPART-Desk moves the focus to the control and checks that it arrived. The helper confirms that the control has it; when it cannot, SPART-Desk asks where the focus is (uia-focused), and the control itself or a part of it (a combo box's edit box) counts. Moving the focus can bring the control's window to the front. If another control keeps the focus, no key is sent. The attempt fails with the retryable BackendError focus_not_moved, which names that control, and the wait tries again. Windows Forms, for one, gives the focus back to a tab control whose tab was just selected, and Right arrow sent there would switch tabs.

Bringing the control on screen

An action (click, fill, press, check, selectOption, ...) whose control is there but off screen asks the backend once to scroll it into view (scrollIntoView, above), then waits for it to show. When a locator scopes into a list or grid (grid.getByRole("row").filter(...)) and finds nothing, the action pages that container (to its top, then a page per attempt: with the mouse wheel when the window is in the foreground, with the container's scroll pattern otherwise, both of which leave the selection alone; PageDown only for a container without a scroll pattern), because a virtualised WPF list has not created the rows it does not show. The view is told by where the rows are: a row outside the list's box is out of view whatever it reports (Qt Quick's ListView keeps the rows it created and reports them on screen), and a row still hidden after it was asked into view is paged toward, a step per attempt. Where the rows the container shows are in order by what the locator asks for (a cell's text in a named column, an item's name, the text of a filter, the option of selectOption), the view is set to where the row should be instead, as dragging the scroll bar there does, and adjusted by what the next look shows until the row is in view: a row thousands down takes a few looks rather than hundreds of pages. Before that, a list that can be asked for an item by its name (WPF's and WinUI's, through UI Automation's ItemContainer pattern) is asked once, and the item is realized into view in one call; a grid's row, asked for by its cells, is not. A container whose rows are not in order, or that takes no position, is paged as before, and the wait's state says why. Between pages or jumps, the wait looks at the container alone, not at the whole window. What it took is appended to the record's detail (paged grid "Orders" 6 times with the mouse wheel, paged grid "Orders" 1 time by its scroll position, asked list "Items" for the item by name through its ItemContainer pattern, scrolled into view). Observations and assertions never scroll or page: they report what is on screen. Details and limits: dotnet-apps.md.

Mouse and keyboard primitives

For controls UI Automation cannot address at all. A window's mouse takes points in the window, from its top-left corner (the pixels of window.screenshot()); a place in a control is better given to the control, click({ position }):

const main = app.window();
await main.getByRole("custom", { name: "Canvas" }).click({ position: { x: 20, y: 20 } });
await main.mouse.click({ x: 120, y: 80 });
await main.mouse.drag({ x: 120, y: 80 }, { x: 300, y: 80 });
await main.keyboard.type("hello");
await main.keyboard.press("Control+S");
await main.touchscreen.tap({ x: 120, y: 80 });                    // a finger's tap
await main.pen.stroke([{ x: 40, y: 100 }, { x: 200, y: 120 }], { pressure: 0.7, tiltX: 30 });

Touch and pen are sent a whole gesture at a time, the finger lifted by its end (input-engine.md).

Points are physical screen pixels of the application's runtime (host or sandbox); CoordinateMapper converts between window-relative, logical (DPI-scaled) and screen coordinates. Prefer locator actions: they resolve fresh and pick the strategy for you.

The foreground rule

Windows delivers injected input only to the foreground window of an interactive, unlocked session. UIA pattern strategies (uia.*), and the window.*, java.* and msaa.* ones, need no foreground window; input.* strategies do. Every background way is tried first. When an action has none left and its window is not in front, the desktop brings the window to the front and runs the action again; the record then has broughtToFront: true. When Windows keeps the window behind, or the desktop was created with foreground: "require", the action fails with BackendError code: "foreground_required" and a hint, rather than clicking the wrong place.

A real click on a control the tree reported also checks the control under its point, in the same request to the helper, just before the cursor moves. Something else of the window's there (a loading overlay, a panel, an in-window popup) refuses the click as occluded, waited out within the timeout. The control found there must also be where it was observed, within a few pixels, and enabled: one still sliding is refused as element_moved, which the wait reads as moving and looks past until two looks find it in one place; one disabled meanwhile as element_not_enabled, observed again. A drag's press and a tap's first contact are checked the same way. When the check cannot answer, the click goes with the window check alone, and its record says "the control under the point was not checked". A control read off the screen (checked against a fresh capture instead), an item a list's or grid's window reads, and a point the caller gives (app.window().mouse.*) are not checked. A modal dialog over the control's window refuses every action, through any strategy, as window_blocked.

// On a desktop someone is working at: never take the foreground.
const desktop = await Desktop.create({ foreground: "require" });

See input-engine.md and ci.md.

On a locked session, or a disconnected Remote Desktop session, no window comes to the front: such a failure is reported as BackendError code: "session_locked" instead, saying so. Everything that works in the background (uia.*, window.*, msaa.*) works on a locked session. desktop.session() tells a test which session it has.

Knowing what ran

desktop.actions.subscribe((r) => console.log(r.action, r.strategy, r.fallbackFrom, r.resolvedBy, r.attempts, r.ok));
console.log(desktop.actions.last); // the operation that finished most recently, failed ones included

Every ActionRecord names the operation, locator, control, resolver (resolvedBy: uia, java, ocr, image, grounding or region), backend, strategy (uia.invoke, input.mouse.click, observe, …), fallback origin, attempts, timings and outcome; the same records flow into desktop.trace. Use them to split suites into "UIA-only" and "needs input" jobs, or to see why something was slow. Splitting for parallel execution: parallelism.md.

Options every action accepts

• timeout — per-call budget, overriding Desktop.create({ timeout }).
• signal — an AbortSignal; aborting cancels the wait and any in-flight backend call with CancelledError.

Sending an action again until it did what it should

Some applications drop an action they were not ready for. A WPF window whose view model loads after the window shows has a Save button that is enabled but bound to no command yet, and a click on it does nothing. click, dblclick, fill and press take a postcondition for such cases:

const main = app.window();
const status = main.getByAutomationId("status");
await main.getByRole("button", { name: "Save" }).click({
  until: () => expectDesk(status).toHaveText(/^Saved/, { timeout: 1_500 }),
  sends: 5,
});

• until is a check that throws while what the action should have done has not happened: an assertion with a short timeout of its own, or a function that is given a signal ending with the action's timeout.
• The action is sent again only after the check failed (an assertion that did not hold, a wait that timed out). Up to sends times in all (3; from 1 to 5), within the action's one timeout. Each send is its own action record, saying it is send 2 of 3 and why.
• It is never sent again when its delivery is unknown, and an error of the check's own (not a failed assertion) ends it.
• Not offered where the action makes sure of a state itself: check, uncheck, expand and collapse act only when the state differs, and wait for the new one.

When the check never holds, an AssertionError (assertion: "until") says how many sends were made, with the last check's failure as its cause.

Application-level operations

app.windows() (each a Window you can act in), app.window({ title }), app.waitForWindow(action?), locator.bringToFront(), app.window().currentTitle() (the main window's title, waiting for a window), app.snapshot() (tree), app.controls({ vision? }) (flat, optionally merged with vision detections), app.window().screenshot({ path? }), app.close(); desktop.launch(), desktop.attach(), desktop.dispose().

minimize() leaves a minimized window alone. Whether a window is maximized is not something Windows reports in a language-neutral way, so maximize() and restore() press the Maximize/Restore button, compare the size the window took with the size it had, and press it once more if the window went the other way (it already was in the wanted state). restore() on a minimized window brings it back to where it was. A minimized Store app's frame shows no title bar at all; restore() then restores it as its taskbar button does, through the helper executable (window.restore).