Desktop conditions: keyboards, the clipboard, dialogs, hangs and displays

Some things that decide whether an action works are not in the control at all: the keyboard layout, a clipboard another process holds, a dialog that blocks the window, an application that stops answering or ends, a window that lies off the display. This page says what SPART-Desk does in each case and what is left to you.

Keyboard layouts, dead keys and characters beyond the keyboard

fill() and pressSequentially() on Windows' own edits post characters, not key presses. Win32, Windows Forms, MFC and Delphi edits take them in the background. So:

• any Unicode text arrives, emoji included;
• the keyboard layout does not matter: German, French and US-International alike;
• dead keys never compose: 'e stays 'e under US-International.

Fields that listen for key presses are typed into with keys. Some fields build their value from key presses and ignore characters, for example a PIN field or a hot-key box.

• When such a field shows nothing after a fill, SPART-Desk types the text again as the keys that make each character in the field's own keyboard layout, still in the background. The action record says so.
• press("5") on such a field presses the key that makes "5".
• A character that no key makes, or that a dead key makes, is still posted as itself.

Chords.

• A chord with a letter (Control+Z) is posted with its modifiers held, and works under any layout.
• A chord with punctuation (Control+/) is sent as real input, as the keys a person of the window's layout presses: Control+Shift+7 under German, Control+Shift+: under French.
• press() names the numeric keypad's keys (Numpad0–Numpad9, NumpadAdd, NumpadSubtract, NumpadMultiply, NumpadDivide, NumpadDecimal, NumpadEnter), ContextMenu, Pause, PrintScreen and F13–F24. Each carries the keyboard's extended flag where it has one, so numpad Enter is not read as Enter.

WPF, Qt and web content have no edit window of their own. fill() sets the value through UI Automation. pressSequentially() types with real input, in the foreground, under the layout of the window in front.

IME composition (Chinese, Japanese and Korean input methods) is outside what fill() and pressSequentially() do.

A left-handed mouse and the double-click time

With the mouse buttons swapped (Settings > Bluetooth & devices > Mouse > Primary mouse button: Right), real input names the physical buttons. The helper reads the setting with each real click and gesture and sends the other physical button, so a click() is still a left click to the application. Its record says so. Windows' double-click time and rectangle are read the same way, and two clicks asked for one after the other are kept apart by them (input-engine.md). Clicks posted to a control's window in the background are messages the application reads as they are, and neither setting changes them.

The clipboard

SPART-Desk never uses the clipboard by itself: it does not paste to type, so a clipboard that another process holds open cannot make an action fail. A test that needs the clipboard uses app.clipboard: readText() after the application copied something, writeText() before it pastes. Both need the helper executable, and in the Windows Sandbox runtime (experimental) they reach the guest's clipboard.

Copy and paste are the application's own, reached through their shortcuts:

const main = app.window();
await main.getByLabel("Source").press("Control+C");
await main.getByLabel("Target").press("Control+V");

Both are posted in the background on Windows' own edits.

If another process holds the clipboard at that moment (a clipboard manager, a remote desktop client), the application's paste does nothing, and nothing can tell. Assert on the result, not on the key press.

Password and read-only fields

Passwords.

• A password field is filled like any other.
• SPART-Desk never reads its value back: textContent() returns the mask, or nothing.
• The action log never quotes it.

Read-only fields.

• A read-only field's value is read with toHaveValue().
• A fill() on it fails with BackendError code read_only and changes nothing: a Windows edit with ES_READONLY, or a field whose Value pattern says it is read-only.

Pattern actions that differ from a click

check(), uncheck() and click() on a check box or toggle button toggle it through UI Automation, which works in the background. WPF and Avalonia then raise Checked, but not Click, and run no command bound to the control.

A WPF check box, toggle button or radio button is therefore given Space with its focus instead, in the background, as a user's key press: its Click handler and bound command run with the change. The pattern is used only when Space changed nothing.

Avalonia's toggles still go through the pattern. When the application reacts to the click or the command there, ask for a real click. It needs the window in front, which the desktop brings it to unless it was created with foreground: "require":

const main = app.window();
await main.getByRole("checkbox", { name: "Notify by e-mail" }).check({ mouse: true });
await main.getByRole("button", { name: "Bold" }).click({ mouse: true });

An expansion the application ignores is tried once more, another way. Qt, for one, ignores UI Automation's expand. When a tree item is still collapsed a second after expand(), SPART-Desk presses the arrow key on it. A check box is never toggled twice.

Dialogs

Windows' common dialogs work in the background:

• Open, with its file name typed;
• Save, with its overwrite question;
• the folder tree;
• the file dialog's folder mode;
• task dialogs, their command links (named by their first line) and check box;
• XP-style dialogs.

In an Open dialog, press Enter in "File name:" rather than clicking "Open". Three buttons carry that name (the two combo boxes' arrows among them), and AutomationId 1 is also the first file's list item.

A click whose handler shows a modal dialog returns while the dialog is open.

• A push button of Windows' own (Win32, MFC, Windows Forms) is pressed through its window (strategy window.press). The application runs the click after the action has returned, so the dialog it opens can be answered at once.
• A WPF button's click returns at once too.
• An action through UI Automation whose handler shows a dialog inside the call, such as a WPF list's SelectionChanged, holds the call open.
  • SPART-Desk sees the window blocked, returns, and names the dialog in the action record.
  • The application then answers no UI Automation request, the dialog's included, until the dialog closes. SPART-Desk reads it through the window manager meanwhile, and a snapshot says so in a note.
  • A message box, or any dialog of Win32 or Windows Forms windows, is read from its child windows and its buttons are pressed in the background.
  • A WPF dialog has no child windows: its buttons are read off its picture and clicked with real input, so it needs the foreground.

A window that a modal dialog blocks is not acted on. An action on a control of that window waits for the dialog to close. If it is still open when the timeout elapses, the action fails with window_blocked, naming the dialog: "the dialog "Order" is open over it". Answer the dialog first. This holds for a control found by a search, which names no window, even when the dialog lies over the control on the screen, and for every way SPART-Desk sends something to a control, posted keys and pressed buttons included.

A real click checks the control under its point. Something else of the application's over the control in its own window (a loading overlay, a docked panel) makes the click wait, and fail with occluded if it stays.

Starting, hanging and ending

Splash screens. launch() returns at the process's first window, even a splash screen. Locators wait for the main window, and a window that closes while it is being read is read again.

A title that changes once a document loads is followed by expectDesk.poll(() => app.window().currentTitle()).

Single-instance applications. A second launch that hands over to the running instance and exits is joined when target: { processName } names the application and the launch says joinExisting: true:

• attachedToExisting is true, also when the second process lingers a moment after handing over;
• close() stops only what the launch started.

Without joinExisting, such a launch fails with LaunchError naming the running instance: a test never acts on a window you had open because the application handed the launch over to it. Without target, the launch fails with "the process exited first".

Launchers and updaters that hand over and exit. target: { windowTitle } waits for a window of that title, and follows what the launcher starts and what those start, through an updater that exits too:

• the wait goes on after the launcher exits while anything it started runs, and fails at once when nothing does;
• close() stops the client the launch was handed over to, also one a broker (WMI, a service) started whose window appeared since the launch;
• a window of the title that was open before the launch is not taken for the launch's own: without joinExisting: true the launch fails, naming its process.

A client that another program starts only after the launcher has exited is named by its process, target: { processName, windowTitle }, whose wait goes on whatever the launcher does.

Closing. app.close() and desktop.dispose() ask the application's windows to close first, as the close button does, and stop its processes only when they are still there 3 s later. An application with unsaved changes may show its prompt for that long before it is stopped; close({ force: true }) stops it at once. A process id Windows gave to another process since the application was attached is never stopped.

Hung applications. When a window of the application takes no messages, SPART-Desk stops reading it, and the wait keeps to its timeout. If it still does not answer by then, the error says so: "the application is not responding: its window "…" takes no messages". An action waits within its timeout for the application to answer again.

Slow UI Automation providers. An application can answer window messages while its UI Automation provider takes seconds to answer: a control that fetches its caption, a peer that waits for data. SPART-Desk tells this apart from a hung application:

• A tree read that took more than 5 s and found nothing says so in the resolver's record: "the tree took 13 s to read while the application answered messages: its UI Automation provider is slow".
• A tree read that got no answer at all (the helper's 10 s limit) while the window answered is not asked again for 30 s. Locators that can read the screen do so at once, and the others say "not asked" in their record until the 30 s pass.
• A wait that fails after such a read explains itself from what it observed, without reading the tree again.

The first attempt of a wait always completes, so a slow provider can still take an action past its timeout. Scope the locator to a container, or name the control (a name or an AutomationId is searched for rather than read whole).

Trees cut short. An observation that could not see everything says so in its notes, and a locator that finds nothing repeats the note in its error: the tree reached the depth it was read to (inspectDepth, 20 by default), or the helper cut it at 20,000 controls or 30 levels.

Applications that ended. An action on an application whose processes no longer run fails at once with application_ended. A wait for its controls to go (waitFor({ state: "detached" })) is over.

Unhandled exceptions. An exception on a Windows Forms application's user interface thread shows the framework's dialog, which is found by its "Continue" button.

Locked sessions

Real input on a locked or disconnected session, or on one that is not interactive at all, reaches no application. SPART-Desk refuses it before it is sent rather than report a click that did nothing, with a code that says why:

Code	The session
session_locked	Is locked: Win+L, an idle timeout, a policy's inactivity limit, or a UAC prompt's secure desktop; or is behind a running screen saver.
session_disconnected	Is a Remote Desktop session whose client disconnected.
session_non_interactive	Takes no real input at all: session 0, where services run, or a window station no display shows (a scheduled task run whether the user is logged on or not). desktop.session() has interactive: false.

focus() is refused there too, at once: on such a session the calls that move the focus say it moved, and no control takes it.

Everything through UI Automation and the controls' own windows still works in all three. press() refuses Ctrl+Alt+Del and Win+L outright (reserved_key): the first is the secure attention sequence, which no injected input makes, and the second would lock the session.

The message, and desktop.session()'s reason, say which sign gave the lock away:

reason	What the helper saw
"lock-flag"	Windows reports the session locked.
"screen-saver"	A screen saver Windows started is running. It runs on a desktop of its own (Screen-saver), secure or not, while Windows' lock flag stays clear; real input would end it rather than reach the application, so the message says the session is behind a running screen saver, and the hint says to turn it off for the account the tests run as. A screen saver started by hand (scrnsave.scr /s) is only a window in front, which the foreground check sees as such.
"input-desktop"	Another desktop takes the input: Winlogon's lock screen, or a UAC prompt's secure desktop. A user's process cannot open it, and real input cannot reach the application either.
"lock-screen"	The lock screen's window (LockApp) is in front.

When a session reported locked takes your own input, the reason says which sign to look at; npx spart-desk doctor prints it too.

Displays and DPI

Bounds are in the display's physical pixels, whatever the application's DPI awareness:

• unaware;
• system;
• per-monitor;
• per-monitor v2.

A DPI-unaware window on a 150 % display is stretched by Windows, and its bounds say so.

A window left of the origin, or off every display, reports its bounds as they are, negative included. Its controls still act through UI Automation.

Short painted labels are read side by side. Windows' OCR reads no isolated two-character token ("A1") of a painted grid on its own, so after a window is read, the engine copies such marks into one strip led by a word it did read, and reads them again there. A label read this way may still be misread ("C1" as "Cl"), so a locator for it needs fuzzy or a pattern, and a DPI-unaware window's stretched picture loses some. A word ("Save") is read in light, dark and high-contrast themes alike. Single glyphs that make up a grid of cells (charmap's) are joined into one word per row, as a larger scale reads them, so getByText("PQR") lands on the Q at every scale; a lone row of glyphs (a painted B/I/U toolbar) stays as read. For painted grids, prefer locators the tree answers, getByImage(), or longer text nearby.