Resolution: the accessibility tree, and the screen where a locator says so

One locator model works whether a control is reachable through UI Automation, only readable on screen, or neither. Consumers write semantic locators; a planner runs the one resolver each locator names, and nothing else.

DesktopLocator
  └─ intentOf(query)          what is being looked for, and which resolver the chain names
       └─ planResolution      the one resolver that answers
            ├─ uia            DesktopDriver.resolve(query)                 — a locator that names nothing of the screen
            │                 (the tree: UI Automation, plus the Java Access Bridge inside Java windows)
            ├─ ocr            capture → text provider → matching rules     — read() and getByOcr
            ├─ image          capture → correlation → matching rules        — getByImage
            └─ grounding      capture → grounding provider → matching      — getByVision

What a locator names decides what runs

The planner is a pure function of two things: which screen resolver the chain names, anywhere in it (a step, the anchor of a relative step, an operand of and, or or a filter, a table's parts), and what this machine can do. Nothing about it depends on what the tree happens to report, so a locator costs the same on every application, and every decision it makes can be printed.

The chain names	What runs	Matched over
nothing of the screen: `getByRole`, `getByText`, `getByLabel`, `getByTitle`, `getByAutomationId`, `getByClassName`, `table()`, `list()`, `tree()`	the accessibility tree	the tree
`read()` or `getByOcr`	the reader	the tree, plus what was read
`getByImage`	correlation	the tree, plus where the picture was found
`getByVision` (experimental)	grounding	the tree, plus what the provider found
a control id found on the screen (`ocr:…`, `image:…`, `grounding:…`)	the resolver that minted it	as that resolver matches

getByText("Save") is answered by the tree, always: a text the tree does not report is not found, however plainly it is painted. getByOcr("Save") reads the screen, always, and finds the text where it is painted; where the tree knows an invokable control at that place, the action goes through it. read() names the reader for a whole chain, so that the steps after it may use what reading contributes: a button named after the text drawn inside it, a field named by the label painted beside it, a grid laid out from what was read where its tree reports nothing (main.read().getByLabel("Name"), main.read().getByRole("cell", { name: "C2" })), tabs painted over the pages a control reports, each named after its page and selected when its page is shown (main.read().getByRole("tab", { name: "Orders" })). Two ways of looking at the screen in one chain are refused as the chain is written, with LocatorSyntaxError: each screen resolver sees only what it found itself.

A pattern is as answerable as a literal for the reader: it reads everything on screen and SPART-Desk matches the result, so getByOcr(/Total: \d+/) works. Grounding is different — it has to be told what to look for, so it needs words, and getByVision(/x/) is refused before any screenshot.

Every failure says what ran:

Resolvers tried:
  ocr        1 match       98ms

When reading found nothing, its line names the closest text it did read, up to three lines ranked by how few characters differ, so a misread shows itself: ocr: 14 line(s) read in 2 windows, 0 match(es); closest text read: "Elugins Help".

Reading text off the screen

On Windows this needs nothing installed. Windows.Media.Ocr is part of the operating system, so SPART-Desk configures a reader for you and starts it on the first locator that needs one — no model, no download, no licence to review. The reader runs in SPART-Desk's helper executable (spart-desk-host.exe, which ships inside the package), started once; it answers every later request over a pipe.

Reading does three things beyond finding text:

A control the tree saw but could not name is given the text drawn inside it. The control keeps its identity, its role and its patterns; only the name is new. That is what lets read().getByRole("button", { name: "Save" }) find a button whose accessible name the application never set. Where the tree already reports a control at the place something was read or laid out (a Krypton button is a pane named after its caption, a status strip after its text; a drop-down's items are listed under the combo box while its popup window paints them), read().getByText("Ready") and read().getByRole("listitem", { name: "Blue" }) find the tree's control alone: what was drawn within its bounds only repeats it, and the control is what to act on. The line itself is still there for getByOcr. A widget drawn over a pane named after the caption it paints (a Krypton button) is the other way round: the pane has no role and cannot be invoked, so the widget stands for it, and getByRole("button") and getByText("Submit") name the same control.
Text that matched is narrowed to the words the locator asked for. A reader returns lines — a menu bar is one line, File Edit View Help — so getByOcr("File") matches that line; the target is the shortest run of its words that still matches by the same rules (substring, exact, or the pattern), so the click lands on File and not in the middle of the line. exact is satisfied by a run of a line's words as well as by the whole line: getByOcr("Help", { exact: true }) finds Help in Plugins Help, and so does read().getByRole("menuitem", { name: "Help" }). Text that is only part of one word (a substring match) is placed within that word by its width, a character's share each: a row of glyphs a program paints, such as charmap's grid, reads as one word (at 96 dpi, where the recognizer reads its glyphs only one by one, they are joined into that word), and getByOcr("PQR") lands on the Q. An exact match is a whole word: Save is not found in Saved.
Text that matched but belongs to no named control is looked up in the tree at the place it was read. When something invokable is there — a list row, a menu item — the action goes through it instead of through the mouse, which is faster and works on a window that is not in the foreground.

A line of text becomes one candidate, never two: if it named a control, it is not also a control of its own.

The language it reads in

The reader built into Windows reads one language at a time, with a recognizer Windows installs per language:

The desktop's languages. ocr.languages lists them, best first: { ocr: { languages: ["nl-NL", "en-US"] } }. The reader reads in the first one installed. Without the setting it reads English (en-US, or another English installed), and where no English recognizer is installed, the user profile's language; its version then says so: Windows 10.0.26200, de-DE (en-US not installed).
A locator's own language. getByOcr("Opslaan", { language: "nl-NL" }) and read({ language: "nl-NL" }) read in that language, whatever the desktop's are. One locator reads in one language: a chain that names two is refused when it is written.
No silent fallback. A language with no recognizer installed is never read in another. The read fails at once, with the recognizers that are installed and the command an administrator installs one with:
```
Add-WindowsCapability -Online -Name "Language.OCR~~~nl-NL~0.0.1.0"
```
With several readers configured (provider: ["windows", "paddle"]), the next one reads instead.
What it reads in is said. The reader's version names its language, and those asked for that are not installed: Windows 10.0.26200, en-US (nl-NL not installed). The version is in the support bundle and in a miss (read by …). npx spart-desk doctor lists the recognizers installed.

A tag is checked where it is written: "nl_NL" is refused, and "nl-nl" is taken as "nl-NL".

A reader that reads more

Where the Windows reader misreads (bitmap fonts such as MS Sans Serif, Delphi forms, painted grids, selected or coloured text), the optional package @spartaficial/spart-desk-ocr-paddle reads with PaddleOCR's PP-OCRv6 instead. Its models ship in the package and run in WebAssembly, so it needs nothing but npm i. It reports a confidence per word, so ocr.minimumConfidence applies to it.

import { createPaddleOcrProvider } from "@spartaficial/spart-desk-ocr-paddle";

await using desktop = await Desktop.create({ ocr: { provider: createPaddleOcrProvider() } });

It reads texts the Windows reader misses, and takes much longer per window than the Windows reader does, which is why it is not the default.

Several readers, tried in order

ocr.provider also takes a list. A locator is looked for in what the first reader read; the next one reads only when the text was not found there, and the first reader that finds it answers.

import { createWindowsOcrProvider } from "@spartaficial/spart-desk";
import { createPaddleOcrProvider } from "@spartaficial/spart-desk-ocr-paddle";

await using desktop = await Desktop.create({ ocr: { provider: [createWindowsOcrProvider(), createPaddleOcrProvider({ characters: "latin" })] } });

Windows then PP-OCRv6 finds more texts than either reader alone, at little extra cost: the Windows reader finds most texts itself, and PaddleOCR's slower read is paid only for the rest. The second read of short labels side by side runs with the last reader only.

Each reader remembers what it read from a capture until something is acted on, so a wait polling an unchanged screen reads it with none of them again.
An observation enriched with text (snapshot({ enrich: "text" })) asks for no text, so the first reader reads it.
A failed lookup says what each reader read, closest text first; records name the readers in order (windows → paddle).

What a control found on screen is called

A control from the tree keeps the id its backend gave it. One found on screen is named by what it is: the resolver that found it, the evidence it was found with, the tree control it sits in, and which occurrence of that evidence it is there, in reading order.

Id	Is
`ocr:Save@12:dlg#2`	the second line reading exactly "Save" inside tree control `12:dlg`
`ocr:File%20Edit%20View@12:menu#1/Edit`	the word "Edit" within that line
`image:save.png@12:toolbar#1`	the first place `save.png` was found in that toolbar
`grounding:save%20icon@12:#1`	the first match for that description in that window

app.window().locator({ type: "controlId", id }) with such an id goes to the resolver that minted it, which looks with the same evidence and picks the same occurrence. That is how an agent's reference to painted text is acted on again. The count is taken within the container rather than the whole window, so a locator scoped to a dialog and a snapshot of the whole window give the same control the same name. The name does not change when the window moves.

How well a control matched

Every control carries a matchQuality. A control from the tree is exact. Text read off the screen is exact when it reads exactly what the locator asked for, and acceptable when the locator's text is part of it or a pattern matched it. A picture is exact at a correlation of 0.98 or more, and acceptable down to its threshold. A grounded control is never exact: it is acceptable at or above grounding.minimumConfidence, and below that it is rejected and listed as low in the diagnostics. The raw confidence a provider reported stays on the control for diagnostics. Scores from different technologies mean different things, and SPART-Desk never compares them with one another.

What it costs

One observation takes one screenshot and one observation of the control tree: the reader observes the tree once itself, to place what it read in it, to match the chain over both, and to act through what the tree has at the place a line was read. Across observations the content of the screenshot decides: an unchanged screen produces the same bytes, so what was read from them is reused — until something is acted on, at which point every reading is history and the next observation reads afresh. Assertions and textContent() do not invalidate anything, because they change nothing.

Looking only where the locator can match

A locator that starts from a container the tree knows — a dialog, a pane, a list, a grid — is only looked for inside it. The tree already says where that container is, so the screen resolvers look only at its area: getByRole("dialog", { name: "Customer" }).getByOcr("Save") reads the dialog and nothing else, getByImage searches only the dialog's pixels, and a grounding provider is handed the dialog's rectangle as region (matches outside it are dropped even when the provider ignores it). When several containers match, the area that holds all of them is used. When none does, the locator cannot match anything, so no screenshot is taken and nothing is read. That is the usual state while a wait is still waiting for a dialog to open.

The leading steps count as a container only when reading could never change what they match: an AutomationId, a window title, a role that no read text can name (a dialog, pane, group, list, grid, table, …), a control id from the tree, and nth over those. A button, a list item or a check box can be named from the text read inside it, and a text step, a label, a filter or a relative step depends on what is read. When any leading step is one of those, the whole window is looked at, exactly as before. Scoping therefore never changes which control a locator finds, only what it costs to find it.

read() reads where it stands: the area the steps before it name, when the tree alone decides them. main.getByRole("dialog", { name: "Customer" }).read().getByLabel("Name") reads the dialog; main.read().getByLabel("Name") reads every window of the application; main.getByRole("button", { name: "Save" }).read()… reads the whole window, since a button may be named from what is read inside it.

While a locator waits, reading happens at most once per resolution.screen.interval (default 1000 ms): a locator the tree answers is polled on every attempt, and one that names the reader waits for the next read. A picture is looked for more often, once per images.interval (default 250 ms), since matching one costs tens of milliseconds where reading costs hundreds.

Window by window

An application with a dialog open has two windows, and each is read from its own picture: the dialog from the dialog's, the main window from the main window's. What was read is therefore placed on the screen exactly, and nothing but the application's own pixels is ever read as the application's text. Text a window in front hides is dropped: the main window's picture still holds what lies under the dialog, but nobody can see it there. An application with one window is read from one picture, without listing its windows first.

Small text is enlarged

Recognition falls off sharply on small text, so a picture is enlarged before it is read when its text is small. A small area (under 400×400 pixels, such as one control) is enlarged by resolution.screen.upscale. A whole window is enlarged by that factor divided by the window's display scale and rounded, so a window drawn at 100 % is enlarged 2×, and one drawn at 150 % or 200 % is read as it is.

A window's display scale comes from SPART-Desk's helper executable, which measures it with the window: the DPI the window draws at, and its monitor's. Windows stretches the picture of a window that draws at less than its monitor's DPI, such as a DPI-unaware application on a 150 % display, and blurs its text. Such a window is read whole as at 100 %, and a small area of it is enlarged by the stretch as well: 3× at 150 %. A stretched toolbar's small captions are read only enlarged that far, while a window drawn sharp at 150 % reads best as it is, and a window drawn at 100 % reads far better enlarged 2× than as drawn; those are the cases the rule follows.

The words a reader still misses are often misread by one character: a letter with an underlined mnemonic (Eile for File, Yiew for View, Elugins for Plugins). Cleaning up the image before reading it does not bring those back, so there is a way to accept them (Accepting a misread).

Look-alike characters, without asking

Text read off the screen is compared once more when the plain comparison fails: with 1, I, l and | as one character, 0 and O as one, and spaces ignored, since the recognizer confuses them in small fonts. It applies to names, getByText, getByLabel, a cell's column and filter({ hasText }), and only to text that was read. Unlike fuzzy, one digit never stands for another: "Customer 10" matches a read "Customer IO", never "Customer 11".

Panels and grids that paint what they hold

A small pane that reports nothing inside it (a Delphi panel of speed buttons) is read again on its own, and what that read finds replaces what the read of the whole window found there: a read of the whole window misses small captions drawn in a 3D frame. A grid the tree reports empty (a Delphi TStringGrid) is laid out from the lines it draws and the text read in it, into a header row of columnheaders and rows of cells, so getByRole("cell", { column }) reads it like any grid. See vcl-apps.md.

Tables and fields drawn where the tree reports nothing

An application can draw a table or a form where its tree reports no control at all: a chart painted by a control without an automation peer, a form whose input boxes are drawn, a window whose tree is empty altogether. What a user recognises them by is drawn, so it is found in the picture, deterministically:

A horizontal line is a run of pixels, of one colour, that differs from the row above it; a line that another crosses runs on through the crossing. Gradients, text and pictures draw no such runs.
Lines of the same extent that stack, divided into columns by the lines between them (the grid-line detector above), are a table. It is laid out as a grid the tree reports empty is — a header row of columnheaders, rows of cells — so getByRole("row").filter({ hasText: "Order 1042" }).getByRole("cell", { column: "12:00" }) reads a painted chart.
Two lines that close a rectangle a field's height, with a border at each end, are a box: a textbox whose value is the text read in it and whose name is the label painted beside it (as for a field the tree left unnamed), or, shaped like a button with a caption read in its middle, a button named by it. A button-shaped box in which the whole read found nothing is read again on its own, enlarged, as a small pane is.

They are looked for where the tree reports no control of its own, and only for what asks for them: a locator for a grid's rows or cells looks for tables, one for a field (getByLabel, a named text box, combo box or button) for boxes, and snapshot({ enrich: "elements" }) for both. A locator for text keeps the lines as they were read: a drawn table would divide a line among its cells (charmap's row of glyphs is one word across a row of cells). What is found is a control of the source region: acted on where it is (real input, as text read off the screen is), after checking that the picture there has not changed. A locator for a grid's rows or cells reads the screen only when the grid it scopes into is not one the tree reports with its rows: rows the tree does not report are paged to, not read.

Tables of plain text in the tree

Text the tree does report can still be a table nobody declared: a panel of rows of labels, lined up in columns under a header. When a locator asks for a grid's rows or cells and the tree has none, a run of at least three rows of text controls — a header of three or more, and rows whose every text starts where a header text starts — is laid out as a table, without reading the screen. The texts become its column headers and cells and keep their identity, so reading and acting on a cell go through the tree as before. A form of labels and values (two columns) is not a table.

Accepting a misread

{ fuzzy: true } on getByOcr, getByText, a named getByRole, getByLabel or getByTitle accepts text that differs from what was asked by a few characters: up to a quarter of its length, and at least one. Characters that readers confuse cost half a difference each: E/F/P, I/l/1/|, O/0, Y/V/U, S/5, B/8 and a few more. Case is ignored. A word shorter than four letters may differ only by one such confusion, so OK accepts 0K but not OR. It applies to every source, and so to the tree too; use it for text read off the screen, where the misreads are.

const main = app.window();
await main.getByOcr("Plugins", { fuzzy: true }).click();    // matches "Elugins" in "Elugins Help"

Finding a control by a picture of it

getByImage("save.png") looks for a picture on screen. Nothing is installed for this either: the matching is the engine's own correlation, and the pictures are yours.

The measure is zero-mean normalised cross-correlation, which is independent of brightness and contrast — a button that lights up on hover still matches the template captured when it was cold — and blind to colour, because colour says little about whether two icons are the same control and a theme change moves it far more than it moves shape. A score of 1 is identical.

Every position is scored at full size wherever that costs no more than a search that starts small. Summed directly, that would be millions of positions of a thousand multiplies each for a 32×32 icon over a window. So the products come from the Fourier transform instead, tile by tile, and they are exactly what the direct sum gives. A template whose detail is as fine as shrinking, like text, stripes or a dither, is always searched this way.

In a large window, a picture that shrinks well is first looked for in a shrunken copy, where a position costs almost nothing:

The copy is compared at four phases of the shrinking grid, so a picture that lies between its blocks is not missed.
Each place that looked promising is followed down through copies twice as large each time, a few positions in each, to full size.

Flat areas — most of any desktop UI — score nothing, because correlation against a constant region means nothing.

Scale is decided, never swept. A template was captured at some DPI (images.scale) and the window is drawn at another, so the template is rescaled by the ratio. Trying a spread of sizes in the hope that one sticks is how a matcher starts finding things that are not there; a caller who genuinely needs several sizes passes scales.

Every window, front first. Each top-level window of the application is looked in from a picture of its own, at the scale that window draws at, the windows in front first. A match that a window in front covers is dropped: a click there would land on the window in front. A picture of something in a dialog is found in the dialog. Studio's Pictures tool looks with the very same matcher, so where it shows a picture found is where a test finds it.

Transparent pixels are left out. A picture saved with a transparent background, such as an icon cut out of its toolbar, is compared only where it is opaque, in grey and in colour. It is found over a toolbar that lights up under the mouse as over a plain one. A picture with no transparent pixel is compared whole.

A miss says how close it came. When no place reaches the threshold, the locator's last state names the closest place and its score: the closest place scored 0.81 at 140,90 (24×24). It also says whether only the colour differed, or that nothing came within 0.25 of the threshold. A picture slightly off (another theme, a highlight) then reads differently from one that is not there.

A picture of one colour is refused. A picture whose grey hardly varies where it is compared (a spread below 6, its opaque pixels when it has transparent ones) would match every patch of that colour, or none. It fails at once with ScreenError plain_picture, naming the picture, rather than waiting out the timeout. An ordinary icon spreads far more than that, so the refusal only ever meets a picture cut from a blank area.

A found picture becomes a control with the role image — that is what was observed. When the accessibility tree has something invokable at the same place, the action goes through that instead, which is how an icon-only toolbar button ends up being invoked rather than clicked.

Asking for more than structure

A locator says what to find; a snapshot says what to look at. Both reach the same providers, and a snapshot names the information it wants rather than the technology that supplies it:

await app.snapshot();                              // the accessibility tree. No pixels.
await app.snapshot({ enrich: "text" });            // + the text painted on screen
await app.snapshot({ enrich: "elements" });        // + interactive regions with no accessible identity
await app.snapshot({ enrich: "text", root: id });  // the same, for one control's area only

What read text adds is exactly what it adds during resolution: a control the tree saw but could not name is given the text drawn inside it, and a line that named nothing becomes a control of its own, placed in the tree where it was read. A snapshot and a locator therefore describe the same screen the same way, and they share the reading — a snapshot followed by a locator over unchanged pixels reads once.

enrichment says what happened: what was requested, what was applied, and which kinds of observation resolvers actually contributed. Asking for more than this machine can give returns what it can and records the difference rather than failing. elements adds the tables, input boxes and buttons drawn where the tree reports nothing (above), and yields applied: "text" where the picture could not be examined for them.

root is arithmetic on data already in hand: the control's bounds become the area to read, so inspecting one panel of a large window costs one small reading instead of a whole-window one. A small area is enlarged before it is read, which is also why a rooted observation often reads better than a whole-window one.

Grounding: a contract, no bundled model

Experimental

Grounding and getByVision may change in minor versions until a grounding provider has been validated on real applications.

Finding a control from a description of how it looks is a contract with no bundled implementation. Supply a provider and getByVision works; supply none and every other locator is unaffected.

SPART-Desk bundles no model deliberately. The open grounding models are vision-language models of several billion parameters whose cost belongs on hardware you choose, and whose names should never leak into automation code. Reading text off the screen, by contrast, uses the operating system's own recognizer rather than a model, with an open reader offered beside it (A reader that reads more).

import { Desktop, type GroundingProvider } from "@spartaficial/spart-desk";

const myProvider: GroundingProvider = {
  name: "my-grounder",
  locate: async ({ image, query, signal }) => [{ bounds: { x: 100, y: 55, width: 32, height: 32 }, confidence: 0.93 }],
};

const desktop = await Desktop.create({ grounding: { provider: myProvider, minimumConfidence: 0.85 } });

A provider is anything that turns a picture plus a description into rectangles: a model running on the machine, or a service of your own that locate posts the PNG bytes and the query to and whose boxes and scores it maps onto GroundingMatches. It never sees a locator, and no provider or model name reaches the test code.

Grounding serves getByVision and nothing else: reaching for something probabilistic is something a caller asks for by name.

Strictness

The rules are the same whichever resolver answered. One match is acted on. Several end the resolution as an ambiguity: narrowing the locator is the caller's job. A match below the configured confidence is never acted on.

A control found only on screen has bounds but no tree identity, so clicks go to the centre of its bounds and text is typed after clicking there (input.* strategies). hover() moves the real mouse over its middle and scroll({ wheel }) turns the wheel there, both through the helper executable, as for a control of a window the helper read. A list found on the screen under read() scrolls up and down with the wheel too, and to its top or bottom by turns of it until its picture stops changing; sideways it is refused. getByOcr, getByImage and getByVision return a ScreenLocator (api.md), which has no scrollIntoView, no scroll by direction or to an end, no state to check and nothing to locate inside: what a place on the screen cannot do does not compile. When the tree does have something invokable at that place, the action goes through it instead.

Configuration

Option	Default	Meaning
`resolution.screen.interval`	`1000` ms	Minimum pause between two looks at the screen that read text or ask grounding, while waiting.
`images.interval`	`250` ms	Minimum pause between two looks for a picture (`getByImage`) while waiting. Raise it for an application that repaints slowly: each look asks the window to paint itself.
`resolution.screen.upscale`	`2`	Factor small text is enlarged by before it is read: a small area by this much, a whole window by this much divided by its display scale.
`ocr.enabled`	`true`	`false` never reads the screen for text: `read()` and `getByOcr` raise `CapabilityUnavailableError`.
`ocr.languages`	English	Language tags to read in, best first, e.g. `["nl-NL", "en-US"]`: the first one installed is read in; none installed fails every read saying which are (The language it reads in). A locator may name its own: `getByOcr(text, { language })`, `read({ language })`.
`ocr.provider`	the one built into Windows	A reader of your own.
`ocr.minimumConfidence`	`0.5`	Lines the reader is less sure of are ignored; a reader that reports no confidence is trusted.
`images.directory`	none	Folder `getByImage("save.png")` reads from, relative to the configuration file.
`images.threshold`	`0.9`	Minimum correlation with the template, 0–1.
`images.scale`	`1`	DPI scale the stored pictures were captured at.
`grounding.provider` (experimental)	none	Your `GroundingProvider`.
`grounding.minimumConfidence` (experimental)	`0.85`	Matches below it are rejected; the timeout error lists them with their confidence.

desktop.config.resolution, desktop.config.ocr and desktop.config.grounding expose the resolved values; desktop.textName and desktop.groundingName name the providers, and desktop.capabilities says what can be used and why not.

The built-in reader is configured for the runtimes SPART-Desk builds ("local", and "sandbox", which is experimental). A runtime you supply yourself — a test double, an environment of your own — gets none, because its screenshots may not be this machine's screen at all; pass ocr: { provider: createWindowsOcrProvider() } if it is.

The contracts

interface OcrRequest {
  readonly image: ScreenImage;     // PNG bytes, the screen rectangle they cover, pixels on demand
  readonly region?: Bounds;        // part of the image to read, in image pixels
  readonly scale?: number;         // enlarge the region before reading it
  readonly signal?: AbortSignal;
}

interface OcrLine {
  readonly text: string;
  readonly bounds: Bounds;         // pixels of the image handed in, crop and scale already undone
  readonly words: readonly OcrWord[];
  readonly confidence?: number;
}

interface OcrProvider {
  readonly name: string;
  read(request: OcrRequest): Promise<readonly OcrLine[]>;
  dispose?(): Promise<void>;
}

A provider never sees a locator. It answers "what text is where", and SPART-Desk decides what that means — so exact, a regular expression and every chained step behave exactly as they do for the tree.

interface GroundingRequest {
  readonly image: CapturedImage;   // PNG bytes plus the screen rectangle they cover
  readonly query: string;          // the caller's description, verbatim
  readonly region?: Bounds;        // image pixels to look in: the container the locator starts from
  readonly signal?: AbortSignal;
}

interface GroundingMatch {
  readonly bounds: Bounds;         // in pixels of the image handed in; the engine maps them onto the screen
  // The control's role is `unknown`: a provider says where something is, not what kind of control it is.
  readonly confidence?: number;
  readonly label?: string;         // what the provider believes it found
}

interface GroundingProvider {
  readonly name: string;
  locate(request: GroundingRequest): Promise<readonly GroundingMatch[]>;
  dispose?(): Promise<void>;
}

A grounding match becomes a control whose description is the caller's words and whose name is the provider's label when it gave one. That split matters: a provider that answers a request for "the save thing" with a control it calls "Save document" has still answered the request, so the locator still matches it, while candidate lists and textContent() show what was actually found.

`getByVision` — the escape hatch

Experimental

Grounding and getByVision may change in minor versions until a grounding provider has been validated on real applications.

const main = app.window();
await main.getByVision("blue floppy disk save icon").click();
await main.locator('vision="Save"').textContent();

It exists for interfaces that expose nothing to UI Automation and paint no text either: canvas-drawn toolbars, remote-desktop windows, games. Prefer getByRole/getByText whenever the control has any accessibility or any text at all. Without a provider it fails immediately with CapabilityUnavailableError.

Failures

Situation	Result
No resolver can serve the locator at all	`CapabilityUnavailableError` naming the capability that would have, thrown immediately
A provider is missing, crashed or misbehaving	`ScreenError` (`unavailable`, `startup_failed`, `protocol`, `provider`) — thrown immediately, not retried
A provider request times out	`ScreenError` `timeout`
A screenshot cannot be decoded	`ScreenError` `decode_failed`
A picture to look for is one colour, give or take	`ScreenError` `plain_picture` — thrown at once, not waited out
Screenshot failure	`BackendError` `capture_failed` (retryable) — the wait continues; reported on timeout
Nothing found	`TimeoutError` listing every resolver that ran, what it matched and what it cost
No confident match	`TimeoutError` whose `lastState` names the rejected candidates and their confidence
Several confident matches	`StrictModeViolationError` with the candidates and their sources

Action records (desktop.actions) carry resolvedBy next to the execution strategy and resolvers with what each resolver did, so a trace shows both how a control was found and how it was acted on.

What a locator names decides what runs​

Reading text off the screen​

The language it reads in​

A reader that reads more​

Several readers, tried in order​

What a control found on screen is called​

How well a control matched​

What it costs​

Looking only where the locator can match​

Window by window​

Small text is enlarged​

Look-alike characters, without asking​

Panels and grids that paint what they hold​

Tables and fields drawn where the tree reports nothing​

Tables of plain text in the tree​

Accepting a misread​

Finding a control by a picture of it​

Asking for more than structure​

Grounding: a contract, no bundled model​

Strictness​

Configuration​

The contracts​

getByVision — the escape hatch​

Failures​