Studio: the Inspector, the Recorder and Pictures

Studio is the window in which you look at an application as SPART-Desk sees it, record a test from what you do in it, and keep pictures for getByImage. This page is for anyone writing tests with it or capturing a support bundle.

npx spart-desk studio                               the Inspector (what SPART-Desk sees, and the locator for each control)
                                                    and the Recorder (what you do in the application, as a test)
npx spart-desk studio --recorder                    start on the Recorder, with recording on
npx spart-desk studio --app CustomerApp             attach to a running application (a pid, a process name or a window title)
npx spart-desk studio --launch C:\Apps\crm\crm.exe  start the program and attach to it
npx spart-desk studio --launch notes.exe --java -- --no-splash
                                                    start it as a Java application, with its own arguments after --
npx spart-desk studio --no-open --port 4321         print the link only, on a fixed port
npx spart-desk studio --support                     keep more evidence per step, for a support bundle
npx spart-desk studio --sandbox                     the application runs in Windows Sandbox (experimental)
npx spart-desk studio --allow-path ..\shared-tests  also let Save… and Append to… write tests under that folder (several: separate with ;)
npx spart-desk images [--check]                    write (or check) the typed index of the image folder's pictures

Studio is three tools over one session. The Inspector shows what SPART-Desk observes in an application and, for the control you pick, every locator that could address it and why one is recommended. The Recorder turns what you do in the application into steps and into the test they make. Pictures saves pictures of the application for getByImage, by name, and shows where a saved picture is found. All three are in one window, titled SPART-Desk Studio; the switch under the title moves between them on the same application, so a control picked in one is the control the others talk about. The window starts on the Inspector, or on the Recorder with --recorder.

The window is the page alone: no address bar, no tabs, no menus. The command opens it with Microsoft Edge or Google Chrome in a window of its own, and in the default browser where neither is installed. Edge or Chrome runs it in a browser profile of Studio's own (studio-browser in SPART-Desk's cache folder): none of your extensions, no account signed in, nothing synced, and no other window of yours in its process. Behind it is a small server on 127.0.0.1 that the command starts.

Nothing about a control is decided by a window: the observation is the engine's snapshot(), the candidates are candidatesFor's, the code is renderTest's.

Choosing the application

Choose application…, in either tool, offers two ways:

• Choose one that is running. Every window a user can see (desktop.windows()), grouped by process; choosing one attaches to that process and observes it. A process that belongs with another stands under it: under the process whose window owns one of its windows (a dialog another process shows for it), or under the process that started it (an application's helper processes, parentProcessId). Explorer and the rest of the shell adopt nobody, so an application started from the Start menu stands on its own. A window above this process's integrity level is marked read-only. The generated test attaches by process name (desktop.attach({ processName })), never by pid.
• Start an application. The program, its arguments, and a check box for a Java application. The generated test launches it the same way (desktop.launch({ executable, args })). What Studio started keeps running when the session ends: Studio never closes an application.

--app and --launch do the same from the command line. --app takes a process id, the name of a process that shows a window, or a window's title.

Java applications

UI Automation sees a Swing or AWT window's frame and nothing in it; what is inside comes from the Java Access Bridge, which a Java runtime loads only when it was told to at start (java-apps.md).

• Started from Studio, the bridge is turned on for that process alone. The engine recognises java.exe, javaw.exe, a .jar, a launcher with a bundled runtime and the common launcher stubs (launch4j, exe4j and install4j .exe files) by itself; the check box, or --java, insists for one it does not recognise.
• Started elsewhere, the application shows its inside only if its Java runtime has the bridge on (jabswitch -enable in that runtime's bin folder, once, for every Java application of the account). Otherwise both tools say so in a banner, with the two ways out: start it from Studio, or turn the bridge on and start it again.
• A launcher that starts another process to show its window is found by that process: --app javaw, or the window's title.
• An application that allows one instance hands a launch to the instance that is running. Studio says so; close that instance and start it from Studio.

The Inspector

┌ SPART-Desk Studio   [Choose application…] [Pick control] [Refresh] [Save state] [now ▾] [window in front ▾] [☀] ┐
│ [Inspector|Recorder]  CustomerApp (pid 4231)                                                                            │
│ Application                       │ Control                                                       │
│  27 controls in 412 ms            │  button "Save" #btnSave                                        │
│  Saved states (2/20)     [Share…] │  [Read this area] [Note on this control…]                      │
│   ▣ main · 25 kept · 1 weak spot  │                                                                │
│  [filter the tree]                │                                                                │
│  window "Customers"               │  recommended · semantic · matches 1 · uia                      │
│    textbox "Name"                 │    app.window().getByRole("button", { name: "Save" })            [Copy] │
│    button "Save" #btnSave         │    ✓ unique in the application  ✓ role and name from the tree  │
│    checkbox "VIP"                 │  identifier · matches 1 · uia                                  │
│  screenshot                       │    app.window().getByAutomationId("btnSave")                     [Copy] │
│                                   │  ▸ Details                                                     │
├ Diagnostics            local · windows · ✓ tree ✓ java ✓ text ✓ picture ○ elements ○ grounding (experimental) [Share…] ▴ ┤

• The tree is the observation, one line per control, with a filter. Clicking a line selects the control. While the mouse is on a line, the control is framed on the screen; the frame goes when the mouse leaves the line.
• Pick control points at a control in the application. While the mouse moves over the application, the control under it is framed and shown; a click selects it and ends the pick, and the click never reaches the application. Over anything that is not the application nothing is framed and nothing is picked. Escape, or the button again, cancels.
• The control is shown by role and name, with its coverage (the tree, the screen only, a grounding provider (experimental) only), and every candidate locator ranked best first: its class, how many controls it matches, the resolvers that may answer it, and reasons marked ✓ ⚠ ✗. Copy copies a locator. Details holds the normalized fields and, opened, what the control says of itself beyond the tree, read through the engine as details() reads it; All properties adds every property and pattern its provider has, by UI Automation's names. Under each candidate, a line says what each resolver it may reach is known to do: ✓ evaluated (the tree's match count, the screen read for Read this area, or what an engine run of that locator found), ○ not evaluated (it would run, and nothing has run it), – unavailable (it cannot run here, and why). Nothing is run to fill the line in.
• Detailed, in the header, shows ⚠ reasons, the observation's notes and what every resolver did in each operation of the drawer. Off, the window shows the problems, ✓ and ✗ reasons and what answered each operation.
• Read this area observes the control's area with the text painted in it (snapshot({ enrich: "text", root })), which adds the text the tree does not report as controls with getByOcr candidates.
• Refresh observes the application again and takes a screenshot. Observing is the same app.snapshot() a test would take, which takes a moment on a real application, so it happens on request, after each recorded step, and when a window of the application moved or changed its size.

The Inspector looks and never acts: nothing in it clicks, types or records.

Saving the application's state

Save state keeps the application as it is now, to share with whoever improves SPART-Desk. Save one of each screen that matters: the main window, a dialog, a tab, an open menu. Every state is of the one application Studio works on; a bundle holds one or many states of it.

• One window at a time, by default. A state is about the window you work in, with the menus, drop-downs and tooltips open over it (each a window of its own without a title). Its pictures, kept locators and weak spots are of that window, and it is named after it (main, or the dialog's title).

  • The chooser beside Save state says which window: window in front (the default), all windows, or one window by its title.
  • Window in front is the application's window that has the focus: the one you are in when the countdown runs out or you press the hotkey. Pressed in Studio, which has the focus then, it is the application's window in front of the others.
  • The tree is observed whole either way: a test searches the whole application, so a kept locator is one that tells its control apart there. A dialog's OK is kept as the OK in that dialog when the main window has one too.

• One press pictures the windows and observes the tree together, then observes again with what reading the screen adds and with what the screen draws where the tree has nothing. It asks Windows what the processes and windows are, and pictures each window a second time to see whether it repaints by itself. The windows are framed on the screen for a moment once it is saved, and the Inspector then shows the tree the state saw.

• A menu, a drop-down, a context menu closes when Studio is clicked. Use one of two ways:

  • Ctrl+Shift+F9, pressed in the application, saves a state with the menu open. Windows keeps the key from the application.
  • The countdown: choose in 3 s, in 5 s or in 10 s beside Save state, press it, and open the menu before it runs out. A click on the button cancels it.

  A tooltip may close at the Ctrl and Shift of the hotkey: use the countdown for one. The list under the application says when the hotkey cannot work, and why: a helper from an older build, or another program that holds the keys.

• The list of saved states shows each one's picture, name (main, or the dialog's title), time, windows, controls, the locators it kept and its weak spots. What it holds lists them. Rename, Note… (what someone should look at) and Delete act on one state.

• Note on this control…, in the control's pane, writes a note on the selected control in the latest state: "found twice", "the grid shows no rows here".

• Kept locators: for each state, up to 25 of the controls a test would use keep their recommended locator and the control it found, so a shared state is a set of regression cases for whoever improves SPART-Desk. Controls new since the previous state come first.

• Weak spots: a control only its position tells apart, a window pictured as one colour, texts only the screen shows, the observation's coverage notes. They are where SPART-Desk is weak on the application: the list of what to fix.

• Studio keeps 20 states. They are of the chosen application: choosing another lets them go, after the page asks. Share… writes them, see Support capture and bundles.

The frame on the screen

A frame belongs to the window it was drawn on. It goes when that window moves, changes its size, is minimised or closes, and the application is then observed again, so the next frame is drawn where the control is. When a pick ends, however it ends, the frame goes with it. A screenshot is taken without it.

The frame is green, the green of SPART-Desk's mark, with an outer rim in the mark's dark teal: the teal keeps it plain against a light application, the green against a dark one. While it shows, a blue stretch travels clockwise around its edge, so that it is seen to be Studio's and not a border the application drew. The frame is four thin windows along the edges of the control: clicks go through them, they never take the focus, and nothing covers the control itself.

The control under the mouse is the one a click there would reach. Where another program lays a see-through window over the screen (the overlay of a display manager, for one), the control is looked for inside the window Windows itself finds under the mouse.

The Recorder

┌ SPART-Desk Studio                                                  [Choose application…] [■ Stop recording] [☀]                      ┐
│ [Inspector|Recorder]  CustomerApp (pid 4231)                                                                                         │
│ Steps  2 steps: 2 semantic   [Undo] [Clear]  │ Test  [whole test ▾] [playwright ▾] □ mask [Edit as text] [Copy] [Save…] [Append to…] │
│  Recording: what you click and type …        │  import { test } from "@playwright/test";                                             │
│  1. fill textbox "Name" with…  [✎][↑][↓][×]  │  …                                                                                    │
│  2. click button "Save"        [✎][↑][↓][×]  │    await app.window().getByLabel("Name").fill("Alex Example");                                 │
│ Add to the test             [Pick control…]  │    await app.window().getByRole("button", { name: "Save" }).click();                           │
│  checkbox "VIP"                              │                                                                                       │
│  Expect… [visible] [hidden] [unchecked] …    │                                                                                       │
│  Wait… [hidden] Add a step [Hover] [Drag…] … │                                                                                       │
├ Diagnostics                                       local · windows · ✓ tree ✓ java ✓ text … □ support capture ▴                       ┤

• Recording is on when Studio starts on the Recorder (--recorder); otherwise Record turns it on. What you do in the application becomes steps: a click becomes click() (a double click dblclick(), a right click click({ button: "right" }), a click that toggles a check box check()/uncheck(), one that opens or closes a tree item, on it or on its expander, expand()/collapse(); a click that opens a drop-down and a click on the item it then shows become one selectOption("…"), and a click on a button's caption or picture is a click on the button), a run of keys into one control becomes a single fill() with the field's value read back afterwards (what the keys themselves typed is kept with the step as evidence), Enter, Tab, the arrows and chords become press(). A drag from one control onto another becomes dragTo(), one that ends on no control a drag between points. Resting the mouse on something becomes hover() only when the next click lands on what the rest revealed (a menu that opens under the mouse, a row's Delete button). Turning the wheel becomes scroll({ wheel }) on the list, tree, grid, table, document or text box under the mouse, one step for a run of notches over one place (a wheel turn at the point when nothing there has a locator); other moves record nothing. Control, Shift and Alt held at a press or a wheel's turn are part of the step: a click, a drag or a scroll with modifiers, and a turn with Control held is a step of its own beside the plain turns. What is typed into a password field is not kept.
• Each step is mapped onto the application as SPART-Desk observes it (after the mouse rested, onto a fresh observation, which holds what the rest revealed) and gets the same candidates the Inspector shows: by UI Automation's identity, else by the control's properties, else by what holds the point. In a Java window the control is the one the Java Access Bridge finds under the press the moment it is reported, and keys go to the control the bridge says had the focus before them. A click nothing addresses is recorded as raw coordinates. What is typed into a control that reports no text (an editor that draws its own) is recorded as keys.
• What is recorded is what you do in the application's own windows. A click in another application is noted in the drawer and makes no step. The click that ends a pick is Studio's own and makes no step either. A window of a process that runs as administrator, when Studio does not, shows nothing to Studio: nothing is picked or recorded there, and the drawer says so once. Run Studio from an elevated terminal to work in it. When Windows removed the mouse hook that ends a pick (it does so to a hook that stalled), Studio puts it back and the drawer asks for the click again.
• Recording happens in the Recorder only: while the window shows the Inspector, or once it is closed, nothing you do is captured; back on the Recorder, recording goes on where it was. Record pauses and resumes.
• Steps carry a badge where the locator is fragile (nth, a value, text read off the screen) or missing. A cell is found by its column in the row a text of the row tells apart, or, where no text does, in the row at its place in its grid. Each step can switch to another of its candidates, take a locator you type (✎), move up or down (↑ ↓), or be removed (×). Clear removes every step, so the test starts over with the same application; input that had not become a step yet (typing not finished, a click still being looked at) is dropped with them.
• A typed locator is written as locator() takes it (role=button[name="Save"], automationId=btnSave, … >> …; see locators) and checked against what Studio observes now. It must match one control: one that matches several is refused, one that matches another control than the step was recorded on is taken with a warning, and one that matches none is taken only when you say so (the control may not be shown now). The step then carries a typed badge.
• Undo (or Ctrl+Z outside a text field) puts the steps back as they were before the last change: a step recorded, moved, removed, given another locator, or the steps cleared; and starting or ending an edit of the test as text. It goes back up to 100 changes, and starts over with another application.
• Add to the test: pick a control in the application (or select one in the Inspector), then say what the test expects of it, what to wait for, or what to do with it. Every button says in its title what it records.
  • Expect… offers what the control can hold: that it is visible or hidden (gone); checked or unchecked, enabled or disabled, expanded or collapsed, focused or not, as it is now; editable or not (enabled and not read-only, as fill() needs it) and selected or not (the chosen item of its list, tab strip, tree or combo box), read through the engine when you ask; a grid's or table's row count (toHaveRowCount(n), toBeEmpty() for none), a list's item count, and, for a picked row, that its grid contains it (toContainRows([{ Name: "Ada", … }]), the cells by their column headers); and that it looks as it does now, as a golden picture (toHaveScreenshot(name), which the test writes on its first run). The text, the value (where the control has one) and the title (for a window or a dialog) each have a line: the text the control shows is filled in, and is expected whole, as a part (toContainText) or as a pattern typed as /source/flags (toHaveText(/^Ready/i)); left empty, the text is read when you press Add, as before. A window's title is asserted on the window itself: await expectDesk(main).toHaveTitle("Customers").
  • Wait… adds waitFor() until the control is visible, hidden, attached or detached.
  • Each assertion is confirmed through the engine before it becomes a step, within 3 s. One that does not hold now (a control expected hidden while it shows, a text the control does not show) is refused and the window asks whether to add it anyway; added, the step carries an unconfirmed badge that says why. A state the control does not report at all (a button asked whether it is checked) is refused outright.
  • ⌖ on a recorded step makes that step's control the subject instead of the picked one ("button "OK" · from step 3; it may be gone now"): the way to expect a dialog hidden, or to wait until it is detached, after it closed and can no longer be picked. A state that has to be read from the control is refused while it is not shown.
• The test re-renders as the steps change: a whole test file in the shape spart-desk init writes, the same with the controls it uses as getters of a page class (page.saveButton.click(); a getter per control, named for its role and name), or the steps alone; for Playwright Test (@playwright/test, with the desktop started in test.beforeAll), Vitest or node:test; with typed values masked when asked. Copy puts it on the clipboard; Save… writes it relative to where the command was started, never over an existing file without asking. It writes only a test's file (.ts, .mts, .cts, .js, .mjs) under that folder, or under a folder --allow-path <dir;dir> names. Append to… adds the steps, as a test of their own, to the end of a test file there that already starts the application: it imports @spartaficial/spart-desk and test and declares app (a file Save… wrote, or one spart-desk init made). It imports expectDesk when the steps assert, numbers a title the file already has, and writes the file whole or not at all. The code carries no quality comments. A step with no stable locator becomes a // TODO with the call commented out; a step that only its position (nth) tells apart keeps its call, under a // TODO that says so.
• Edit as text turns the test into text you edit in place. Copy and Save… then take your text. New steps still change the list of steps, not your text, and a banner says so; Re-render renders the test from the steps again, discarding your edits (Undo brings them back). Steps are appended only from the rendering, never from edited text.
• Add a step performs a step on the picked control through the engine, with the locators, auto-waiting and strategies a test would use, and records it: Click, Double-click, Right-click, Fill…, Type…, Press…, Check, Uncheck, Select option…, Hover, Expand, Collapse, Scroll into view, Focus, Clear, Tap, Scroll… (notches of the wheel, a direction, or to an end) and Drag onto…, which starts a pick: the next control picked (or selected in the Inspector) is the one it is dropped on, dragTo(target); Escape cancels the pick and Cancel the drag. For a picked window: Close, Minimize, Maximize, Restore and Place/size… (setBounds), each located by the window alone (await main.minimize()). In its window, and with no control picked in the main window: Press key in window… and Type in window… (window.keyboard) and Put on clipboard… These are offered always, beside native recording, which leaves the engine's own input out; on a desktop without the helper executable they are the way to record.
• ✎ value gives a recorded step its own value anew: a fill's value, the text typed, a key, an option, an expected text, value or title and how it is matched (whole, contains, a pattern as /source/flags), a row or item count, the wheel's notches, a golden picture's name. Nothing is replayed: the application stays as the step left it, and the edited value is what the test will do. A text read when it was recorded becomes what is expected. A password field's value is never edited into the test. Undo goes back on the edit.

What is recorded is the intent, not the input: you clicked Save with the mouse, and the generated click() replays as the engine sees fit, in the background where the control allows it.

Pictures

Some controls have nothing a locator can name: an icon-only toolbar, a status lamp drawn red or green, a widget the application paints itself. A test finds those by a picture of them (getByImage, locators.md). Pictures makes those pictures, keeps them by name, and shows where each one is found.

It needs an image folder: images: { directory: "tests/images" } in spart-desk.config.ts, under the folder Studio was started in (or one --allow-path names). Without one, the tool shows what is there and says how to set it.

Taking a picture. The tool shows each window of the application as getByImage sees it, with a window chooser above it when there are several. Take picture takes them again; the zoom (Fit, 1×, 2×, 4×) shows every pixel.

• Drag around what a test should find.
• Click a control to select it, then trim the rectangle with its handles. A control's whole area often holds a label, or a background that changes, and that makes a poor picture.
• The arrow keys move the rectangle a pixel (with Ctrl, ten). Shift with the arrows resizes it. Escape lets it go, Enter saves.

The Inspector's Capture as picture opens the tool with the selected control selected. A recorded click's Picture… opens the screen copy taken as the mouse pressed.

What the selection makes. While you select, the side pane says what a test would make of the picture:

• unique, or found in several places (those are outlined);
• too plain: one colour, give or take, which matches everywhere (a test refuses it at once, plain_picture);
• coloured: a match must have its colours too;
• very small, or large;
• resized to the scale pictures are kept at (images.scale), with the score it is found again at. When that costs too much, it suggests setting images.scale to this display's scale.

It also says whether a picture is needed at all, from what the tree knows there and what is read there:

• The selection is a control the tree knows (it covers most of it, as a dragged rectangle around an icon button does): a picture is not needed. The tool says so and gives that control's locator to copy, for example app.window().getByAutomationId("btnSave"). That locator keeps finding the control across themes, scales and colours. A picture is the way only when how it looks (its colour) is what a test checks.
• The selection lies inside a control the tree knows, but the tree knows nothing of what is drawn there (an LED painted on a dashboard): a picture is the way, scoped to that control. The code it shows looks for it there alone: app.window().getByAutomationId("dashboard").getByImage(images…).
• Text is read inside the selection: getByOcr("Ready") may find it without a picture, unless what matters is how it looks.

Saving. Name the picture by Application, Screen and Picture: crm/customer-dialog/save, saved as tests/images/crm/customer-dialog/save.png.

• Studio suggests each part:

  • the application from its process;
  • the screen from the dialog, the tab page or the window that holds the selection (main for the application's main window);
  • the picture from the control the selection is; else from the text read in it or just under it (an icon's label), with the colour of a coloured picture, so lamp-red beside lamp-green; else a number free in that screen, picture-1, picture-2. A control that only holds the selection never names the picture.

  Existing applications and screens are offered as you type.

• A name is lowercase letters and digits joined by hyphens.

• Save picture writes it, then looks for it again as a test would, in a fresh picture of every window: "found in 1 place (0.98)".

The typed index. With every save, Studio writes index.mjs and index.d.mts beside the pictures. A test or a page object imports images and walks it, with the editor completing each step:

import { images } from "../images/index.mjs";
const main = app.window();

const pics = images.crm.customerDialog; // one branch per screen, for a page object
await main.getByRole("dialog", { name: "Customer" }).getByImage(pics.save).click();

A picture renamed or deleted turns every use into a compile error.

• Pictures added by hand are indexed by npx spart-desk images.
• --check fails a build whose index is not current.
• The library says when the index is out of date, and offers Update index.

The library lists every picture of the folder by application and screen. Choosing one looks for it on the application now:

• Every place is drawn on the picture and on the screen, each with its score: solid where a test finds it, dashed for a near miss.
• A threshold slider and, for a coloured picture, the colour check sort the places. The code beside them carries what they set: app.window().getByImage(images.crm.main.save, { threshold: 0.85 }).
• Rename… and Delete… change the file and the index, and Uses lists the lines of your tests that name the picture. Studio does not change your tests: the type check finds what still names the old name.
• A deleted picture can be put back until Studio closes.

In the other tools.

• Inspector. A saved picture found inside the selected control is one more of its candidates, after the tree's ("saved picture · visual"), looked for in that control alone (getByAutomationId("dashboard").getByImage(…)); a picture of the control itself is offered as it is. It is recommended only where the tree has no candidate that addresses the control other than by position.

• Recorder. Every recorded click is looked at for saved pictures where the mouse pressed, in the screen as it was just before the press. A control that changes on the press itself (a toggle, a lamp drawn on a canvas) is therefore looked at as it was.

  • A click at a point with one picture there becomes a click on that picture.
  • A click on a control the tree knows offers the pictures found there ("use a picture…"). Used, a picture of something inside the control is looked for in that control alone; a picture of the control itself is not scoped.

  ↩ goes back, and so does Undo.

• Generated tests write app.window().getByImage(images.crm.main.save), and import images relative to where the test is saved or appended.

Every tool

• Diagnostics (the drawer) lists every operation the engine performed: strategy, the resolver that answered, what each resolver did and cost, attempts, whether a window was brought to the front.
• Closing the window ends the session, 15 seconds after it closed (a reload in the meantime keeps it). Ctrl+C in the terminal ends it at once.
• A window can be reloaded. It keeps working; a window opened from an address someone copied has no session and says so.
• One window at a time. The first window to connect holds the session. A second one (the link opened again, a tab duplicated) says that the session is open in another window, and what is asked in it is refused, so two windows never interleave what they ask over one recording. When the holding window closes, the other takes over and says so.
• What is selected is shared. A control picked or selected in one tool is the selected control in the other, also in a window that is reloaded afterwards.
• What is shown is the chosen application's. When another application is chosen, what the last one showed goes from both tools and nothing is selected, and what you did in the last one and had not become a step is not recorded in the new one, also when the new one cannot be observed: the Inspector then says so, and Refresh looks again. Of two applications chosen at the same moment, the one chosen last is the session's.
• SPART-Desk's mark is the window's icon, in its title bar and on the taskbar.
• A window that waits says so. The mark draws itself in the left pane, with what is awaited under it, until the server has answered and while an application is attached to or started. A ring turns beside the application's name while a command that takes long runs (reading an area, a screenshot, an assertion, a step through the engine, the list of windows, a support bundle) and whenever the application is observed, whoever asked: this window, a recorded step, a window of the application that moved. Neither shows for a wait shorter than a quarter of a second, and neither moves where Windows is set to show no animations.
• Light or dark. The window follows the system until the switch in the header says otherwise: it shows a sun in a light window and a moon in a dark one, and a click turns to the other. The choice is the session's: a window reloaded afterwards arrives in it, and the next session starts with the system's. Nothing is written to disk for it. The title bar is the browser's own and keeps the browser's appearance (in Microsoft Edge, Settings, Appearance): neither the page nor Windows can change it for a window of its own.

Support capture and bundles

When an application is not handled well, Studio makes the evidence portable:

• Support capture (the drawer's check box, or --support) keeps a screenshot after every recorded step, beside what is always kept: the observations, the candidates of every control acted on, and the engine's own records.

• Share… (in the list of saved states, and in the drawer) writes the bundle from one form:

  • the folder, suggested as spart-desk-states/<process>-<yyyymmdd-hhmm> where the command was started;
  • what someone should look at, kept as notes.md;
  • redaction, on unless taken off, and a regular expression for captions and titles to redact too;
  • one zip to attach, on.

  The bundle holds every saved state (states.json, with their observations and pictures), the recording and the engine's records. Open folder then shows the zip in Explorer. Studio writes a bundle into a new or empty folder; a folder that holds a bundle already is replaced only when the form is told to, and a folder of other files never.

• What a bundle holds: a folder, relative to where the command was started, holding manifest.json (every file with its SHA-256), environment.json (versions, capabilities, the configuration with paths masked, the doctor report, the readers that read text off the screen with their versions, and the display scales: each window's and, per display, the scale its windows draw at), application.json (the target, its windows, and what Windows says of its processes), states.json (the saved states: each one's observations and pictures by id, its windows' facts, its timings, its kept locators, its weak spots and its notes), recording.json (the steps with their evidence), trace.jsonl (every operation with the resolvers that ran and what they cost, plus Studio's own notes), snapshots/ (every observation kept, with each control's key), candidates/, screenshots/ (window captures and the screen copies taken at mouse presses, each with its position on the screen), helper/ (the raw conversation with the helper executable), notes.md and a README.txt.

• Redaction is on by default. Every value, every name of a cell, row, list item, text or document, and everything read off the screen becomes a token such as «t17», the same wherever the text appears, a locator's code included, so a bundle keeps its structure and carries none of the text. The places of those texts are painted over in the screenshots. Captions of buttons, menus, tabs and labels, AutomationIds, class names and bounds stay: they are what locators are made of. A text that is such a caption stays wherever else it appears too: the text a toolkit puts inside a button as a control of its own, and the caption read off the screen again. A title or caption that matches the form's pattern is redacted too. Taking redaction off in the form, or --no-redact on the command line, keeps every text.

• Screen copies at mouse presses are painted over as screenshots are, by where each was on the screen, whatever the step. One with no observation before its press is left out while redacting, and the copy of a step that was removed or cleared is never written.

• Nothing is uploaded. The folder is written where asked and the README says so. Review it before sending it anywhere: a caption or a title can still name a person.

• One file to attach: the form's zip, or --zip on the command line, also writes the bundle as <folder>.zip beside the folder, which stays. The zip holds exactly the folder's files; Windows' own tar -xf and Explorer open it.

• A removed step's screen copy stays in Studio's memory while undo can bring the step back, and goes once another application is chosen. A bundle holds only the copies of the steps there are.

npx spart-desk support capture --app <pid|process|"title"> --out <dir> [--zip] (or --launch <exe> [-- <arguments>]) writes a bundle of a running application in one command, without a window and without acting on it. It is the first thing to run on an application nobody has seen yet:

• Every window of the application is observed three times (the tree; the tree with what reading the screen adds; and with the tables and boxes drawn where the tree has nothing), each timed.
• --properties reads every property and pattern of each control of the tree's observation into properties.json (at most 400 controls, redacted as the rest is): what an unknown application's controls expose, by UI Automation's names.
• Each window is pictured twice, 2.5 s apart. The bundle says whether a picture is blank (one colour throughout, as a window drawn on the graphics card or protected from capture gives) and whether the window repaints by itself.
• application.json (version 3) records what Windows says of the application's processes: the executable's path and version, its bitness, integrity level and session, its loaded module names, and the DLLs beside it.
• The moment is one saved state, capture, in states.json: each window's class, display affinity, child window classes and picture, the locators it kept and the weak spots, which the terminal lists.
• The terminal names the libraries that say how the application is built (WPF, Windows Forms, WinUI, Qt, Direct3D, OpenGL, a browser engine, a control suite), each window's blank or repainting picture, the timings and the coverage notes.

A launched application is closed again afterwards; an attached one is left running. Redaction applies as above.

npx spart-desk support export --out <dir> [--from <trace.jsonl>] [--notes "<text>"] [--redact-pattern <regex>] [--no-redact] [--zip] writes a bundle without a window, from a trace file a run left behind (diagnostics.traceFile) and this machine's doctor report.

Opening a bundle

npx spart-desk studio --bundle <dir> opens a bundle in Studio where the application is not: the tree of its latest saved state (or its first observation) stands as the application, with the picture taken of it. The Inspector works as it does live: the tree, each control's candidates and reasons, Details (through the engine, over the captured tree), the picture. The session is read-only, and says so in its banner: starting a program, acting, asserting, recording and sharing are refused. From a library, openBundle(directory) (@spartaficial/spart-desk/studio) gives the desktop and the target to pass to startStudio({ desktop, readOnly }).

In the Windows Sandbox

Experimental

The Windows Sandbox runtime may change in minor versions until its prerequisites are validated against a live Windows Sandbox.

--sandbox runs the session on the sandbox runtime (runtime-windows-sandbox.md): the sandbox is started or reused as Desktop.create({ runtime: "sandbox" }) starts it, the application chooser lists the guest's windows through the helper kept running there, and observations, screenshots and steps cross the connection to the helper's agent in the guest, a round trip over TCP each (runtime-windows-sandbox.md). Pick control and recording run the helper's studio mode inside the guest, as another child of the agent (runtime.guestHelper): the frame is drawn in the guest, where you look through the Windows Sandbox window, and the guest sees what you do in that window; the control under the mouse is reported ten times a second there, thirty on the host. --launch starts the program in the guest. The generated test creates its desktop with runtime: "sandbox". A support bundle from such a session holds the guest helper's conversation.

From your own code

startStudio from @spartaficial/spart-desk/studio starts the same Studio over a desktop you created, for a tool or a harness that opens the window where it likes (api.md):

import { Desktop } from "@spartaficial/spart-desk";
import { startStudio } from "@spartaficial/spart-desk/studio";

const desktop = await Desktop.create();
const app = await desktop.launch({ executable: "C:\Apps\CustomerApp.exe" });
const studio = await startStudio({ desktop, application: app, view: "recorder" });
console.log(studio.url); // open it in a browser or a window of your own
await studio.closed;
await desktop.dispose();

The desktop stays yours: closing Studio does not dispose it. Picking and input capture are on by default on the local runtime. A bundle written from it holds the helper's conversation only when the runtime you created keeps one (transcript: { onTraffic }): the spart-desk studio command does that through the runtime it creates.

Security

• The server binds 127.0.0.1 on a random port (or --port) and answers only requests whose Host is that address, with no CORS. A request a browser marks as coming from another site (Sec-Fetch-Site) is refused, and a command larger than 1 MiB is refused, what arrives past that limit dropped rather than kept.

• The window is opened through a one-time link (/open/<nonce>) that serves its page with a 256-bit token inside. Every later request carries the token; the token is never written to disk. The window keeps it for as long as it lives and moves to Studio's own address (/studio), which serves the page without a token, so a reload works and an address someone copied opens nothing. Anyone who can read the terminal, the browser's command line or its memory as the same user can drive the desktop, which is the boundary the agent package's session host has too (security.md).

• The page renders every name, value and title from the application as text, never as HTML, under a content security policy that loads nothing from anywhere. SPART-Desk's mark is inside the page itself, as the script and the stylesheet are.

• Studio never brings windows to the front (foreground: "require"), starts an application only when asked to, never closes one, and writes only what Save…, Append to…, Share… and the Pictures tool's Save picture, Rename…, Delete… and Update index are asked to write. Every path is checked by one rule:

  • Save… and Append to… write a test's file under the folder the command was started in, or one --allow-path names;
  • a picture is a .png inside the image folder, and the index is its two files there; the image folder must lie under those folders too;
  • a bundle is a folder, and its .zip beside it, under those folders too: written into a new or empty folder, beside where it belongs first and then put there whole. A folder that holds a bundle is replaced only when the form says so; a folder of other files never.
  • Open folder shows only the bundle Share… wrote last; the page never names a path to open.

  Pictures, the index and appended tests are written whole or not at all. The page sends rectangles, never pixels: the server cuts pictures from the pictures it took, and serves a saved picture only by a name the folder holds.

• A redaction pattern (the dialog's, or --redact-pattern) runs against every text of a bundle, so one that could take exponential time is refused: a group repeated by *, + or {n,m} that holds a quantifier or an alternation ((a+)+, (a|aa)*), a backreference, or more than 200 characters.

• What you do is captured only while recording is on, an application is chosen and Studio's window shows the Recorder. Ctrl+Shift+F9 is held for Studio while an application is chosen, and saves a state; it is never recorded as a step.

• The page loads no external assets and the engine makes no network requests: the server, the windows and the helper all run on this machine, and nothing is uploaded.

What Studio reuses

Shown	Comes from
The tree, the windows, the notes	app.snapshot() (DesktopSnapshot, with coverage notes)
Candidates, classes, reasons, resolvers	candidatesFor(roots, id, { strategy, capabilities })
The recommended locator	recommendLocators, the same first candidate an agent's snapshot shows
The code	renderTest(recording, { mode, runner, maskValues }), the same renderer as desktop_export_test
The drawer	desktop.actions records: strategy, resolvedBy, resolvers, work, timings
Windows to choose from	desktop.windows()
Pictures, and where they are found	each window's picture as getByImage takes it (windowImages), and the image resolver's matcher (findPicture)
Starting an application	desktop.launch({ executable, args, java, keepRunning: true })

Programmatic use: startStudio({ desktop, cwd }) from @spartaficial/spart-desk/studio; the commands are the usual way in.