verify-openhands
This skill should be used to "verify OpenHands features", "test the Canvas UI like a user", "drive Agent Canvas", "check a UI change in the real app", "create or update the feature map", "run the daily pass" or "run the weekly feature audit". Ships control-openhands (launch, doctor, browser, LLM profiles, conversations, evidence, map tooling, cleanup) and the maintained map of every user-facing feature.
Savant verdict: Pending analysis
Imported from the source hub; Savant's analysis is still running.
Live evaluation
Not evaluated yet. Workspaces can request a live evaluation.
Safety (NVIDIA SkillSpector)
Scan pending.
Structure
- 8 referenced files aren't in the packagescripts/, references/feature-map/README.md, references/report.md, references/mapping.md, references/daily.md, references/maintenance.md, …. They may be binary, too large to catalog, or missing upstream.
- No license declaredConfirm you may reuse this skill before importing it into your repository.
SKILL.md
--- name: verify-openhands description: > This skill should be used to "verify OpenHands features", "test the Canvas UI like a user", "drive Agent Canvas", "check a UI change in the real app", "create or update the feature map", "run the daily pass" or "run the weekly feature audit". Ships control-openhands (launch, doctor, browser, LLM profiles, conversations, evidence, map tooling, cleanup) and the maintained map of every user-facing feature. triggers: - /verify-openhands - feature map - verify feature --- # Verify OpenHands through the real app Prove what a user sees, not that a route rendered or CI passed. Three parts work together: 1. **`control-openhands`** ([scripts/](scripts/)) is the lever. It launches this checkout as an isolated real stack (Agent Server, automation, static frontend, ingress), keeps one browser alive between commands, and turns every user step into a command you can rerun. 2. **The feature map** ([references/feature-map/](references/feature-map/README.md)) lists every user-facing behavior with stable IDs, user entry points, exact `control-openhands` recipes, observable results and gotchas. 3. **Evidence**: screenshots, ARIA snapshots and a pass/fail/blocked/not-run ledger (plus xfail/xpass for known bugs) that survive cleanup. Neither this skill nor the map authorizes external writes, paid models beyond the budget you were given, or product fixes the user did not ask for. ## The CLI comes first Put it on `PATH` and read its help before anything else: ```sh export PATH="$PWD/.agents/skills/verify-openhands/scripts:$PATH" control-openhands --help # then: control-openhands <command> --help ``` Requirements: Node >=24 (the launcher's engine), npm dependencies installed (`npm ci --ignore-scripts`), `uv`/`uvx`, and a Chromium Playwright can launch (set `CONTROL_OPENHANDS_BROWSER=/path/to/chrome` when the pinned browser is not installed; `launch` then reports the running stack and `browser start` picks it up). `browser record` also needs an ffmpeg: one with libx264 on `PATH` writes MP4 (and GIF), and Playwright's bundled one (`npx playwright install ffmpeg`) writes WebM. Every command prints one JSON object; exit 0 ok, 1 action failed, 2 usage, 3 environment. If a user path cannot be driven with the CLI, that is a **harness gap**: extend `scripts/control-openhands.mjs` (keep it executable, document the verb in `--help` and below), prove the new verb live, then write the recipe. Never work around a gap with an untracked one-off script the next agent cannot rerun. Other agents on the machine may be running the same script: edit a copy, run `node --check` and the tests on it, then move it into place in one step. ## Launch → doctor → drive → evidence → cleanup ```sh export OH_VERIFY_RUN=$(control-openhands launch --new --print-run) # builds if needed; isolated run control-openhands doctor # read-only; must be ok before driving control-openhands llm preset deepseek # deepseek-flash (active) + deepseek-pro; key from $DEEPSEEK_API_KEY or --api-key-file control-openhands onboard --skip # consent + onboarding (walk it instead when F01 is under test) control-openhands browser goto /settings/secrets control-openhands browser testids # discover handles on the current page control-openhands browser click 'testid=add-secret-button' control-openhands browser screenshot --feature F14.create --name form control-openhands evidence add --feature F14.create --result pass --entry "Settings > Secrets > Add" \ --expected "add form" --actual "add form" --artifact evidence/F14.create/form.png control-openhands stop # stops only this run; evidence stays ``` - **Which run.** Commands use `--run DIR` (anywhere on the line), else `$OH_VERIFY_RUN`, else the only live run. With several live runs they refuse to guess, so export `OH_VERIFY_RUN` in every shell command when other agents share the machine: an agent that drives someone else's run corrupts both evidence ledgers. For the same reason never `pkill -f` or `killall` by pattern; `control-openhands stop` ends only your run. - **Launch** refuses to start when less than about 2 GB of memory is free: each run holds an Agent Server, automation, a static frontend and Chromium (about 1.5 GB together). Stop runs you are done with; several agents on one machine should each `launch --new`, export their own `OH_VERIFY_RUN`, and never touch another agent's run. - **Launch** starts `bin/agent-canvas.mjs` from this checkout with a private `HOME`, state, session key and free port block, so it never touches a user's `~/.openhands` or another run. `launch --new` starts a second independent run (for a baseline, or to drive two backends); `--public` exercises the API-key login screen (`control-openhands login`). It still binds to 127.0.0.1; it only stops injecting the session key into the page. `--sdk-version`, `--sdk-ref` or `--sdk-path` (and the `--automation-*` equivalents) choose other backends; record them. Version variables exported in your shell are not forwarded. The block includes the VS Code editor port (`ports.vscode`) only when the checkout's launcher sets VS Code up (the port is reserved even if the agent-server has no editor binary and nothing listens there). Where that is opt-in (#17660), `--vscode` turns it on; where the editor is bundled (before #17660, or since #18048), it is always included and `--vscode` changes nothing. - **Doctor** checks the launcher's process group, ports, served build revision, unauthenticated rejection, authenticated settings, Agent Server pin and the UI's minimum Agent Server version, automation health and a throwaway-tab UI probe. Run it first, after every surprising failure, and before blaming the product. A wedged UI on a healthy stack: capture evidence, `browser reload` or `goto /`, retry once. - **Drive** through the real UI with `control-openhands browser ...`. Selectors are `testid=`, `role=button[name="Save"]`, `label=`, `text=`, chained with ` >> `. Use `browser testids` and `browser snapshot` to find handles; prefer scoped test IDs and accessible names over CSS. Failures return a hint and a screenshot path; read the screenshot before retrying. A click returns the URL from *before* any client-side navigation: use `click ... --expect-url '<regex>'` (or `browser wait-url`) after every navigating click, because many routes redirect (`/settings` → `/settings/agents`, `/customize` → `/mcp`). - **Arrange, don't fake.** `llm`, `fixture` and `api ... --write` exist to set up preconditions (a configured profile, a git repo, a dummy secret). They never count as proof that the UI path works; the map says which steps are UI proof. Never intercept routes or add mock LLM responses to make a live check pass. - **Essential pathways** are single commands so recipes can start from a known state: `onboard`, `llm preset|set`, `conversation start --prompt ... --wait` (add `--workspace qa-repo` to run it in a `fixture git-repo`), `conversation events <id>`, `workspace open`. - **State control** reaches states a happy path never shows, without mocks: `browser reset` (fresh browser profile: first run again), `service stop automation|agent-server` (backend-down UI), `restart` (same state after a backend restart: persistence and reconnection) and `restart --rotate-key` (stale session key). `restart` keeps the browser on its page. While a service is stopped, a reload replaces the page with the backend-unavailable screen, so drive backend-down states on the page that was already loaded. `service pause automation` and `service resume automation` queue and release real requests for in-flight race checks. Resume before stopping the run. `browser network` (with `--bodies`, what the page sent, credentials redacted), `browser toasts` and `browser media` observe requests by origin, toasts and sound without changing anything; `click --observe SEL` and `scroll --observe SEL` record the transient states SEL shows while the action's effects play out (a "Saving..." label, a loading row). Check `--help` before calling a state unreachable: most "can't be driven" claims predate a verb. - **Evidence** goes under `<run>/evidence/<feature-id>/` and the append-only ledger `<run>/evidence/ledger.jsonl`. Nothing is overwritten: a repeated screenshot name is saved as `<name>-2.png`, so cite the path the command prints; `evidence report` renders the table from [the report contract](references/report.md), fail and blocked rows first with their `--note` (a blocked row names its missing prerequisite there), and `--baseline <yesterday's run>` lists what changed since that ledger. When timing or a transition is the point (a menu closing on Escape, a reply streaming in), record it: `browser record start --feature ID --name N`, drive the steps, then `browser record stop` saves a video of the active tab under the same folder (`--gif` adds one that GitHub shows inline). Around a long wait, such as the agent working, run `browser record pause` and `browser record resume`: the video cuts there. The caret is hidden in recordings, as in screenshots. Keys, logs, browser profile and downloads stay in `<run>/private/`. Evidence is not automatically public: review every image before publishing it. The CLI masks password fields in `snapshot`, `value` and `testids`; a screenshot of a visible key field is still a leak. - **Cleanup** with `control-openhands stop` (add `--purge-private` to delete keys and state once you have checked the evidence). It only signals the process group it launched and verifies the ports closed. Delete run-owned fixtures through the UI when deletion is the path under test. ## LLM budget Features that start a real agent with automation or debugging skills (for example "Debug with OpenHands") can act on other fixtures by name: stop those conversations when their recipe is done. Use `deepseek-flash` for everything that needs a model; switch to `deepseek-pro` only for checks that need a second profile or a stronger model. Keep prompts small and confined to the run workspace. The local stack is not a sandbox. The agent's terminal runs as your user, and its environment includes the run's own keys. A prompt that leaves the model without a clear task can send it exploring outside the workspace, and what it reads goes to the model provider. That covers an image with no text, a bare trigger word whose skill is not loaded, and an open-ended setup command. On 2026-10-08, deepseek-flash read the run directory, the checkout, other sessions' files under `/tmp` and `env` in such cases. So add `Do not run any tools` (or name the exact command) to every prompt whose check does not need more. Watch `conversation events <id> --kinds ActionEvent` while a run that needs tools is open, and run `conversation pause <id>` as soon as it leaves the workspace. Do not launch the stack as a user whose files hold credentials you cannot let a model read. Without a key, run every credential-free recipe and record model-dependent ones as `blocked` with the missing prerequisite; never substitute a mock and call it a pass. ## Choose the job - **Verify a change or a PR**: `control-openhands map affected --base <ref>` (or `--paths` with the PR's file list) maps the changed paths to the families whose `Source:` lines own them, and separates shared code to widen, `src/` paths no family owns (a map gap) and non-user-facing paths. Drive every entry point those features list at desktop and phone viewports, and report with [the report contract](references/report.md). - **Verify a fix for a known bug** (an open PR or a merged commit that claims to fix a bug the map records): launch the baseline and the fix as two runs (`launch --new` from each worktree, same backend pins), then run the note's known-bug block twice on each with this checkout's CLI: `control-openhands bug run <ID> --attempts 2 --label baseline --run $BASE_RUN` must report `xfail` both times, and `--label fix --run $FIX_RUN` `xpass` both times. `control-openhands bug proof <ID> --baseline $BASE_RUN --baseline-label baseline --fix $FIX_RUN --fix-label fix` gives the verdict: `verified`, `not-fixed`, or `inconclusive`. A baseline that does not reproduce twice is inconclusive, never "fixed". When the note has no block yet, write one first ([mapping.md](references/mapping.md)) and prove it reproduces on the baseline. Every attempt runs the same recipe, viewport and fixtures, and screenshots or recordings stay part of the proof. - **Run the daily pass**: follow [references/daily.md](references/daily.md): static checks (`map check`, `map coverage`, `map testids`), the changed families since the `Maintenance baseline` line of the map index (`control-openhands map baseline`; `map affected` starts there by default), a smoke row per family, and a date-derived rotation that gives every family a full live pass once a week. A pass that finished its changed families proposes `TARGET` as the next baseline with `map baseline --set "$TARGET"` in its PR; compare ledgers with `evidence report --baseline` when yesterday's is at hand. - **Create or extend the map**: follow [references/mapping.md](references/mapping.md). It teaches how to discover features, write entries against the CLI and prove each one live. `control-openhands map coverage` measures what is still unmapped. - **Maintain the map (periodic or weekly)**: follow [references/maintenance.md](references/maintenance.md): index hygiene, a source wave, one live pass over every feature, PR-intent reconciliation for the week's changes, and at most one PR of proven corrections. ## Triage what you find - **Map drift**: the map describes something the app no longer does by design. Fix the entry with current source and live evidence. - **Harness gap**: the app works but the CLI cannot drive it. Fix the CLI, re-drive. - **Product bug**: the app is broken. Keep the evidence, search existing issues, report it separately (right repository: Canvas UI here, Agent Server/SDK in `OpenHands/software-agent-sdk`, scheduling/dispatch in `OpenHands/automation`). Never rewrite an expected result to bless broken behavior. Then record it in the map, which should know every open bug: a `Known failure (reproduced <date>, #<issue>)` note on the bullet that reproduces it, next to the expected result, or a `Known issue #<issue>` Gotchas entry when no recipe can drive it. Make the note executable with a `known-bug` block where a recipe can drive it, so `control-openhands bug run <ID>` reports it as an expected failure (`xfail`) and flags it (`xpass`) the day it stops reproducing. Passes re-drive these notes once the issue closes. - **Blocked**: name the missing prerequisite (account, entitlement, OS, binary) and the route you attempted. Unreachable is never a pass. See [references/adaptation.md](references/adaptation.md) for where these ideas come from and what was deliberately left out.