Agent tools

The tools bb-box gives a thread’s agent, with their parameters as the agent sees them.

Tool	What it does
box_acquire	Get a fresh box for this thread: a Linux or macOS computer for commands, builds, tests, servers or Docker, away from the thread’s own machine.
box_copy	Copy one file, of any size, between this thread’s workspace or its storage folder ($BB_THREAD_STORAGE) and a box.
box_device	Control an ios or android box’s device, as the panel’s toolbar does: Home, Back and Recents, lock and wake, rotate, fold, appearance, text size, location, status bar, battery, network, install an app, switch device, and reboot.
box_enrol	Add a ready box that could not join BB, or is not a BB machine, to BB again, so threads can run on it.
box_fork	Make a new box of this thread from a snapshot of one of its boxes, or from the box as it is now, to try two things from one set-up state.
box_grant	Grant a program in a macOS box macOS permissions, such as Accessibility or Screen Recording, with no prompt.
box_image_draft	Draft a project image’s script (049) from the project’s own files, for you to finish, try with box_image_try and register with box_image_set.
box_image_set	Set this thread’s project’s image (049): a shell script run as root on the standard image of its kinds, so the project’s boxes start with what it needs, such as a language’s toolchain.
box_image_try	Try a project image’s script as a build runs it (049), before box_image_set: in a new box of the kind’s current standard image, on a backend that builds images, the script runs as root with BB_BOX_IMAGE, BB_BOX_KIND, BB_BOX_BACKEND and BB_BOX_ARCH, then the check as the box user, then finalizing, which empties /tmp, /var/tmp and the caches.
box_job	Get the exit code and output of a box_run command that went on in its box as a job: one still running after 270 s, or when BB Box restarted, as for an update.
box_list	List this thread’s boxes and the kinds of box you can acquire.
box_record	Record a desktop box’s screen, to show your work in the thread as a short video (059).
box_release	Release a box you no longer need.
box_resize	Change a box’s size: its cores, memory or disk, by a preset or by parts, and its GPU, where its backend can.
box_restore	Restore a box to one of its snapshots when an attempt went wrong, rather than undoing it by hand.
box_run	Run a shell script in a box, as its user, and wait for it.
box_snapshot	Take a snapshot of a box: its disk, or with memory its disk and running apps where its backend can, so you can restore it if a change goes wrong.
box_snapshot_delete	Delete a snapshot of a box that an agent of this thread took and nobody kept.
box_snapshots	List the snapshots of a box, or of every box of this thread and the machine it runs on: per box, what its snapshots hold or that it has none; per snapshot, its name, ID, when, by whom, what it holds, its size, its price, whether it is kept, and where it is.
box_start	Start a stopped box again, with its disk as it was, and wait up to 20 s more while it reconnects as a BB machine.
box_status	Show one box: its state, kind, address, whether it is a BB machine, and how long it took to start and to become one.
box_stop	Stop a box to free its memory and CPU while keeping its disk.
box_thread	Start a child thread on a box, as a BB machine, in this thread’s project, with the task as its prompt.
computer_act	Click, double-click, right-click, scroll or set a value on a desktop or device box: on an element by its reference from computer_observe, or at x and y in your latest image of a window or the desktop.
computer_launch	Open an app on a desktop box by its name or path, or open a URL in the box’s browser.
computer_observe	Read a window on a desktop or device box as text: its accessibility tree, one element per line, with a reference such as e12 for each element you can act on, and on a device the element’s identifier where the app gives one.
computer_screenshot	Take an image of a window on a desktop box, or of the whole desktop, or of a device box’s screen.
computer_take_control	Take control of a desktop box’s desktop, or a device box’s device, back from a person who took it in BB, so the computer tools work again.
computer_type	Type text, or press a key or chord such as Escape, Return, Tab, ctrl+a or ctrl+shift+t, into an element by its reference, or into the focused element of a window.
computer_windows	List the windows on a desktop box’s screen, with the window IDs the other computer tools take.

box_acquire

Get a fresh box for this thread: a Linux or macOS computer for commands, builds, tests, servers or Docker, away from the thread’s own machine. It starts in seconds and stays until you release it or the thread ends.

Parameter	Type	Required	Description
kind	string	no	The kind, from box_list: linux, linux-desktop and the rest, or linux-docker for a linux box that must have Docker. Defaults to the user’s default kind.
size	string	no	small, medium or large. Defaults to the kind’s default preset; box_list shows each kind’s sizes.
cores	integer	no	Cores, for a custom size. Parts you leave out come from the preset.
memory_gib	integer	no	Memory in GiB, for a custom size: at least 2 for a Linux box.
disk_gib	integer	no	Root disk in GiB, for a custom size. Presets keep the kind’s own disk.
wait	boolean	no	Wait until the box answers, and up to 20 s more while it becomes a BB machine. Defaults to true.
on	string or array of string	no	Where the box goes, from box_list: a provider such as Docker on devpc, or a list of them for the first with room among them. Leave it out for the first with room.
image	string	no	The image the box starts from, from box_list: a project’s image by name, or “standard” for bb-box’s own. Leave it out for this project’s image for the kind, or the standard one.
device	string	no	For an ios or android box, the device it boots, from box_list, such as iPhone 18 Pro or Pixel Fold. Defaults to the kind’s first.
slim	boolean	no	For an ios box: its iOS Simulator runs without the services a device box does not need, such as widgets, Siri, iCloud, Health and Wallet, for about a quarter of the memory and almost no idle CPU. Defaults to true. Set false for an app that uses one of them.
gpu	none, shared, whole	no	The box’s GPU: none; shared, one card other boxes may share; or whole, a card to itself, on which bb-box places no other box, where its provider has whole cards on. Left out: the kind’s GPU when not asked for, which box_list gives; shared for linux-desktop and none for linux unless the user changed it. Only linux and linux-desktop take one, where a provider has cards; box_list says which do, the room on each card, and whether whole cards are on.
dedicated	boolean	no	For an ios box: a macOS VM of its own, with sudo, brew install, its own cores, ports and desktop, instead of a macOS user on a shared host. It takes one of the Mac’s two VMs, so it may wait for one. Without it, an ios box is shared where a shared iOS host has room, and a VM of its own otherwise. Give it with a size, cores, memory_gib or disk_gib: where a shared iOS host is set up, a size without it is refused.
shared	boolean	no	For an ios box: a macOS user on a Mac or VM shared with other ios boxes, ready in seconds, with its own device, files and keychain, but no sudo, Homebrew installs or app windows; an ios box is one already where a shared iOS host has room, and shared: true refuses rather than give it a VM of its own. For a macos box: a macOS user on a Mac or VM shared with ios boxes, ready in seconds, for commands, builds and the iOS Simulator, with its own files and keychain, but no sudo, Homebrew installs, desktop or app windows, and cores, ports and processes shared. Only where a shared iOS host is set up; box_list says. Without it a macos box is a VM of its own, which takes one of the Mac’s two VMs.

box_copy

Copy one file, of any size, between this thread’s workspace or its storage folder ($BB_THREAD_STORAGE) and a box. Every copy’s size and SHA-256 are checked at both ends, and one that does not match fails. A file over 16 MiB streams through BB Box’s worker on the thread’s machine. A relative path on this side is under the workspace; a path under the storage folder, absolute or starting with $BB_THREAD_STORAGE/, is there. A relative box path, or one starting with ~/, is under the box user’s home. Tar a directory first.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
direction	to_box, from_box	yes	to_box copies from the workspace or storage folder into the box; from_box copies from the box into the workspace or storage folder.
from	string	yes	The file to copy.
to	string	no	Where to put it. Defaults to the file’s own name, in the box user’s home or in the workspace’s directory.

box_device

Control an ios or android box’s device, as the panel’s toolbar does: Home, Back and Recents, lock and wake, rotate, fold, appearance, text size, location, status bar, battery, network, install an app, switch device, and reboot. Each action takes only its own parameters. The answer says what happened, how long it took, and the device’s state after it. While a person has control of the box, every action but install is refused, and so is an install that would replace an installed app. Screenshots, taps and URLs are the computer tools’.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
action	home, back, recents, lock, wake, rotate, posture, appearance, text_size, location, status_bar, battery, network, install, switch, reboot	yes	What to do. iOS has no back, recents, posture or network; only foldables take posture. box_status lists the device’s actions.
turn	left, right	no	rotate: a quarter turn from where the device is, the way the toolbar’s arrows point.
to	string or number	no	rotate: upright, turned-left, upside-down, turned-right. posture: closed, half-open, open. appearance: light, dark. text_size: larger, smaller, default, or a step of the platform: an iOS content size from extra-small to accessibility-extra-extra-extra-large, such as accessibility-large, or an Android font scale such as 1.3. status_bar: clean, live. network: online, offline, slow.
latitude	number	no	location: with longitude, from -90 to 90. Leave both out to clear the simulated location.
longitude	number	no	location: with latitude, from -180 to 180.
level	number	no	battery: with charging, a whole number from 0 to 100. Leave both out for the device’s own. On iOS only the status bar’s picture changes.
charging	boolean	no	battery: with level, whether it charges.
paths	array of string	no	install: in the box, as box_copy puts them there. On iOS one .app folder built for the simulator; on Android one .apk, or the split APKs of one app.
device	string	no	switch: a model this box offers, from box_list, such as iPhone 17. Each model keeps its own apps and data.

box_enrol

Add a ready box that could not join BB, or is not a BB machine, to BB again, so threads can run on it. BB’s installer runs in it as its user. It does not restart, stop or resize the box. It stops only the BB daemon of a machine that BB no longer has. A box whose machine BB still has is left as it is, and recorded as a BB machine.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.

box_fork

Make a new box of this thread from a snapshot of one of its boxes, or from the box as it is now, to try two things from one set-up state. The fork gets the snapshot’s files and its own identity: a new BB machine, address and keys, so it never acts as its source. Everything else on the disk comes with it, project secrets and signed-in browsers too. It starts fresh, with no apps running, at the snapshot’s size.

Parameter	Type	Required	Description
box	string	yes	The box to fork, by its name or ID: one of this thread’s boxes, not one it runs on.
snapshot	string	no	The snapshot’s name or ID, from box_snapshots. Left out, bb-box takes one of the box now, named “for a fork”, and forks that.
name	string	no	The new box’s name: a lower-case letter first, then letters, digits and “-”, up to 32 characters, not ending in “-”. One is made up when left out.
gpu	none, shared, whole	no	The fork’s GPU. none, the default, starts it with no card. shared places a shared card, and whole a card to itself where this install has whole cards on, before anything is copied; only a desktop can take either.

box_grant

Grant a program in a macOS box macOS permissions, such as Accessibility or Screen Recording, with no prompt. On Tart’s macOS boxes, and on Macs of the SSH host pool with SIP off; Linux and Windows boxes refuse. The program must be the box user’s, or an app in /Applications, and signed: sign it with the box’s identity, codesign -s “bb-box local code signing”, so a rebuild keeps the grant. Commands box_run runs have the microphone and camera already.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
program	string	yes	The app’s path, such as /Users/bbx-3/Applications/Probe.app, or a program’s full path in the box.
permissions	array of accessibility, screen-recording, input-monitoring, microphone, camera, system-audio, automation	yes	The permissions to grant.
target	string	no	For automation: the bundle ID of the app the program sends Apple events to, such as com.apple.finder.

box_image_draft

Draft a project image’s script (049) from the project’s own files, for you to finish, try with box_image_try and register with box_image_set. Read these from the repository and pass each one’s path and text: .devcontainer/devcontainer.json (or .devcontainer.json, or one in a subfolder of .devcontainer) and the Dockerfile it names; a Dockerfile; .tool-versions; mise.toml; .nvmrc or .node-version; .python-version; rust-toolchain.toml or rust-toolchain; flake.nix with flake.lock, or shell.nix. It returns a script for .bb/box-setup.sh, each part commented with the file it came from, a check for .bb/box-check.sh, and what it could not translate. The script runs as root on BB Box’s standard Ubuntu 24.04 image, with no checkout of the repository: what needs one, such as npm ci, stays in the thread. With box, it also reads what a Linux box of this thread has that its image had not, for a box someone set up by hand: packages, package sources, npm globals, mise’s tools, the Rust toolchain, uv tools, and what is new in /usr/local, /opt and ~/.local/bin.

Parameter	Type	Required	Description
files	array of object	yes	The project’s files the draft reads. An empty list gives a template.
box	string	no	A Linux box of this thread to read what was set up in it by hand, by name or ID.

box_image_set

Set this thread’s project’s image (049): a shell script run as root on the standard image of its kinds, so the project’s boxes start with what it needs, such as a language’s toolchain. Pass the files’ text: commit them in the repository as .bb/box-setup.sh and .bb/box-check.sh, and give their paths. The script runs in a booted box with the network, on any Linux backend, and reads BB_BOX_IMAGE, BB_BOX_KIND, BB_BOX_BACKEND and BB_BOX_ARCH where it must differ. It gets none of the project’s secrets. Setting it changes every new box of those kinds in the project, so ask the user first unless they asked for it.

Parameter	Type	Required	Description
name	string	yes	The image’s name, such as “rust-tools”: a lower-case letter first, then letters, digits and “-”.
kinds	array of string	yes	The Linux kinds it builds on, such as [“linux”, “linux-desktop-cpu”]. box_list shows the kinds.
script	string	yes	The script’s text, run as root.
check	string	no	A check’s text, run as the box user after the script, such as cargo --version: the image fails when it fails.
bake	auto, always, never	no	auto (the default) builds it ahead once two boxes started within 14 days; always keeps a build; never runs the script at each start.
script_path	string	no	The script’s path in the repository, such as .bb/box-setup.sh.
check_path	string	no	The check’s path in the repository, such as .bb/box-check.sh.

box_image_try

Try a project image’s script as a build runs it (049), before box_image_set: in a new box of the kind’s current standard image, on a backend that builds images, the script runs as root with BB_BOX_IMAGE, BB_BOX_KIND, BB_BOX_BACKEND and BB_BOX_ARCH, then the check as the box user, then finalizing, which empties /tmp, /var/tmp and the caches. A box of the build then runs the check again and the kind’s own check. It returns each check’s result, the time and the build’s last 200 lines, and removes the boxes and the build, unless keep is true, when a box of it stays for you to look inside. Each try starts from a clean standard image, as a build does, so fix the script and try again until every check passes. It can take as long as the script, plus about a minute.

Parameter	Type	Required	Description
kind	string	yes	The Linux kind to try it on, such as “linux”. box_list shows the kinds.
script	string	yes	The script’s text, run as root, as in .bb/box-setup.sh.
check	string	no	The check’s text, run as the box user, as in .bb/box-check.sh: it should fail when the script did not do its work.
name	string	no	The image’s name, which the script reads as BB_BOX_IMAGE. “try” when left out.
on	string	no	A backend’s ID from box_list, such as proxmox-lxc. The first that builds the kind’s images when left out.
keep	boolean	no	Keep a box of what it built, to look inside with box_run: of the build, or of the build box as a failing script left it. It is this thread’s box until you release it.

box_job

Get the exit code and output of a box_run command that went on in its box as a job: one still running after 270 s, or when BB Box restarted, as for an update. box_run’s answer names the job.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
job	string	yes	The job’s ID, from box_run’s error.

box_list

List this thread’s boxes and the kinds of box you can acquire.

It takes no parameters.

box_record

Record a desktop box’s screen, to show your work in the thread as a short video (059). start begins recording and returns at once: then do what the video should show, with the computer tools and box_run. While recording, step captions what you test next (“It keeps the draft when saving fails”), and result gives its outcome with passed true or false (“Draft kept; no error toast”); both are burned into the clip and listed as chapters, at the top of the video, or at the bottom with position “bottom” at start, which suits a check near the top of the screen, such as a menu bar or a window’s title. A step whose captions would cover more of what changes during it than the other side would has them on the other side. A cut still stretch shows “▶▶ 18 s later” in a corner, and a faster clip its speed. The clip draws a larger pointer that glides to each click, a ring where each click lands, a pill for each shortcut or special key and one “Typing” pill for typing into an element by its ref, never the text typed; nothing shows for a password, or for typing whose field BB Box cannot tell, as into a window, single keys, or a person’s typing. It zooms in gently where you click, drag and type by ref, and shows the screen inset on a plain background (073): zoom false or frame false at stop turns either off. stop ends it and makes the clip in this thread’s storage folder, with a page that plays it in the thread. Idle stretches, where nothing moves but a blinking caret or a clock, are cut to half a second unless stills is “keep”, so do not pause between steps for the viewer. A recording ends by itself after 5 minutes, and stop still gives its clip. Linux desktops, on X11 and Wayland, macOS and Windows desktops. On an ios or android box it records the device’s own screen, not the host’s: the device in its frame, turning as it turns, with touches shown as circles and the captions in a band above or below it; a stretch the device or its server lost is captioned in the clip.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
action	start, step, result, stop	yes	start begins recording; step captions the next check; result gives its outcome; stop ends it and makes the clip.
text	string	no	With step: what you check next. With result: what you saw. One short line each.
passed	boolean	no	With result: whether the check passed.
position	top, bottom	no	With start: where step captions usually go. Defaults to “top”; “bottom” when what you check is near the top of the screen. A step moves its captions to the other side when theirs would cover more of what it changes.
speed	number or number or number	no	With stop: the clip plays this many times faster. Defaults to 1.
stills	cut, keep	no	With stop: cut each idle stretch to half a second (the default), or keep it.
zoom	boolean	no	With stop: zoom in gently where you click, drag and type, and out between (the default), or false for the whole screen throughout.
frame	boolean	no	With stop: show the screen inset on a plain background with rounded corners (the default), or false for the screen edge to edge.

box_release

Release a box you no longer need. Its disk and everything on it are deleted. Copy out anything you want to keep first.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.

box_resize

Change a box’s size: its cores, memory or disk, by a preset or by parts, and its GPU, where its backend can. Parts left out stay as they are, and the disk only grows. A change that needs a restart is made only with restart true, and a GPU change and cores or memory in the same call share one restart. check true only says what would change.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
size	string	no	small, medium or large, from box_list: the preset’s cores and memory. The disk stays as it is.
cores	integer	no	Cores. Left out, the box keeps its own.
memory_gib	integer	no	Memory in GiB: at least 2 for a Linux box. Left out, the box keeps its own.
disk_gib	integer	no	Root disk in GiB, which only grows. Left out, the box keeps its own.
gpu	none, shared, whole	no	The box’s GPU: none, shared, one card other boxes may share, or whole, its card to itself, where this install has whole cards on. Left out, the GPU stays as it is. Shared and whole switch live where the box’s card allows; adding a GPU, removing it or moving the box to another card restarts it on Proxmox LXC and cannot be done on Docker. A box with no NVIDIA userspace for the host’s driver gets it installed in the same restart, about 60 s more, and a linux-desktop-cpu box becomes a GPU desktop, linux-desktop. A stopped box stays stopped. Call with check true first: the answer says what would change, and a resize whose plan changed since that check is refused, saying what changed. box_list says what each provider can change.
restart	boolean	no	Allow a restart, which ends everything running in the box, where a part needs one. Defaults to false: bb-box then changes nothing and says so.
check	boolean	no	Only say what the resize would change, how, and whether there is room. Defaults to false.

box_restore

Restore a box to one of its snapshots when an attempt went wrong, rather than undoing it by hand. Everything on its disk since then goes, and the box restarts, so every process in it ends. BB Box always takes an undo first, a snapshot of the box once stopped. While Settings turn agents’ restores off, this opens a form for a yes given in the thread and answers at once: then end your turn. A box this thread or another runs on, or one of the person’s machines, is restored only through such a form, and only where BB Box allows it; this says so when it does not.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
snapshot	string	yes	The snapshot’s name or ID, from box_snapshots.
replace	boolean	yes	Must be true: you mean to replace the box’s disk with the snapshot’s.
request	string	no	An ID you make, so a repeat of this call is the same restore, not a second one.

box_run

Run a shell script in a box, as its user, and wait for it. You get the exit code, stdout and stderr. box_status says whether the box user can use sudo.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
command	string	yes	A shell script, run by the box user’s shell (bash on Linux, PowerShell 7 on Windows).
cwd	string	no	Directory to run in. Defaults to the box user’s home.
timeout_seconds	integer	no	Seconds before the command, and everything it started, is stopped. Defaults to 600. box_run returns when the script’s shell exits, or after 270 s, as BB ends a tool call at about five minutes: a command still running then goes on as a job for box_job, except a Windows one in the desktop session, which is stopped then. Start a server in the background with its output redirected, so it keeps working after that: nohup <cmd> >server.log 2>&1 &, or on Windows Start-Detached { <cmd> } server.log
stdin	string	no	Text to send to the command’s standard input.
desktop	boolean	no	Run in the box’s desktop session so it can open windows. Desktop boxes only.

box_snapshot

Take a snapshot of a box: its disk, or with memory its disk and running apps where its backend can, so you can restore it if a change goes wrong. Take one before a change you cannot undo, such as a migration, an upgrade, a deletion or a change to the system, and when the person asks; never on a schedule of your own.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
name	string	no	Its name, at most 60 characters, unique on the box, such as “before the migration”. The time, such as Thu 10:42, when left out.
note	string	no	A note, at most 1 KB: what the box held, or why you took it.
memory	boolean	no	Hold the box’s memory too, so a restore brings back its running apps: only where its backend can, and slower. Defaults to false where its backend can leave it out, and to true where every snapshot holds it, as on a macOS box on Tart, where false is refused.

box_snapshot_delete

Delete a snapshot of a box that an agent of this thread took and nobody kept. A kept snapshot, the person’s and another thread’s are the person’s to delete.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
snapshot	string	yes	The snapshot’s name or ID, from box_snapshots.

box_snapshots

List the snapshots of a box, or of every box of this thread and the machine it runs on: per box, what its snapshots hold or that it has none; per snapshot, its name, ID, when, by whom, what it holds, its size, its price, whether it is kept, and where it is.

Parameter	Type	Required	Description
box	string	no	The box’s name or ID. Every box of this thread, and the machine it runs on, when left out.

box_start

Start a stopped box again, with its disk as it was, and wait up to 20 s more while it reconnects as a BB machine.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.

box_status

Show one box: its state, kind, address, whether it is a BB machine, and how long it took to start and to become one.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.

box_stop

Stop a box to free its memory and CPU while keeping its disk. box_start starts it again.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.

box_thread

Start a child thread on a box, as a BB machine, in this thread’s project, with the task as its prompt. It works in a worktree of the project’s repository on the box, cloned there once, and is archived with this thread.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
prompt	string	yes	The child thread’s first message: the task it should do on the box.
title	string	no	A title for the child thread. Defaults to “On box <name>”.

computer_act

Click, double-click, right-click, scroll or set a value on a desktop or device box: on an element by its reference from computer_observe, or at x and y in your latest image of a window or the desktop. Or press a key or chord with action key and keys, as computer_type does. On a device box a click is a tap, right_click a long press of 1.2 s, scroll a swipe within the list that holds the element, which stops at the list’s end, set_value sets text fields only, and drag drags from ref or x and y to to_x and to_y, as for a slider. Says what changed in the window, which menus or popups opened or closed, and on the desktop which windows opened, closed or changed.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
action	click, double_click, right_click, scroll, set_value, key, drag	yes	What to do: click, double_click or right_click on ref or at x and y; scroll in direction; set_value to value; key to press keys; drag, on a device box, to to_x and to_y.
ref	string	no	The element’s reference from computer_observe, such as e12.
window	integer or string or string	no	With x and y: the window whose image they are pixels of, or “desktop”. For key without ref: the window that gets the key, or “desktop” for whatever has the focus.
x	integer	no	Pixels from the left of your latest image of the window.
y	integer	no	Pixels from the top of your latest image of the window.
value	string	no	For set_value: the value, such as a field’s text or a slider’s number.
direction	up, down, left, right	no	For scroll.
amount	integer	no	For scroll: how many steps. Defaults to 3.
keys	string	no	For key: the key or chord, such as Escape or ctrl+a.
to_x	integer	no	For drag: where to drag to, in pixels from the left of your latest image of the device.
to_y	integer	no	For drag: where to drag to, in pixels from the top of your latest image of the device.

computer_launch

Open an app on a desktop box by its name or path, or open a URL in the box’s browser. Says which windows it opened; for a URL, waits up to 10 s for the page’s title. On a device box, open an app by its bundle ID or package, such as com.apple.Preferences or com.android.settings, or its name, such as Settings, or open a URL, which the device’s own handler takes.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
app	string	no	The app’s name, such as firefox or mousepad, or the path of its program. On a device, its bundle ID or package, or its name.
url	string	no	A URL to open in the default browser.

computer_observe

Read a window on a desktop or device box as text: its accessibility tree, one element per line, with a reference such as e12 for each element you can act on, and on a device the element’s identifier where the app gives one. Pass query to see only matching elements, or screenshot for an image of the window as well.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
window	integer	yes	The window’s ID, from computer_windows or computer_launch.
query	string	no	Show only elements whose line contains this text, ignoring case, with their ancestors.
limit	integer	no	How many lines of the tree to show. Defaults to 300.
screenshot	boolean	no	Also return an image of the window. Defaults to false.

computer_screenshot

Take an image of a window on a desktop box, or of the whole desktop, or of a device box’s screen. Use it when appearance matters or the tree does not show what you need. Pixels you give computer_act are pixels of your latest image of that window, which may be smaller than the window or screen: the result says the scale.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
window	integer or string or string	no	The window’s ID, or “desktop” for the whole screen. Defaults to “desktop”.

computer_take_control

Take control of a desktop box’s desktop, or a device box’s device, back from a person who took it in BB, so the computer tools work again. Keys they held down are let go, and the desktop gets back its size. Use it when the person asks you to continue, or when the thread makes clear they have finished; otherwise wait for them to hand control back.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.

computer_type

Type text, or press a key or chord such as Escape, Return, Tab, ctrl+a or ctrl+shift+t, into an element by its reference, or into the focused element of a window. On a device box, text a US keyboard has goes as keys and the rest as one paste; cmd is the Command key on iOS, and ctrl+a selects all on Android, where ctrl+v pastes nothing in Chrome: send the text instead. On a device, text into an element waits after the tap that focuses it, up to 5 s, until a field has the focus and its keyboard shows. For an open menu or popup, give its window ID, or window “desktop” for whatever has the focus. Says what the element reads afterwards.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.
text	string	no	Characters to type.
keys	string	no	One key or chord to press, such as Escape, Return, Tab, ctrl+a or ctrl+shift+t: modifiers and a key joined by “+”. Write ctrl++ for ctrl and the + key.
ref	string	no	The element’s reference from computer_observe, such as e12.
window	integer or string or string	no	Without ref: the window whose focused element gets the input, a menu or popup by its window ID, or “desktop” for whatever has the focus.

computer_windows

List the windows on a desktop box’s screen, with the window IDs the other computer tools take. A device box has one, window 1, the device’s screen.

Parameter	Type	Required	Description
box	string	yes	The box’s name or ID, from box_acquire or box_list.