microsandbox backend

This backend runs Linux boxes as microVMs on one Linux host with KVM, with microsandbox’s msb. It is an option beside the Docker backend, not a replacement and not the default: keep it after Docker in the backend order, and ask for it with --on (or box_acquire’s on) for a box that needs a kernel of its own. On bare metal it makes boxes as fast as Docker, and removes them faster; under nested KVM, as in WSL 2, Docker is about twice as fast. Either way a microVM holds far more memory while idle, as it keeps its own page cache (053). It offers two kinds, from the same images as the Docker backend:

linux: Ubuntu 24.04 with the same tools as a Docker or Proxmox LXC linux box, and Docker of its own;
linux-desktop-cpu: XFCE and Chrome, drawn by the CPU, with computer use through Cua Driver.

Each box has a kernel of its own, so Docker in it needs no privilege on the host, where a Docker box with Docker is a privileged container, and Chrome runs with its own sandbox. That isolation is what this backend is for. Boxes join BB as machines, as other Linux boxes do.

The host is the machine that runs BB, another machine over SSH, or a BB machine, through BB. bb-box installs its own msb there, at the release it pins, and reaches inside a box with msb exec, so it needs no network path to the boxes. How bb-box drives a box is in the architecture.

What you need

A Linux host on x86_64 with /dev/kvm: the machine itself, or a VM with nested virtualisation on. bb-box is tested on a Windows PC’s WSL 2, where Hyper-V nests KVM, with glibc 2.28 or later.
The user who runs the boxes allowed to use /dev/kvm: on most distributions, in the kvm group. On a BB machine that is the user BB’s app runs as; over SSH, the user bb-box logs in as; on BB’s server, the user BB runs as.
curl and tar on the host, with which bb-box installs msb.
Docker on the host, to build the images, as for the Docker backend: about 28 GB free to build both. bb-box imports them from Docker into msb.
Disk for msb: about 6 GB for both images once imported, and what each box writes. A box’s disks are sparse, so they take only what it writes.
Memory for the boxes: bb-box keeps 4 GiB of the host’s memory free by default, and counts each running box’s full size, so a host for a few small boxes needs 16 GiB. A box that has joined BB holds about 0.5 GiB of the host’s memory while idle, once bb-box has had it drop its page cache (below). A host of mostly idle boxes can take more: see Fit more boxes on a host.
For a host over SSH: the user BB runs as logs in to it with a key and no prompt.

Before you rely on it:

Beta. msb is 0.7.7 and may change before 1.0. bb-box pins one release, checks its SHA-256, and moves to a new one only in a bb-box release that passes the conformance suite (see Update).
Not a security boundary. A box has its own kernel, but its user may use sudo, and bb-box does not aim at isolation.
Not in LXC. A Proxmox LXC container with /dev/kvm passed in runs boxes, but its seccomp filter stops msb’s snapshots and flat roots, which spares from forks will need. Use a VM or the machine itself.
No GPU. msb has no GPU, so there are no GPU desktops.
Enrolment. A box reaches BB’s server through msb’s network, with the profiles public,private, so BB on a private or tailnet address works. BB on the host’s localhost does not.

Steps

1. Get the host ready

On the host, as the user who runs the boxes:

ls -l /dev/kvm                 # it is there
id                             # the user is in its group, usually kvm
sudo usermod -aG kvm "$USER"   # if not, then sign in again

On a BB machine, BB’s app keeps the groups it started with. After adding its user to kvm, restart that user’s services: sudo systemctl restart user@$(id -u <user>).service. In WSL, run wsl --shutdown from Windows instead, and start the distribution again.

2. Build the images

bb-box never pulls an image. Build them with Docker on the host, from a checkout of bb-box, as for the Docker backend:

git clone <bb-box repository> && cd bb-box
setup/docker/build.sh linux desktop

bb-box imports bbbox/linux:1 and bbbox/linux-desktop:1 into msb with docker save and msb load: the linux image in about 6 s and the desktop in about 20 s, at the first box of each kind, or by its upkeep within a few minutes. It imports an image again when Docker’s has changed. A host that is also a Docker backend’s shares these images with it.

A host without Docker, such as a Mac, needs no step here: bb-box builds each image there itself, in a microVM of its own, bb-box-builder-<id>, from Docker’s docker:dind image, pinned by digest, the only image bb-box pulls. It runs setup/docker/build.sh for the host’s architecture inside, moves the image into msb, and removes the builder. The linux image took about 3 minutes on a Mac; it needs 80 GiB free while it builds, and the first box of a kind is refused, saying so, until its image is built.

On a Mac

A Mac with Apple silicon runs the arm64 kinds, linux-arm64 and linux-arm64-desktop-cpu (040), from bbbox/linux:1-arm64 and bbbox/linux-desktop:1-arm64, beside Tart’s VMs of the same kinds and after them in the backend order. msb uses the Hypervisor framework, so the Mac needs nothing installed: reach it as a BB machine, bb:<machine ID>, or over ssh:// as Tart’s backend does. bb-box keeps msb, its state and the images in its data folder on the Mac, about 2.4 GiB with the linux image and 6 GiB with both. A Mac does not take back memory a box has used, so count each box at its full size there.

3. Configure bb-box

On BB Box’s settings page in BB, set up the microsandbox backend: fill in its fields, press Check, then Save and turn on.

Field	Default	Meaning
`host`	empty	`bb:<machine ID>` for a BB machine, which bb-box reaches through BB, or `ssh://user@host`. Empty uses the machine that runs BB
`linux-image`	`bbbox/linux:1`	The image of linux boxes. Empty turns them off
`desktop-image`	`bbbox/linux-desktop:1`	The image of linux-desktop-cpu boxes. Empty turns them off
`linux-arm64-image`	`bbbox/linux:1-arm64`	The image of linux-arm64 boxes, on an arm64 host such as a Mac. Empty turns them off
`desktop-arm64-image`	`bbbox/linux-desktop:1-arm64`	The image of linux-arm64-desktop-cpu boxes, on an arm64 host. Empty turns them off
`docker`	on	Each box has a Docker daemon of its own, with its data on an ext4 disk of `msb`’s own, which a snapshot holds too. Off, boxes have no Docker
`docker-disk-gib`	64	The most each box’s Docker disk may grow to
`disk-gib`	32	The most a box’s root disk may grow to, when no disk size is asked for
`network`	`public,private`	`msb`’s network profiles for each box: `public` reaches the internet, `private` the host’s local networks and tailnet
`boxes`	20	The most bb-box microVMs on the host, running or stopped, of every bb-box install that uses it
`reserve-memory-mib`	4096	A new box is refused when the host would have less memory than this left, after the memory of the running boxes
`memory-overcommit`	1	How many times the host’s memory, less the memory kept free, the running boxes’ sizes may add up to. 1 counts each box at its full size

bb machine list gives a BB machine’s ID. From a terminal, bb box config show --json prints the configuration, and bb box config apply <file> saves it. A backend on a BB machine is:

{ "id": "microsandbox", "type": "microsandbox", "enabled": true, "fields": { "host": "bb:host_abc123" } }

Another microsandbox backend, on another host, has an ID of its own, such as microsandbox:devpc.

bb-box installs msb on the host the first time it needs it, which takes about 2 s. It keeps msb, its state and the images in a folder of its own:

Host	Folder
A BB machine	`microsandbox` in bb-box’s data folder on the machine, such as `~/.bb-machines/<BB server>/plugins/bb-box/host-data/microsandbox`
Over SSH, or BB’s server	`~/.local/share/bb-box/microsandbox`

Each box’s Docker disk is msb’s own, and goes with the box. A box an earlier bb-box made has its Docker disk in the folder’s disks instead, which also goes with it.

A box keeps the files it reads cached in its own memory, which the host cannot share. Every 3 minutes bb-box has each idle box drop that cache, so the host gets the memory back: an idle box that has joined BB holds about 0.5 GiB.

Fit more boxes on a host

bb-box counts each running box at its full size, though an idle box holds about 0.5 GiB. On a host whose boxes are mostly idle agent boxes, raise memory-overcommit to fit more. At 2, the boxes’ sizes may add up to twice the host’s memory, less the memory kept free. A 64 GiB host that keeps 4 GiB free then takes 30 boxes of 4 GiB rather than 15. Raise boxes too, which is 20 by default.

Above 1, bb-box also checks the memory the host has available now before it starts a box. A new box needs up to 2 GiB as it starts, or its size if less, on top of the memory kept free. So while the boxes are busy, a new box is refused, with the reason, until memory is free again.

If busy boxes still need more memory than the host has, Linux ends one of the boxes rather than BB’s daemon, as bb-box raises each box’s oom_score_adj. Raise the setting a little at a time, such as to 1.5, and watch the host’s memory.

On a Mac, memory a box has used is not given back, so bb-box counts each box there at its full size, whatever the setting, and the doctor warns when it is above 1. Tart’s VMs share the Mac’s memory too.

msb needs a short path for its sockets, so when the folder’s is long, msb reaches it through a link, /tmp/bb-box-msb-<uid>. bb-box never uses your own ~/.microsandbox.

Spares

A kind’s spares (032) are boxes made ahead and enrolled, so a claimed box is a BB machine at once: set a kind’s spares in the configuration. They go to the first backend in the order with room, so for microsandbox to hold them it must come before the kind’s other backends. microsandbox makes a spare as it makes any box: a new microVM, about 2 s, then BB’s enrolment, about 20 s, which the spare hides. A spare stays running while it waits, as msb stop does not keep a box’s session.

On Linux, an idle enrolled spare holds about 0.5 GiB of the host’s memory once the upkeep has had it drop its page cache, and is counted at its full size.
On a Mac, a box keeps all the memory it has used, so a spare costs about 1.5 GiB from enrolment, and up to its full size, 4 GiB for a small linux-arm64 box, for as long as it waits. Keep spares there at 0 unless the Mac has room to spare.
bb-box does not fork spares from a prepared box. On a Mac, a fork took 1.75 s against 1.8 s for a new box; on an ext4 host a fork copies the box’s disk and is slower still (053).

Snapshots

bb-box takes, restores, forks, keeps and deletes snapshots of microsandbox boxes with msb’s own snapshots (Snapshots). They live in msb’s folder on the host, apart from each box’s microVM, and outlive it.

What one holds. The box’s disk with its Docker data, or with --memory its memory and running processes too. /run is in memory, so a disk snapshot leaves it out. A take is under a second; with memory the box pauses for about a second.
A restore makes the box a new microVM from the snapshot, with the box’s ID, name, labels and identity value, so it stays the same BB machine, then deletes the old one. The undo is a snapshot of the box once stopped.
A fork boots a new microVM from the snapshot’s disk only, never its memory and never with msb fork, so nothing of its source runs in it. Its identity value is new, so its identity reset runs before its daemons start. Forks need the identity reset turned on for the backend (general.identityResetBackends), as for the other backends.
Chains. msb keeps each box’s snapshots in a chain, each taken after the one before, and refuses to delete one that others were taken after. Each holds its own data, so bb-box deletes any of them with --force, and says on each which one it was taken after.
Keep writes the keep date in the folder’s kept, as msb cannot change a snapshot’s labels. A kept snapshot stays on the host after its box’s release, at its whole size, until its date.
Only with msb 0.7.7, the release whose snapshots bb-box measured. With any other, the doctor’s Snapshots check warns and boxes take none.
Not on a box an earlier bb-box made: its Docker disk is a file of the host’s, which msb leaves out of a snapshot. Such a box says why, and works as before otherwise.
Not in LXC, as above.
On Linux hosts and Macs alike. The probes ran on both (docs/research/box-snapshots/microsandbox-unit-10.sh and microsandbox-mac.sh).

Check it worked

Run bb box doctor. For the microsandbox backend it checks:

the host answers, runs Linux on x86_64, and its cores, memory and free disk;
msb is installed at the release bb-box pins;
/dev/kvm is there and the user may use it;
each image is in msb, or in Docker for bb-box to import, and how old it is;
whether the host’s msb is the release whose snapshots bb-box measured;
whether the host is an LXC container;
the room on the host for a box at each kind’s default size.

Its notes say how much disk msb’s state and images take.

To try the backend without BB, run the conformance suite from a checkout of bb-box on the host. It acquires a box of each kind, checks it, and releases it; without BB it skips enrolment, and says so:

npm install
echo '{ "backends": [{ "id": "microsandbox", "type": "microsandbox", "enabled": true, "fields": {} }] }' > msb.json
npm run conformance -- --backend microsandbox --settings msb.json

Update

msb. A bb-box release that pins a new msb installs it the first time it needs it, beside the old one in bb-box’s folder, after checking its SHA-256. Do not run msb self update on bb-box’s msb: bb-box would put its own release back. To free the space, remove the old release’s folder, named by its version, such as 0.7.7.
Images. On a host with Docker, rebuild them with build.sh, or let the nightly run rebuild Docker’s (setup). bb-box imports each changed image into msb within a few minutes.
Images on a host without Docker, such as a Mac: the nightly run rebuilds them there itself, once set up. It builds each in a builder microVM with BB’s app, so boxes enrol without downloading it, under a tag of its own, then checks a box of the new build. Only when that passes does the build take the image’s name. It needs 80 GiB free on the volume where msb keeps its images and build files, measured through any link to another volume. With less, the image waits for the next night, and bb box doctor says so. bb box images build <name>, with the name bb box images list gives, rebuilds one now. The build runs detached in the builder, and bb-box checks on it every 10 seconds. When a build fails, its last lines in bb box images list --json include the guest’s kernel log and memory (guest:) and the builder’s last msb logs lines (msb:), as bb-box removes the builder.
Boxes made before keep running from the image they started with.

Remove

In BB, release the backend’s boxes, turn the backend off and remove it on BB Box’s settings page.

On the host, remove bb-box’s folder (see the table under step 3) and its link:

bin=<folder>/bin/msb
"$bin" ls                       # any boxes left
"$bin" rm --force <name>
rm -rf <folder> /tmp/bb-box-msb-$(id -u)

Remove the images from Docker if no Docker backend uses them: docker image rm bbbox/linux:1 bbbox/linux-desktop:1.

When it goes wrong

“has no /dev/kvm”: the host has no KVM. Use the machine itself, or turn on nested virtualisation for its VM.
“BB’s user may not use it”: add the user to the kvm group (step 1), then restart its services so it has the group.
“has no image”: build the images with Docker on the host (step 2).
Installing msb fails: the host needs curl and tar, and to reach GitHub. “SHA-256 is …, not …” means the download was not the release bb-box was tested with, so bb-box did not install it. Try again later.
A box does not join BB: check that BB’s server URL resolves and connects from the host. Its address must be one the network profiles reach: keep private for a private or tailnet address.
“belongs to another user”: someone else owns /tmp/bb-box-msb-<uid>. Remove it.
A box does not come up: <folder>/bin/msb logs bbbox-<box ID> shows its output. bb-box’s error says what it saw.