CPU, memory, storage and GPUs

Override resources at creation time. You do not need a new template:

from sandweave import Sandbox

with Sandbox(cpu=4, memory="8GiB") as env:
    print(env.run("python --version").stdout)

The same arguments work with target="lab" or a printed cluster address.

CPU sharing

from sandweave import CPU, Sandbox

with Sandbox(cpu=CPU(vcpus=2, weight=200, quota=1.5)) as env:
    print(env.info["cpu"])

Setting	Meaning
vcpus	Virtual CPU count advertised to the sandbox.
weight	Relative share of CPU time under contention; default 100.
quota	CPU-time budget measured in cores; 1.5 allows about 1.5 cores of CPU time.

gVisor enforces weights and quotas through a sampled userspace controller. These settings share the worker's eligible CPUs; they do not reserve dedicated physical cores or grant more CPUs than the host allocation supplies.

A busy sandbox can borrow CPU time left unused by other sandboxes, including peers that are still doing some work. For example, on a four-CPU worker, if one equally weighted sandbox uses one CPU, another can use the remaining three. Weights determine shares when there is enough runnable work to compete for them. Explicit quotas still limit consumption when spare capacity is available.

vcpus controls the guest's reported CPU count and gVisor's execution concurrency per address space. Separate guest processes can together use more host CPUs than this count. It is neither a sandbox-wide CPU-time ceiling nor a request to resize an application's thread pool automatically. Use quota for a CPU-time ceiling.

The controller reads each sandbox's aggregate host CPU time and runnable count; it does not scan every guest process. Each sandbox has its own asynchronous control connection, so a stalled runtime does not block other sandboxes. Demand estimates use unthrottled observation windows of about 100 ms; scheduling normally updates every 20 ms. It preserves observations across its own pauses and distinguishes brief wakeups from threads that keep waiting for CPU. Changes in demand take time to observe, and short overshoot is possible. Runtime and transport CPU are counted, but only guest execution is throttled. These are not hard host cgroup limits.

If CPU control fails, the sandbox stays alive and releases its CPU weights and quota. Broker pauses expire after 500 ms without renewal. After a sustained failure, env.info["cpu_control"] reports state="degraded" and the reason. Explicit env.pause() remains separate and does not expire.

This requires SDK 0.2.29 and its updated engine. New workers automatically update older prepared engines while retaining guest images. Existing workers and sandboxes keep their running code. Memory snapshots pinned to an older engine use compatibility accounting outside the broker's scheduling loop; their pause recovery relies on the launcher watchdog instead of the new engine lease.

Guest and runtime memory

from sandweave import Memory, Sandbox

with Sandbox(memory=Memory(guest="4GiB", runtime="1GiB")) as env:
    print(env.info["memory"])

Guest memory is the budget for sandbox application pages. Runtime memory is a separate guard for the processes implementing the sandbox. Both count toward cluster admission. Neither is the host machine's total memory.

A string such as memory="8GiB" overrides guest memory and retains the template's runtime budget. Sizes can also be supplied as integer byte counts.

A 32 GiB worker fits three sandboxes configured with 8 GiB guest memory and 512 MiB runtime memory. With 256 MiB for each budget, it fits up to 64, subject to its slot limit and other reservations. Slots do not grant additional memory.

Worker budgets are capped at visible cgroup hard limits, including narrower job steps and ancestor limits. An environment variable cannot raise that ceiling. If several workers or unrelated processes share a memory allocation, their budgets still need to fit within that shared allocation.

Experimental memory sharing

Give guest and runtime memory separate limits and smaller admission reservations:

from sandweave import Memory, Sandbox

memory = Memory(guest="16GiB", reservation="4GiB",
                runtime="4GiB", runtime_reservation="512MiB", experimental=True)
env = Sandbox(memory=memory)
print(env.info["memory"])
# Run your agent, then release its sandbox.
env.terminate()
env.close()

This sandbox sees 16 GiB and can allocate beyond 4 GiB automatically. There is no burst timer or extra call for the agent. Sandweave counts 4 GiB plus 512 MiB of runtime memory for admission; the runtime guard remains 4 GiB. It does not allocate or pin that RAM upfront. Either reservation can be set independently. The reservation is scheduling accounting, not a protected minimum of physical RAM. Four such sandboxes reserve 18 GiB on a worker with a 32 GiB budget, even though their combined guest limits are 64 GiB. Ordinary sandboxes continue to reserve their full guest and runtime budgets.

The same memory= object works with Pool, local workers, and Weave targets. Guest reservations require Sandweave 0.2.28 or newer. Runtime reservations require 0.2.38 or newer on the client, controller and workers. Older endpoints are rejected before launching a sandbox with unsupported accounting. Pool weights govern scheduling; they do not divide RAM between running sandboxes. There is no automatic ballooning, memory weighting, or reclamation of another sandbox's live private data.

This opts into possible host out-of-memory failures. Guest limits and runtime guards remain in effect, but reservations are not host RSS caps. If combined usage exceeds available RAM, the host can kill sandbox or other processes in the enclosing allocation. Sandweave does not guarantee that only a borrower is killed. Existing host swap, if any, follows the host's policy; this option does not create swap or disk backing. Leave capacity for host and worker processes.

reservation must be at least 64 MiB and no greater than guest, and requires experimental=True. runtime_reservation must be at least 32 MiB and no greater than runtime, with the same experimental opt-in. Omitting either reservation counts that component's full limit, even with experimental=True. Services use their own reservations; image-import allowances remain fully counted. With disk-backed memory, these settings change admission accounting only; the existing kernel RAM cap remains guest + runtime.

Snapshots retain both settings. On restore, an explicit Memory(...) can add, change or remove reservations without rebuilding the image or changing saved guest contents; live restores still require the same guest, runtime and disk sizes. A scalar memory= override selects full reservation, retaining the saved runtime budget.

The CLI equivalent is:

sandweave run --memory 16GiB --memory-reservation 4GiB --runtime-memory 4GiB --runtime-memory-reservation 512MiB --experimental-memory-sharing -- "python --version"

Writable disk storage

New sandboxes store writable files on the worker's disk. File contents no longer consume the guest's application-memory budget. Linux can cache disk pages in host RAM and reclaim that cache; this does not add RAM to the sandbox.

from sandweave import Sandbox, Storage

env = Sandbox()  # Disk-backed writable filesystem by default.
print(env.info["storage"])  # Mode and actual private directory on the worker.
env.terminate()
env.close()

env = Sandbox(storage=Storage(path="/data/sandbox-disks"))
env.terminate()
env.close()

Each sandbox gets a private random directory, removed when it stops. With no explicit path, this directory is under the worker's runtime workspace. The path is on the worker, including when using SSH or Weave. The same storage= option works with Pool; templates can set resources.storage.

The runtime workspace uses SANDWEAVE_LOCAL_DIR on the worker, falling back to TMPDIR and then the system temporary directory. Set SANDWEAVE_LOCAL_DIR to a disk directory before starting workers if their /tmp is backed by RAM. Storage(path=...) overrides this location for a sandbox's writable files.

The root writable layer and persistent Docker/containerd mounts use disk. /run, /tmp, /dev/shm, and other volatile mounts retain their RAM semantics. Explicit external mounts keep their existing backing and snapshot policy. Use storage="memory" to request the former RAM-backed writable filesystem.

Filesystem and memory snapshots capture writable contents. Restore creates new backing files, so clones are independent and do not need the source directory. A restore can use a different Storage(path=...). Existing snapshots retain their saved mode; older snapshots without a mode remain memory-backed. To change modes, restore a filesystem snapshot with an explicit storage= option.

Disk storage requires an updated worker/controller and engine. It does not require host sudo or delegated cgroups. Available space and I/O performance come from the chosen filesystem; Sandweave does not reserve a disk quota. This is separate from Memory(disk=...), which backs application memory as described below.

Disk-backed memory

from sandweave import Memory, Sandbox

memory = Memory(
    guest="4GiB",
    disk="16GiB",
    disk_path="/scratch/my-memory",
)
with Sandbox(memory=memory) as env:
    print(env.run("free -h").stdout)
    print(env.info["disk_memory"])

Applications see 20 GiB of guest memory and allocate it normally. They need no special calls or awareness of a second memory tier. Linux caches the backing file in RAM and reclaims its pages to disk under pressure. There is no timer for borrowing memory, and the guest does not show this storage as Linux swap.

guest is the RAM allowance; disk adds to the guest's total page budget. runtime defaults to 512 MiB and retains its separate guard. The kernel caps the sandbox's host memory at guest + runtime, including runtime processes and file cache. The exact portion resident as guest pages depends on runtime overhead. Weave reserves that RAM amount, rather than guest + disk + runtime. Worker control processes and launch supervision remain outside the sandbox cap; leave capacity for those and other host processes when setting worker budgets.

disk_path is required and names an absolute directory on the worker. Sandweave creates it if needed, then creates a private random subdirectory per sandbox. It does not derive this path from SANDWEAVE_HOME, a cache, or /tmp. env.info["disk_memory"] reports the actual directory and host limit in bytes. The backing files have no filenames and are not mounted into the guest. Their storage is released when the runtime exits, including a killed runtime. Normal cleanup also removes the empty private directory.

The backing file covers the entire guest page budget, including pages cached in RAM. Allow space for 20 GiB, plus filesystem overhead, in this example. Storage grows as pages are populated. This space is separate from disk space used by images, checkpoints and caches. A full filesystem can cause allocation failures, just as reaching the guest memory budget can.

This mode requires:

• gVisor and a disk filesystem supporting unnamed temporary files and allocation/ hole punching, such as local ext4 or XFS. RAM filesystems cannot provide overflow.
• A delegated cgroup-v2 memory controller, or a Slurm allocation whose steps enforce memory limits. Sandweave creates a separate capped step automatically on Slurm and preserves the worker's CPU pool. It does not enable host swap, use sudo, or change existing allocations.

Hosts without either memory-control path cannot enforce this mode. Ordinary RAM-only sandboxes retain their existing requirements. Native Apptainer and shared CUDA MPS partitions do not support disk memory. Disk memory extends guest CPU memory, not GPU VRAM; pinned memory still needs RAM.

The same memory=Memory(...) argument works with pools. Each sandbox gets its own backing file, even when pools share image caches. All eligible workers must provide the selected path and memory-control support.

Memory snapshots preserve guest contents. Restores create fresh backing files; to restore on another disk, pass the same Memory(...) sizes with a different disk_path. Filesystem caches retain files but restart application processes.

Disk paging is workload-dependent. On the measured scratch NVMe, a workload with 99% of accesses in a warmed 2 GiB region averaged about 5.2 times the RAM baseline; random access across 20 GiB cost about 260 times as much. See the profiling report.

Use a GPU

from sandweave import Sandbox

with Sandbox(template="cuda", gpu=True) as env:
    print(env.run("nvidia-smi").stdout)

Use a model name to select among eligible devices:

with Sandbox(template="cuda", gpu="L40S") as env:
    print(env.info["gpus"])

The worker must already have access to that GPU. Weave currently reserves whole GPUs for cluster admission; fractional GPU admission is not implemented.

In the 0.2.20rc7 preview, accept any of several models with GPU(model=("H100", "A100")). Import GPU from sandweave and pass it as the gpu argument. Model matching is case-insensitive and accepts substrings of the device name. This still requests one GPU.

Inspect configuration

from pprint import pprint
from sandweave import Sandbox

with Sandbox() as env:
    pprint(env.info)

The summary includes identity, state, template, worker hostname, CPU and memory settings, selected GPU devices, and VNC details where applicable. Resource settings are budgets, not current usage. The dashboard shows live measurements separately.

Choose a runtime

gVisor is the default. For code using the host kernel directly:

with Sandbox(runtime="apptainer") as env:
    print(env.run("python --version").stdout)

Native Apptainer supports commands, files, GPU exposure, pause/resume, and filesystem caches. It uses the host network and does not support CPU weights, CPU quotas, offline networking, or memory snapshots.