AI engineering teams often need to compare evaluation runs without losing track of which command ran, where it ran, or what it produced. A public repository-based workflow described by Docker introduces a lightweight structure for doing that with Docker Sandboxes, YAML evaluation definitions, and machine-readable runtime evidence.
The approach is useful for teams that want more inspectable evaluation execution. It does not, however, run AI models, calculate model scores, or automatically select evaluation judges. Instead, it provides infrastructure for defining commands, selecting an executor, collecting evidence, and grouping evaluations into suites.
What changed
The workflow centers on the public SBX AI Evaluation Kit, a Docker Sandboxes mixin named AI Evaluation Kit. Its specification declares schemaVersion 1 and describes a workflow in which evaluation definitions are separated from the mechanism used to execute them.
Definitions are stored in YAML. An evaluation can select either a local executor or an SBX executor, with the latter delegating execution to a Docker Sandbox. The documented example uses a command such as python3 -c 'print("hello from sbx")'. The runner then records structured JSON evidence, including the selected executor, command, standard output, standard error, exit code, and duration.
The kit also records or generates configuration-related metadata, including a digest intended to associate an evaluation artifact with its configuration. Multiple definitions can be grouped into an evaluation suite, producing per-evaluation artifacts and an aggregated summary.
That separation is the central developer benefit. The same evaluation definition can be used for a local run or a sandboxed run, while the resulting evidence makes the execution context more visible than a prose description alone.
What the kit includes
The repository specification declares pinned dependencies for pytest 8.3.4 and PyYAML 6.0.2. It also declares pypi.org and files.pythonhosted.org as allowed package domains. These details make the repository setup more specific, although they do not establish identical environments or outcomes across machines.
The repository includes a GitHub Actions workflow that runs on pushes and pull requests targeting main. It uses ubuntu-latest and Python 3.12, installs the repository requirements, runs python run_evaluation.py, and then runs pytest.
This is evidence of automated validation for the repository’s runner and tests. It should not be interpreted as proof that Docker Sandbox execution is supported or recommended inside CI. The available material also does not document artifact retention, replay, export, long-term storage, or comparison features.
Docker Sandbox requirements and boundaries
Using the SBX executor requires the Docker Sandbox environment itself. Docker’s installation documentation lists support for macOS Sonoma 14 or later on Apple silicon, Windows 11 with a 64-bit Intel or AMD processor and Windows Hypervisor Platform, and Ubuntu 24.04 or later with supported 64-bit hardware, KVM virtualization, and membership in the kvm group.
The sbx CLI requires Docker sign-in, followed by sbx login. Docker states that using the CLI does not require Docker Desktop or Docker Engine on the host, although Linux installation can install the CLI with or without Docker Engine.
The sandbox security model matters when evaluation commands can modify files, install packages, access credentials, or use external services. Docker documents a microVM with a separate kernel, network isolation, a separate Docker Engine, workspace controls, and credential isolation. Credentials are injected through a host-side proxy rather than exposing raw credential values inside the VM. Outbound TCP traffic is proxied and governed by network policy, while direct external UDP and ICMP traffic are blocked.
These boundaries are not absolute restrictions on everything an agent can do inside the VM. Docker states that sandboxed agents have full privileges within the VM, including sudo, package installation, access to a private Docker Engine, and read-write access to configured in-sandbox resources.
Workspace selection also changes the risk and reproducibility profile. In direct workspace mode, changes are shared with the host in real time. Clone mode provides a private in-VM clone with the repository mounted read-only, while a mountless sandbox has no host workspace mount. Local stdio MCP servers run outside the VM and should therefore be treated as trusted host integrations. Shared skills can also create a trust relationship across sandboxes because they use a shared host-side store.
Why the workflow matters to engineering teams
Evaluation results are easier to investigate when the record includes the command, executor, output, errors, exit status, and duration. That information can help a team distinguish a failed evaluation command from a failed model judgment, or identify when two runs used different execution paths.
The YAML-and-executor model can also reduce duplication. Teams can keep evaluation intent in definitions while changing whether a run is local or delegated to a sandbox. Suites provide a way to run several definitions together and review both individual artifacts and an aggregate summary.
Pinned dependencies and a declared schema version improve the precision of the repository’s setup. They are not a guarantee of full reproducibility. The supplied documentation does not provide measured reproducibility results, a guarantee of identical results across machines or over time, or a complete schema for YAML definitions, JSON artifacts, suite summaries, and digests.
What you should do
- Define the objective and command. Store the evaluation definition in YAML and make the executable command explicit.
- Choose the executor deliberately. Use local execution for host runs or the SBX executor when the evaluation should run through Docker Sandboxes.
- Inspect the generated evidence. Review the executor, command, output, errors, exit code, duration, and configuration-related metadata rather than relying only on a pass or fail description.
- Use suites for related checks. Group multiple evaluation definitions when a repeatable workflow needs per-evaluation artifacts and an aggregated summary.
- Validate the repository workflow. The documented repository process uses Python 3.12, installs requirements, runs
python run_evaluation.py, and runspytest. - Review sandbox policies first. Check workspace mode, credentials, network policy, and MCP integrations before allowing an agent to interact with host or external resources.
Availability and limitations
The AI Evaluation Kit is available as the public karanverma/sbx-ai-eval-kit repository. The available sources do not establish a separately packaged or formally released distribution, a maintenance relationship with Docker, or a complete supported model, agent, API, and authentication matrix.
For now, the most accurate way to view the kit is as evaluation workflow infrastructure. It can organize commands and collect runtime evidence, but it is not documented as an automated model-scoring or model-judging system. Teams adopting it should also treat the declared package domains and root-privileged installation behavior as supply-chain considerations, without assuming that the kit provides a security guarantee.
