fix: export large SmolVM artifacts through native pipes

docs/sandbox-environment.md

@@ -105,6 +105,8 @@ Same-path local sources are relocated under `/sources`, updating `BACKINTEL_SQLI
 `BACKINTEL_FILES_DIR` and `BACKINTEL_PARQUET_DIR`. The original source remains unchanged.
 Artifact reads and exports reuse the bounded tar-stream validation: regular files only,
 no links/traversal/special files, supported suffixes, at most 256 files and the byte budget.
+Binary downloads use native pipes rather than SmolVM's 11 MiB capped command-output
+stream; see the [export fix and real-VM checks](smolvm-artifact-export.md).
 
 ## Verification
 

docs/smolvm-artifact-export.md

@@ -0,0 +1,66 @@
+# SmolVM artifact export fix — INT-498
+
+Large binary reads and report exports now use SmolVM's native stdin/stdout pipes
+(`-i`, without a pseudo-terminal) rather than its capped `--stream` command-output
+path. Stdin is explicitly closed for downloads. The trusted guest wrapper still
+applies the same analyst UID, read-only root and process limits. Existing archive
+validation, file/byte limits, deadlines and partial-output cleanup are unchanged.
+Ordinary shell/Python command output retains its existing caps.
+
+## Confirmed cause
+
+SmolVM 1.31.1 limits cumulative streaming-exec output to **11 MiB**. BackIntel permits
+artifact payloads up to **128 MiB**, so valid report bundles could be truncated before
+Python finished reading their tar archive. Extending a timeout cannot fix that cap.
+
+A minimal real-VM reproduction wrote a 25 MiB artifact. The original exporter failed
+with `ReadError: unexpected end of data` in 0.233 seconds and removed partial output.
+Capturing the underlying command separately returned exit code 1, 11,536,384 bytes
+(the cap plus one frame), and this stderr:
+
+```text
+streaming output exceeded 11534336 byte cap; exec terminated (output truncated)
+```
+
+The same native pipe path transferred a 26,224,640-byte tar containing a 25 MiB binary
+payload intact in 0.470 seconds. This discriminates the transport cap from timeout,
+tar parsing and artifact quota failures.
+
+Pinned upstream source inspected at `e6ec9b5338145c92efb8073c207f8f3c43812573`
+(tag `v1.31.1`):
+
+- [Streaming-exec cap and collection](https://github.com/smol-machines/smolvm/blob/e6ec9b5338145c92efb8073c207f8f3c43812573/src/agent/client.rs#L3840).
+- [Native pipe versus streaming CLI dispatch](https://github.com/smol-machines/smolvm/blob/e6ec9b5338145c92efb8073c207f8f3c43812573/src/cli/machine.rs#L4070).
+
+The existing pipe mode keeps tar execution inside the constrained guest wrapper.
+It requires no guest archive staging, host mount, network, extra dependency or
+privileged direct-file copy. Both `read_bytes` and `export_artifacts` use it.
+
+## Direct verification
+
+| Check | Observed result |
+| --- | --- |
+| Simulated capped CLI, public read/export methods | Both failed before the fix; both pass after it, including binary bytes and EOF on stdin |
+| Real SmolVM 25 MiB binary | Read and exported bytes match SHA-256 |
+| Real SmolVM three 8 MiB files | Combined 24 MiB export preserves every byte |
+| Normal PNG/SVG/offline HTML/XLSX output | Existing real-VM acceptance passes |
+| Source/root writes, symlink export and bounded single-file read | Existing denial checks pass; rejected export is removed |
+| Sparse 65 MiB artifact under a 64 MiB test profile | Explicit quota rejection; partial host directory removed |
+| Retained normal service report | 10 files / 4,947,856 bytes; all source manifest hashes match |
+| Retained three-report combined payload | 32 files / 19,769,798 bytes; all source manifest hashes match |
+
+The retained reports are the original INT-498 design-review artifacts, copied into a
+read-only mounted source and exported without rerendering or changing their contents.
+Local reproduction receipts and exported bundles are in the canonical checkout at
+`data/output/int498-export-fix-20261114/`. No model calls were made.
+
+Run the portable regression and opt-in real-VM checks:
+
+```sh
+uv run pytest tests/test_analysis_sandbox.py -k binary_transfer_outgrows -q
+BACKINTEL_SMOLVM_TESTS=1 uv run pytest tests/test_analysis_sandbox.py \
+  -k 'real_large_binary_read_and_aggregate_export or real_persistent_workspace_boundary_and_export' -q
+```
+
+This fixes transport. It does not relabel the earlier taxi attempt as successful or
+recover its deleted HTML/workbook output; those historical records remain unchanged.

src/backintel/analysis_sandbox.py

@@ -141,7 +141,9 @@ def _checked(self, arguments, *, timeout=30):
             raise RuntimeError(f"SmolVM operation failed: {result.stderr[:1000]}")
         return result.stdout
 
-    def _exec_command(self, arguments, *, input_data=False, timeout=30):
+    def _exec_command(self, arguments, *, pipe_io=False, timeout=30):
+        # Native pipes (-i without -t) preserve binary I/O without SmolVM 1.31.1's
+        # 11 MiB --stream output cap. Callers still enforce their own byte/deadline limits.
         return [
             self.engine,
             "machine",
@@ -154,7 +156,7 @@ def _exec_command(self, arguments, *, input_data=False, timeout=30):
             "/workspace",
             "--timeout",
             f"{max(1, int(timeout))}s",
-            *(["-i"] if input_data else ["--stream"]),
+            *(["-i"] if pipe_io else ["--stream"]),
             "--",
             "python",
             "-I",
@@ -325,7 +327,7 @@ def _execute(self, arguments: list[str], *, input_data: bytes | None = None) ->
             timeout = min(self.profile.command_timeout_s, remaining)
             started = time.monotonic()
             proc = subprocess.Popen(
-                self._exec_command(arguments, input_data=input_data is not None, timeout=timeout),
+                self._exec_command(arguments, pipe_io=input_data is not None, timeout=timeout),
                 stdout=subprocess.PIPE,
                 stderr=subprocess.PIPE,
                 stdin=subprocess.PIPE if input_data is not None else None,
@@ -438,9 +440,11 @@ def read_bytes(self, path: str, max_bytes: int = 2 * 1024 * 1024) -> bytes:
             source_path = PurePosixPath(path)
             proc = subprocess.Popen(
                 self._exec_command(
-                    ["tar", "-C", str(source_path.parent), "-cf", "-", "--", source_path.name]
+                    ["tar", "-C", str(source_path.parent), "-cf", "-", "--", source_path.name],
+                    pipe_io=True,
                 ),
                 env=host_environment(),
+                stdin=subprocess.DEVNULL,
                 stdout=subprocess.PIPE,
                 stderr=subprocess.DEVNULL,
             )
@@ -504,8 +508,9 @@ def export_artifacts(self, destination: str | Path) -> list[str]:
             output.mkdir(parents=True, exist_ok=True)
             output = output.resolve()
             proc = subprocess.Popen(
-                self._exec_command(["tar", "-C", "/artifacts", "-cf", "-", "."], timeout=60),
+                self._exec_command(["tar", "-C", "/artifacts", "-cf", "-", "."], pipe_io=True, timeout=60),
                 env=host_environment(),
+                stdin=subprocess.DEVNULL,
                 stdout=subprocess.PIPE,
                 stderr=subprocess.DEVNULL,
             )

tests/test_analysis_sandbox.py

@@ -3,6 +3,8 @@
 import json
 import os
 import subprocess
+import sys
+import tarfile
 import time
 import uuid
 from pathlib import Path
@@ -30,6 +32,77 @@ def test_reject_invalid_resource_and_write_contracts():
         sandbox.write_file("/workspace/program.py", "x" * (256 * 1024 + 1))
 
 
+@pytest.mark.parametrize("operation", ["read", "export"])
+def test_binary_transfer_outgrows_exec_output_cap(tmp_path, operation):
+    """Model SmolVM's bounded --stream path, including binary bytes and pipe backpressure."""
+    payload = bytes(range(256)) * 512
+    source = tmp_path / "payload.png"
+    source.write_bytes(payload)
+    archive = tmp_path / "payload.tar"
+    with tarfile.open(archive, "w") as stream:
+        stream.add(source, arcname="payload.png")
+    engine = tmp_path / "smolvm"
+    engine.write_text(
+        f"#!{sys.executable}\n"
+        "import sys\nfrom pathlib import Path\n"
+        f"data=Path({str(archive)!r}).read_bytes()\n"
+        "if '--stream' in sys.argv:\n"
+        " sys.stdout.buffer.write(data[:65536]); sys.exit(1)\n"
+        "assert '-i' in sys.argv and '-t' not in sys.argv\n"
+        "assert sys.stdin.buffer.read() == b''\n"
+        "sys.stdout.buffer.write(data)\n"
+    )
+    engine.chmod(0o700)
+    sandbox = AnalysisSandbox()
+    sandbox.engine, sandbox.started_at = str(engine), time.monotonic()
+    if operation == "read":
+        assert sandbox.read_bytes("/artifacts/payload.png") == payload
+    else:
+        output = tmp_path / "delivered"
+        assert sandbox.export_artifacts(output) == ["payload.png"]
+        assert (output / "payload.png").read_bytes() == payload
+
+
+@smolvm
+def test_real_large_binary_read_and_aggregate_export(tmp_path):
+    import hashlib
+
+    profile = ResourceProfile(memory_mib=768, cpus=1, workspace_mib=128, temporary_mib=128, artifacts_mib=64)
+    with AnalysisSandbox(profile=profile) as sandbox:
+        result = sandbox.run_python(
+            "from pathlib import Path; import hashlib,json\n"
+            "data=bytes(range(256))*102400\n"
+            "Path('/artifacts/large.png').write_bytes(data)\n"
+            "print(hashlib.sha256(data).hexdigest())"
+        )
+        assert result.exit_code == 0, result.stderr
+        expected = result.stdout.strip()
+        data = sandbox.read_bytes("/artifacts/large.png", max_bytes=32 * 1024**2)
+        assert len(data) == 25 * 1024**2 and hashlib.sha256(data).hexdigest() == expected
+        delivered = tmp_path / "single"
+        assert sandbox.export_artifacts(delivered) == ["large.png"]
+        assert hashlib.sha256((delivered / "large.png").read_bytes()).hexdigest() == expected
+        result = sandbox.run_python(
+            "from pathlib import Path\n"
+            "Path('/artifacts/large.png').unlink()\n"
+            "for i in range(3): Path(f'/artifacts/report-{i}.html').write_bytes(bytes([i])*8*1024**2)"
+        )
+        assert result.exit_code == 0, result.stderr
+        delivered = tmp_path / "aggregate"
+        assert set(sandbox.export_artifacts(delivered)) == {f"report-{i}.html" for i in range(3)}
+        for i in range(3):
+            assert (delivered / f"report-{i}.html").read_bytes() == bytes([i]) * 8 * 1024**2
+        result = sandbox.run_python(
+            "from pathlib import Path\n"
+            "for path in Path('/artifacts').iterdir(): path.unlink()\n"
+            "with open('/artifacts/oversized.html','wb') as output: output.truncate(65*1024**2)"
+        )
+        assert result.exit_code == 0, result.stderr
+        with pytest.raises(ValueError, match="exceeds run limits"):
+            sandbox.export_artifacts(tmp_path / "oversized")
+        assert not (tmp_path / "oversized").exists()
+
+
 @smolvm
 def test_real_persistent_workspace_boundary_and_export(tmp_path):
     source = tmp_path / "source.csv"