Repository navigation
perf(headless-computer): send files straight to and from the bucket - #7421
Open
reachjalil wants to merge 1 commit into
Open
reachjalil wants to merge 1 commit into
reachjalil wants to merge 1 commit into
Conversation
A person's files reached their computer through the runner: one stat per file, then each file read whole from the bucket into the runner and written through the provider's file API, 30 seconds at most. Which files were already copied was only remembered in memory, so every new runner process asked about every file again. Outputs came back the same way, each read through the runner after every command. With the S3 store the computer now moves the bytes itself. The runner signs a link per file, for one object and one method, that stops working within minutes; the computer runs curl over all of them in one command. The links are written to a file that is removed when the command ends, never to a command line, a log or what the model sees. An output is sent to a holding object. The runner measures what arrived, applies the per-file and per-conversation limits, and only then copies it into place inside the bucket, so a kept file is never larger than its row says and a version that doesn't fit leaves the kept one untouched. What is already on a computer is read in one command: the folder's names and sizes, and a record kept in /workspace/.jobs/.copied.json. The record only counts while the file is still there, because a replacement computer carries .jobs over but not /workspace/files. A disk store has no links, and a file that doesn't arrive by link goes through the runner as before. After a transfer in which nothing arrives, links are skipped for ten minutes. Signed-off-by: Jalil <reachjalil@gmail.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this about?
A Workbot computer now fetches the person's files from the files bucket, and sends its outputs back, by itself. The bytes no longer pass through the runner.
What problem does it solve?
Every file was held whole in the runner's memory on its way in or out, one at a time, 30 seconds each. A large video could time out or crowd the cell that serves all of a person's chats.
What was the situation before?
Before a command, the runner asked the computer about each upload, read it from the bucket and wrote it through the provider's file API. After a command it read each changed output back the same way. Every new runner process asked about every upload again.
How is this implemented?
blobs.ts: the S3 store signs a link for one object and one method that expires within minutes, and can measure and copy an object.saved-files.ts: an output is sent to a holding object;finishmeasures it, applies the limits, then copies it into place.computer.ts: one command reads the folder and a record of copied uploads;curlthen moves all pending files in one command. Links go in a file removed afterwards, never on a command line.What else changed?
README.md: how the bytes move.Release note
Large files move between your Files and a Workbot computer directly, without the earlier 30-second copy limit.
Evidence
ee/packages/headless-computer/src/computer-files.test.ts,ee/apps/headless-runner/scripts/saved-file-links.test.ts— before: each file went through the runner; after: the computer moves it withcurl, and an output larger than announced is not kept. Not run against a real bucket. No E2E.