Route image-processing workers to the GPU Celery queue - #171
Merged
Conversation
Flip isGPUWorker from "false" to "true" for the six Image Processing category workers (registration, histogram_matching, gaussian_blur, rolling_ball, crop, h_and_e_deconvolution). Their compute jobs are long-running (registration measured at 900-1500s on prod) and were saturating the 4-slot cpu queue, delaying fast interface requests platform-wide. Interface requests still route to the cpu queue regardless of this label, so parameter forms stay fast while the heavy compute moves to the gpu boxes. No build or runtime changes: these remain CPU-bound workers that will simply run on the GPU hosts' CPUs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HuW5HKHQFfo5qUZcm2jEs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flips the
isGPUWorkerDocker label from"false"to"true"for the sixImage Processingcategory workers so their compute jobs route to thegpuCelery queue instead of the sharedcpuqueue:workers/annotations/registration/workers/annotations/histogram_matching/workers/annotations/gaussian_blur/workers/annotations/rolling_ball/workers/annotations/crop/workers/annotations/h_and_e_deconvolution/(
deconwolf, the seventh image-processing worker, already carriesisGPUWorker="true".)Motivation
These workers process whole datasets frame-by-frame and can run for a long time — registration compute was measured at 900–1500s on prod. The CPU box has only 4 concurrency slots, so tool parameter forms queued behind these jobs for 16–25 minutes platform-wide.
Only
compute/previewrequests route by this label;interfacerequests always go to thecpuqueue. So parameter forms stay fast on the cheap box while the heavy compute moves to the GPU hosts.Non-changes
blob_overlap_workeris deliberately left on the CPU queue (out of scope for this change).Dockerfile_M1variant, and theirWORKERNAME.mddocs don't reference queue routing, so no other files needed touching.Deployment note
The dispatcher reads the label from the deployed image, so this takes effect once these images are rebuilt and pushed to ECR.
🤖 Generated with Claude Code
https://claude.ai/code/session_016HuW5HKHQFfo5qUZcm2jEs
Generated by Claude Code