Feature proposal: unified parallel evaluation interface for RoboTwin and LIBERO using Labtasker - #28
luocfprime wants to merge 6 commits into
Conversation
wayrise
left a comment
wayrise
left a comment
There was a problem hiding this comment.
I recommend revising the evaluation granularity and repository placement before merging.
1. Schedule at the episode (or small episode-batch) level
The current design assigns one complete task to one Worker. For RoboTwin, a Task normally contains the full episode set (typically 100 episodes). This creates a long-tail problem: once most tasks finish, the remaining slow task can continue occupying only one Worker while all other Workers are idle. The overall evaluation time is therefore bounded by the slowest task, and idle Workers cannot help with its remaining episodes.
Please consider making the schedulable unit an episode, or at least a small episode batch, for example (task, mode, episode_start, num_episodes). This would allow idle Workers to claim more work from the same benchmark task. The implementation should preserve deterministic episode seeds and ordering, aggregate results back to task-level and benchmark-level success rates, and retry only the failed episode/batch where possible. Each Worker can still use its own exclusive policy endpoint; the endpoint ownership constraint does not require the Labtasker Task to represent an entire benchmark task.
For LIBERO, the existing trial_start / num_trials interface appears to provide a natural foundation for this split. RoboTwin may need an evaluator entry point that accepts an episode range or a single-episode invocation while preserving the original seed semantics.
2. Extend the existing benchmark directories instead of adding parallel copies
Please integrate the Labtasker entry points into the existing benchmarks/robotwin and benchmarks/libero directories as an additional execution mode. Adding separate robotwin-labtasker and libero-labtasker trees duplicates evaluator, interface, configuration, and launcher code, which will make future fixes and protocol changes prone to drift between the old and new paths.
The corresponding tests should be moved to the repository-level tests/ directory (for example, tests/benchmarks/robotwin/ and tests/benchmarks/libero/) rather than living under benchmarks/*/tests. This keeps test discovery and CI ownership consistent with the rest of the repository.
These changes are important for both practical evaluation throughput and long-term maintainability.
Thanks, this is very helpful. This was intentionally an early version submitted to get feedback before settling on the final implementation.
I’ll revise the implementation in this direction and follow up here once the changes and tests are ready. Please let me know if I’ve misunderstood the intended scope. |
luocfprime
changed the title
6490052 to
6d9c849
Compare
|
Hello, this implements issue #29.
This supersedes the earlier whole-task implementation in this draft. |
|
Here are the reproduced LIBERO and RoboTwin results using this evaluation implementation. LIBERO: ❯ uv run --isolated --with labtasker python \
benchmarks/libero/labtasker_summarize.py \
full-libero-eval-20260915 \
--queue fastwam \
--output-dir /tmp/libero-summary
Installed 108 packages in 348ms
[labtasker] connected server=remote transport=https url=https://labtasker.xxxxxxxx.site
Suite summary
Suite Success/Trials Success rate Done/Total Pending Running Failed
-------------- -------------- ------------ ---------- ------- ------- ------
libero_10 492/500 98.40% 100/100 0 0 0
libero_goal 495/500 99.00% 100/100 0 0 0
libero_object 499/500 99.80% 100/100 0 0 0
libero_spatial 496/500 99.20% 100/100 0 0 0
Task summary
Suite Task Success/Trials Success rate Done/Total Pending Running Failed
-------------- ---- -------------- ------------ ---------- ------- ------- ------
libero_10 0 47/50 94.00% 10/10 0 0 0
libero_10 1 50/50 100.00% 10/10 0 0 0
libero_10 2 49/50 98.00% 10/10 0 0 0
libero_10 3 50/50 100.00% 10/10 0 0 0
libero_10 4 48/50 96.00% 10/10 0 0 0
libero_10 5 50/50 100.00% 10/10 0 0 0
libero_10 6 49/50 98.00% 10/10 0 0 0
libero_10 7 50/50 100.00% 10/10 0 0 0
libero_10 8 49/50 98.00% 10/10 0 0 0
libero_10 9 50/50 100.00% 10/10 0 0 0
libero_goal 0 50/50 100.00% 10/10 0 0 0
libero_goal 1 50/50 100.00% 10/10 0 0 0
libero_goal 2 48/50 96.00% 10/10 0 0 0
libero_goal 3 50/50 100.00% 10/10 0 0 0
libero_goal 4 50/50 100.00% 10/10 0 0 0
libero_goal 5 50/50 100.00% 10/10 0 0 0
libero_goal 6 49/50 98.00% 10/10 0 0 0
libero_goal 7 50/50 100.00% 10/10 0 0 0
libero_goal 8 48/50 96.00% 10/10 0 0 0
libero_goal 9 50/50 100.00% 10/10 0 0 0
libero_object 0 50/50 100.00% 10/10 0 0 0
libero_object 1 50/50 100.00% 10/10 0 0 0
libero_object 2 50/50 100.00% 10/10 0 0 0
libero_object 3 50/50 100.00% 10/10 0 0 0
libero_object 4 50/50 100.00% 10/10 0 0 0
libero_object 5 50/50 100.00% 10/10 0 0 0
libero_object 6 50/50 100.00% 10/10 0 0 0
libero_object 7 49/50 98.00% 10/10 0 0 0
libero_object 8 50/50 100.00% 10/10 0 0 0
libero_object 9 50/50 100.00% 10/10 0 0 0
libero_spatial 0 50/50 100.00% 10/10 0 0 0
libero_spatial 1 48/50 96.00% 10/10 0 0 0
libero_spatial 2 50/50 100.00% 10/10 0 0 0
libero_spatial 3 50/50 100.00% 10/10 0 0 0
libero_spatial 4 49/50 98.00% 10/10 0 0 0
libero_spatial 5 49/50 98.00% 10/10 0 0 0
libero_spatial 6 50/50 100.00% 10/10 0 0 0
libero_spatial 7 50/50 100.00% 10/10 0 0 0
libero_spatial 8 50/50 100.00% 10/10 0 0 0
libero_spatial 9 50/50 100.00% 10/10 0 0 0
Output files
Suite summary: /tmp/libero-summary/suite_summary.csv
Task summary: /tmp/libero-summary/task_summary.csvRoboTwin (demo_clean): ❯ uv run --isolated --with labtasker python \
benchmarks/robotwin/labtasker_summarize.py \
full-robotwin-eval-clean-20260915 \
--queue fastwam \
--output-dir /tmp/robotwin-summary/clean
Installed 108 packages in 1.10s
[labtasker] connected server=remote transport=https url=https://labtasker.xxxxxxxx.site
Benchmark summary
Mode Instructions Success/Episodes Success rate Done/Total Pending Running Failed
---------- ------------ ---------------- ------------ ---------- ------- ------- ------
demo_clean unseen 4715/5000 94.30% 1000/1000 0 0 0
Task summary
Mode Instructions Task Success/Episodes Success rate Done/Total Pending Running Failed
---------- ------------ ------------------------- ---------------- ------------ ---------- ------- ------- ------
demo_clean unseen adjust_bottle 100/100 100.00% 20/20 0 0 0
demo_clean unseen beat_block_hammer 99/100 99.00% 20/20 0 0 0
demo_clean unseen blocks_ranking_rgb 100/100 100.00% 20/20 0 0 0
demo_clean unseen blocks_ranking_size 76/100 76.00% 20/20 0 0 0
demo_clean unseen click_alarmclock 98/100 98.00% 20/20 0 0 0
demo_clean unseen click_bell 100/100 100.00% 20/20 0 0 0
demo_clean unseen dump_bin_bigbin 96/100 96.00% 20/20 0 0 0
demo_clean unseen grab_roller 100/100 100.00% 20/20 0 0 0
demo_clean unseen handover_block 98/100 98.00% 20/20 0 0 0
demo_clean unseen handover_mic 86/100 86.00% 20/20 0 0 0
demo_clean unseen hanging_mug 51/100 51.00% 20/20 0 0 0
demo_clean unseen lift_pot 100/100 100.00% 20/20 0 0 0
demo_clean unseen move_can_pot 99/100 99.00% 20/20 0 0 0
demo_clean unseen move_pillbottle_pad 99/100 99.00% 20/20 0 0 0
demo_clean unseen move_playingcard_away 100/100 100.00% 20/20 0 0 0
demo_clean unseen move_stapler_pad 99/100 99.00% 20/20 0 0 0
demo_clean unseen open_laptop 100/100 100.00% 20/20 0 0 0
demo_clean unseen open_microwave 87/100 87.00% 20/20 0 0 0
demo_clean unseen pick_diverse_bottles 87/100 87.00% 20/20 0 0 0
demo_clean unseen pick_dual_bottles 93/100 93.00% 20/20 0 0 0
demo_clean unseen place_a2b_left 98/100 98.00% 20/20 0 0 0
demo_clean unseen place_a2b_right 94/100 94.00% 20/20 0 0 0
demo_clean unseen place_bread_basket 94/100 94.00% 20/20 0 0 0
demo_clean unseen place_bread_skillet 92/100 92.00% 20/20 0 0 0
demo_clean unseen place_burger_fries 98/100 98.00% 20/20 0 0 0
demo_clean unseen place_can_basket 79/100 79.00% 20/20 0 0 0
demo_clean unseen place_cans_plasticbox 98/100 98.00% 20/20 0 0 0
demo_clean unseen place_container_plate 99/100 99.00% 20/20 0 0 0
demo_clean unseen place_dual_shoes 96/100 96.00% 20/20 0 0 0
demo_clean unseen place_empty_cup 100/100 100.00% 20/20 0 0 0
demo_clean unseen place_fan 98/100 98.00% 20/20 0 0 0
demo_clean unseen place_mouse_pad 99/100 99.00% 20/20 0 0 0
demo_clean unseen place_object_basket 93/100 93.00% 20/20 0 0 0
demo_clean unseen place_object_scale 96/100 96.00% 20/20 0 0 0
demo_clean unseen place_object_stand 97/100 97.00% 20/20 0 0 0
demo_clean unseen place_phone_stand 94/100 94.00% 20/20 0 0 0
demo_clean unseen place_shoe 100/100 100.00% 20/20 0 0 0
demo_clean unseen press_stapler 91/100 91.00% 20/20 0 0 0
demo_clean unseen put_bottles_dustbin 95/100 95.00% 20/20 0 0 0
demo_clean unseen put_object_cabinet 86/100 86.00% 20/20 0 0 0
demo_clean unseen rotate_qrcode 93/100 93.00% 20/20 0 0 0
demo_clean unseen scan_object 97/100 97.00% 20/20 0 0 0
demo_clean unseen shake_bottle 100/100 100.00% 20/20 0 0 0
demo_clean unseen shake_bottle_horizontally 100/100 100.00% 20/20 0 0 0
demo_clean unseen stack_blocks_three 99/100 99.00% 20/20 0 0 0
demo_clean unseen stack_blocks_two 100/100 100.00% 20/20 0 0 0
demo_clean unseen stack_bowls_three 89/100 89.00% 20/20 0 0 0
demo_clean unseen stack_bowls_two 97/100 97.00% 20/20 0 0 0
demo_clean unseen stamp_seal 98/100 98.00% 20/20 0 0 0
demo_clean unseen turn_switch 77/100 77.00% 20/20 0 0 0
Output files
Benchmark summary: /tmp/robotwin-summary/clean/benchmark_summary.csv
Task summary: /tmp/robotwin-summary/clean/task_summary.csv
The results produced by our evaluation implementation are consistent with the OpenWAM originally reported results, with all completed benchmark settings differing by less than 1 percentage point. |
|
Hello! I believe the requested changes are now in place, and I’ve verified the implementation by running the full evaluation suite and reproducing the results. I’d appreciate another review. Please let me know if anything needs further adjustment or clarification! |
Review notes from running this PR on a multi-GPU nodeHow these were produced: checked out Overall this is solid work — content-addressed sealed manifests with atomic publication, the stdin-lifeline process guardian, RNG domain isolation, and the layered result validation are all well designed, and the integration tests (real Labtasker server + real worker subprocesses + fake simulators) exercise caching, retries, and idempotent resubmission end to end. Findings, roughly by severity: 1.
|
|
Thanks for the detailed report. Issues other than problem 2 seem easy to address. I’ll address the other issues as soon as possible. For problem 2, this appears to be caused by the interaction between the automatic daemon’s filesystem ownership lock ( labtasker-server serve --database /path/on/local/storage/server.dbIn my own setup, I run the Labtasker Server on an independent CVM with its database on local storage. Clients can connect to it whether they run inside or outside the cluster, so this issue does not occur. However, that workaround is insufficient when the Labtasker Server itself runs inside a cluster environment. Persistent storage there is often NFS-backed, while local pod or node storage is ephemeral and disappears when the allocation is released. I’ll review this design shortly, particularly whether Labtasker should use a different SQLite journal mode and ownership-locking mechanism for shared filesystems. |
KraHsu
left a comment
KraHsu
left a comment
There was a problem hiding this comment.
This review found three issues that should be addressed before merging: policy endpoint ownership, completion reporting after an interrupted submission, and readiness checks in environments with proxy settings. Reproduction conditions and suggested changes are in the inline comments.
Validation included CPU regression tests and bounded GPU checks covering real LIBERO policy rollouts, policy reuse and reload, process and GPU-memory cleanup, and protocol checks with the official RoboTwin checkpoint. RoboTwin simulator checks used the existing Python environment with freshly checked-out pinned sources. The fresh RoboTwin environment's CUDA error and the trajectory differences observed during replay have not been attributed to this PR, so neither is counted as a confirmed PR defect. These checks do not constitute a full benchmark evaluation.
| with connect(f"ws://{args.host}:{args.port}", open_timeout=1) as ws: | ||
| ws.send(json.dumps({"type": "ping"})) | ||
| if json.loads(ws.recv(timeout=1)).get("type") == "pong": | ||
| break |
There was a problem hiding this comment.
Verify that the ready endpoint belongs to the newly started policy process
If another policy server already occupies the target port while the new process is still loading its model, that existing server's pong passes this check and identity is subsequently set to the new model. The evaluator can therefore connect to the wrong checkpoint, or share a stateful policy server with another Worker. I reproduced this with a real WebSocket endpoint and a still-running child representing the new server's loading phase. Please verify that the endpoint belongs to this launch before marking the model ready, and fail explicitly on a port conflict.
| if status not in counts: | ||
| raise ValueError(f"unsupported Labtasker status: {task.status}") | ||
| counts[status] += 1 | ||
| return counts | {"expected": len(tasks)} |
There was a problem hiding this comment.
Account for tasks missing from an interrupted submission
The submission metadata does not record the complete expected task range, while expected here is just the number of tasks that currently exist. I reproduced this in both benchmark adapters: request 50 episodes/trials in batches of 5, interrupt creation of the second batch, and finish the first batch. The summary then reports Done/Total=1/1 and Pending=0, with no indication that the remaining 45 episodes/trials were never submitted. Please persist the expected task and episode/trial ranges so the summary can distinguish "all created tasks completed" from "the entire submission completed."
| if labtasker.cancellation_requested(): | ||
| raise RuntimeError("Task cancelled during model loading") | ||
| try: | ||
| with connect(f"ws://{args.host}:{args.port}", open_timeout=1) as ws: |
There was a problem hiding this comment.
Disable proxy use for the local policy readiness check
This direct websockets.connect call inherits proxy settings in versions that support automatic proxy discovery. If the proxy configuration does not bypass the policy host, a healthy local server can fail the readiness check and then be terminated by startup cleanup. In a controlled comparison, the existing WSPolicyClient successfully pinged the same newly started server directly, while this readiness check timed out. Please reuse the version-compatible proxy=None handling already implemented in benchmarks/utils/transport.py.
3 participants
Summary
benchmark:robotwinandbenchmark:liberoValidation
These one-episode smoke runs validate GPU compatibility, native Task lifecycle, and parallel scheduling; they are not benchmark-scale accuracy or throughput claims.
Proposal scope