Request: remote iterative evaluation service / web UI for sandbox backends

[sakura-gui]

Hi NatureBench team,

Thank you for releasing NatureBench and its iterative evaluation protocol.

The current open-source runner gives each agent an EVAL_SERVICE_URL and lets it repeatedly call POST /evaluate, using the best score across attempts. The existing eval_service.py works well when the agent container and evaluator share a host filesystem: the request passes an output_dir path that the service can read.

We are integrating NatureBench with remote sandbox/orchestration backends (for example, a sandbox running on a different worker from the controller/evaluator). In that setup, the agent and evaluation service do not share the local /workspace/output path. Running only a final verifier would lose NatureBench's official solve-evaluate-improve loop and make results less comparable.

Would you consider either:

1. open-sourcing any remote/hosted evaluation website or service used internally, if one exists; or
2. adding a deployable remote-submission mode (and optionally a minimal web UI) to the existing evaluation service?

A remote mode could support:

• authenticated, run-scoped tokens;
• uploading outputs or referencing an object-storage/shared-storage URI instead of a host-local path;
• asynchronous evaluation plus polling for long-running evaluators;
• pausing the solve timer while evaluation is in flight;
• the same multi-attempt best-score tracking and submissions.jsonl records as the official runner;
• container/deployment documentation, while keeping ground truth accessible only to evaluation workers.

Some task outputs can be quite large, so object-storage or shared-storage references would likely be more practical than sending the archive directly in every /evaluate request.

Is there already an internal implementation or roadmap for this? If not, would the maintainers be open to a community PR adding a remote submission backend to eval_service.py?

Thanks!

[ruuang]

Thank you for raising this and for explaining the remote sandbox/orchestration use case so clearly. We agree that this is an important limitation to address.

We do not currently have an internal remote/hosted evaluation service. However, we would be very happy to welcome a community PR adding a remote submission mode.

If you would like to contribute, you are very welcome to open a PR or discuss your proposed implementation with us here beforehand. We would be glad to work through the design with you.

Alternatively, if you would prefer us to implement it, please feel free to share any additional requirements or constraints that are important for your integration. We will take them into account and work on it as soon as we can.

Thank you again for the thoughtful suggestion and for offering to contribute!