Skip to content

Benchmark launcher finalization rewrites completed instance results #608

Description

@R-Palazzo

Environment Details

  • SDGym: v0.14.3

Error Description

When a benchmark run times out, the benchmark launcher finalizes the run by filling in missing results*.csv files from available per-job artifacts. However, BenchmarkLauncher.finalize() processes every launched instance, including instances that already completed successfully and already uploaded both results*.csv and metainfo*.yaml.

For those completed instances, finalization should leave results*.csv and metainfo*.yaml untouched. The finalized completed_date format is also inconsistent with normal benchmark completion:

  • normal benchmark path: %m_%d_%Y %H:%M:%S
  • launcher finalize path: %d_%m_%Y %H:%M:%S

Expected behavior:

  • Completed instances should be skipped during finalization.
  • Existing results*.csv files should not be rewritten.
  • Existing completed metainfo*.yaml files should not have completed_date changed.
  • Finalization timestamps should use the same format as normal benchmark completion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions