Skip to content

Multiprocess pipeline artifacts should be written under a single pipeline output folder #1092

Description

@dhensle

Is your feature request related to a problem? Please describe.
When running in multiprocess mode, ActivitySim writes many pipeline artifacts directly into the output root (one per multiprocess step and subprocess/core). There is currently no straightforward way to keep all pipeline artifacts inside one pipeline-specific subfolder.

Describe the solution you'd like
A user should be able to configure a pipeline path such that all main and subprocess pipeline artifacts live under one common subdirectory. (And log files too!)

Describe alternatives you've considered
Centralize pipeline path construction so prefixing is path-aware:

  • Split pipeline_file_name into parent + basename.
  • Apply prefix only to basename.
  • Rejoin with parent path.

Use the same logic consistently for main and subprocess pipeline paths in checkpoint and mp_tasks code paths.

Additional context
The base pipeline name is configurable via pipeline_file_name, parsed from settings, but in multiprocess mode, subprocess names are prefixed onto the full pipeline name/path string
Result: pipeline artifacts are scattered in the output root with names like mp_accessibility_0-pipeline (or equivalent parquet directories), rather than grouped under a single pipeline folder.

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions