Fix:cascade subjob failure

[BelhsanHmida]

Description

Closes #2404.

When a subjob in a sequential scheduling chain fails and its scheduler defines no fallback, the chain never reaches a terminal state. RQ only enqueues the dependents of a job that succeeded, so the remaining subjobs and the wrap-up job stay deferred forever. The wrap-up job's id is what POST /assets/<id>/schedules/trigger hands to the client, so that client polls a job that will never move, and never learns what went wrong.

• Cascade the failure: when a scheduling job fails and no fallback job was created, the deferred jobs that depend on it are moved to a terminal failed state, recording an UpstreamSchedulingFailure that names the device which could not be scheduled.
• Let the wrap-up job run and report: it inspects every subjob and fails with a message listing the devices that failed and the devices that were consequently never scheduled. A subjob that failed but whose fallback job succeeded does not count as a failure.
• Cascade from the original job when a fallback job fails, since the fallback was standing in for the job whose dependents are waiting.
• Added changelog item in documentation/changelog.rst and documentation/api/change_log.rst
• Documented how to retry after a failed job in documentation/api/introduction.rst

Scope after #2252

This branch was opened against main before #2252 (retiring the storage fallback scheduler) landed, because the bug did not depend on it: trigger_optional_fallback only creates a fallback job when scheduler_class.fallback_scheduler_class is not None, and the base Scheduler defaults that to None, so ProcessScheduler and every custom scheduler that did not opt in already wedged chains.

Two smaller wedges are closed along the way. The old code only did anything at all for InfeasibleProblemException, so any other failure (a bug, a database error) wedged the chain regardless of fallbacks; the cascade now runs for every failure. And a failing fallback job used to leave the original job's dependents deferred, which the fallback's own success_callback was the only thing that would have rescued.

Now that #2252 has merged, no scheduler ships with a fallback at all, so this affects every sequential schedule whose devices are scheduled by a built-in scheduler. The merge is included in this branch, and three things were adjusted for it:

• test_create_sequential_jobs_without_storage_fallback, which Feat/retire fallback scheduler #2252 added to assert the wedged behaviour (deferred subjobs stay deferred, plus a cleanup block to stop them leaking into the next test), is superseded by test_create_sequential_jobs_without_fallback here and has been removed. Its assertions were not in finished_jobs, which a cascaded (failed) job still satisfies, so it would have passed while asserting the opposite of the intended behaviour.
• The fallback race test can no longer use StorageFallbackScheduler, which Feat/retire fallback scheduler #2252 deleted. It now makes the storage scheduler stand in as its own fallback, which is the situation a custom scheduler that defines one is still in.
• The end-to-end API test needed a new source of infeasibility: Feat/retire fallback scheduler #2252 relaxes SoC bounds and targets by default, so the old "uni-directional station cannot reach a target below its initial SoC" case is now a priced breach rather than a failure. It now uses a soc-usage above the device's power-capacity, which stays a hard constraint.

Client-visible outcome

GET /api/v3_0/jobs/<uuid> on the returned job id now answers 422 Unprocessable Entity:

{
    "status": "FAILED",
    "message": "Scheduling job failed with UpstreamSchedulingFailure: Sequential scheduling failed for sensor 15 (Test charging station - power): InfeasibleProblemException: infeasible. As a result, no schedule was computed for sensor 16 (Test charging station (bidirectional) - power).",
    "result": null
}

GET /sensors/<id>/schedules/<uuid> answers UNKNOWN_SCHEDULE with the same reason. Polling a cascaded subjob directly gives that subjob's own terminal reason, naming the device that failed upstream.

Notes for reviewers

Why not Dependency(allow_failure=True) for the wrap-up job? That is the obvious RQ-native mechanism, and it is wrong here. RQ enqueues an allow_failure dependent the moment its dependency fails — but when the failing subjob has a fallback scheduler, the fallback job is enqueued by that same failure, so both sit in the queue at once. A single worker pops FIFO and happens to run the fallback first, which hides it; with more than one scheduling worker, the wrap-up job can run while the fallback is still pending, find a device without a schedule, and report the chain as failed moments before the fallback schedules that device after all. The wrap-up job therefore keeps a plain dependency (so RQ never enqueues it on failure, exactly as before this PR), and is marked in its meta data with RUNS_ON_CHAIN_FAILURE; the cascade queues such jobs itself, only after the rest of the chain has reached a terminal state. test_create_sequential_jobs_fallback_for_last_device pins this invariant down by stopping the worker right after the failing subjob and asserting the wrap-up job is still deferred.

This still matters after #2252: the fallback machinery remains for custom schedulers that define one (FLEXMEASURES_FALLBACK_REDIRECT is documented as applying to exactly those), so the race would be reachable for them.

Why does naming the device swallow SQLAlchemyError? The cascade runs for every failure now, and naming the device costs a database look-up. A job that failed on a database error leaves the session needing a rollback, and nothing rolls it back between jobs, so that look-up raises — which would abort the failure callback before it cascades, and wedge the chain precisely for the failures where a terminal state matters most. _describe_scheduled_device therefore falls back to the bare reference (sensor 4) and logs a warning. Losing the device's name is a small price for still reporting the failure.

Out of scope

@job_cache is left alone. Re-triggering an identical request now hands back the failed wrap-up job — terminal, with a reason — which is the same behaviour a failed single-sensor trigger already has, and satisfies the issue's requirement that a re-trigger must not hand back a wedged job. Getting a fresh attempt still needs force-new-job-creation (now documented in documentation/api/introduction.rst). Making the cache skip terminally-failed jobs would change behaviour for every trigger endpoint, which felt like a separate decision.

Relatedly, job_cache's requeue=True path would requeue only the wrap-up job of a chain, which is wrong — but nothing in the API passes it, so that is pre-existing and untouched.

Testing

• test_create_sequential_jobs_without_fallback — makes the first device infeasible, asserts the second subjob and the wrap-up job both end up failed, that the reason names the device, and that nothing is left deferred. Replaces test_create_sequential_jobs_without_storage_fallback.
• test_asset_sequential_schedule_without_fallback_fails_terminally — the same end-to-end over the API, with a genuinely infeasible first device (no mocked compute): triggers, polls the returned job id for a 422 naming the device, and re-triggers to check the job handed back is not waiting on a chain that will never complete.
• test_create_sequential_jobs_fallback_for_last_device — a fallback for the last device, whose wrap-up job depends on the failing subjob directly.
• test_describe_scheduled_device_survives_an_unusable_session — the device look-up degrades instead of raising.

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34002559 | 📁 Comparing c053b66 against latest (3327ad4)

  🔍 Preview build  

39 files changed · + 1 added · ± 38 modified

+ Added

• _autosummary/flexmeasures.utils.sentry_utils.html

± Modified

• changelog.html
• configuration.html
• genindex.html
• getting-started.html
• index.html
• py-modindex.html
• _autodoc/RELAX_CAPACITY_CONSTRAINTS.html
• _autodoc/RELAX_CONSTRAINTS.html
• _autodoc/RELAX_SITE_CAPACITY_CONSTRAINTS.html
• _autodoc/RELAX_SOC_CONSTRAINTS.html
• and 28 more...