Skip to content

Document checkpointing work inside a retried task - #200

Open
mangeshraut712 wants to merge 1 commit into
Netflix:masterfrom
mangeshraut712:docs-resumable-processing
Open

mangeshraut712 wants to merge 1 commit into
Netflix:masterfrom
mangeshraut712:docs-resumable-processing

Conversation

@mangeshraut712

Copy link
Copy Markdown

Refs Netflix/metaflow#2405.

Adds a section to "Dealing with Failures" showing how a step that loops over many items can store each finished result in S3, so a @retry attempt picks up where the last one stopped instead of starting over. It's based on the gist linked in the issue, with a deterministic failure (current.retry_count == 0) so the example behaves the same on every run, and results sorted by key.

I ran the example with --datastore=s3 against a local mock S3. The first attempt failed at item 5, the retry processed items 5–9, and end printed all 10 results. yarn build passes.

The notes cover the limits: items are matched by position, an item can be processed twice if it fails midway, and checkpoints don't carry over to a new run or resume.

@vercel

vercel Bot commented Nov 2, 2026

Copy link
Copy Markdown

@mangeshraut712 is attempting to deploy a commit to the Anaconda Platform Team on Vercel.

A member of the Team first needs to authorize it.

@mangeshraut712

Copy link
Copy Markdown
Author

Current head 7f7ff43f90216e01065596a6788e43ad70e9481f is blocked by Vercel deployment authorization. Could a member of the linked Vercel team authorize the preview deployment(s), or let me know what contributor-side work is needed first? The failed deployment checks link to the authorization page; I have not treated them as successful builds.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant