Add a live-processing example ClusterWorkflowTemplate #1433
Open
hazdl wants to merge 1 commit into
Open
Conversation
davehadley
requested changes
Jul 8, 2026
Collaborator
There was a problem hiding this comment.
@hazdl this was a good learning exercise to learn how to write Argo Workflows Templates. But I think that what is included in this PR goes beyond the scope of https://jira.diamond.ac.uk/browse/AP-1097. The point of the ClusterWorkflowTemplates in the examples directory is to provide minimal examples that demonstrate how to do something. In this case, our aim is simply to show the users how to run live-processing. I think that the CT analysis is distracting from the point of the template, to demo the "workflows.diamond.ac.uk/type: "live"" annotation.
Let's discuss a more minimal example that satisfies the requirements JIRA ticket.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new example Argo Workflows ClusterWorkflowTemplate named surface-analysis to the workflows repository to demonstrate how users can submit jobs intended for live processing.
The example is primarily intended as a learning exercise for a new team member and provides a simple end-to-end scientific workflow that demonstrates:
Creating and registering a ClusterWorkflowTemplate
Submitting workflows from a cluster-scoped template
DAG execution and task dependencies
Artifact passing between workflow stages
Parallel execution of independent tasks
Generating output reports from upstream processing results
The example also demonstrates how workflows intended for live processing can be identified using workflow metadata annotations and labels, allowing the platform to distinguish between live and offline processing workloads.
The workflow itself performs a synthetic surface analysis pipeline consisting of:
3D volume reconstruction
Surface segmentation
Crack detection
Surface roughness analysis
HTML report generation
Testing verified by:
Creating the ClusterWorkflowTemplate
Submitting workflows from the template
Executing the DAG successfully
Verifying artifact passing between tasks
Producing the final report artifact
Confirming the workflow is correctly identified as a live-processing example via metadata