Skip to content

Add a live-processing example ClusterWorkflowTemplate #1433

Open
hazdl wants to merge 1 commit into
mainfrom
hz/sample-cluster-wftemplate
Open

Add a live-processing example ClusterWorkflowTemplate #1433
hazdl wants to merge 1 commit into
mainfrom
hz/sample-cluster-wftemplate

Conversation

@hazdl

@hazdl hazdl commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

This PR adds a new example Argo Workflows ClusterWorkflowTemplate named surface-analysis to the workflows repository to demonstrate how users can submit jobs intended for live processing.

The example is primarily intended as a learning exercise for a new team member and provides a simple end-to-end scientific workflow that demonstrates:

Creating and registering a ClusterWorkflowTemplate
Submitting workflows from a cluster-scoped template
DAG execution and task dependencies
Artifact passing between workflow stages
Parallel execution of independent tasks
Generating output reports from upstream processing results

The example also demonstrates how workflows intended for live processing can be identified using workflow metadata annotations and labels, allowing the platform to distinguish between live and offline processing workloads.

The workflow itself performs a synthetic surface analysis pipeline consisting of:
3D volume reconstruction
Surface segmentation
Crack detection
Surface roughness analysis
HTML report generation

Testing verified by:
Creating the ClusterWorkflowTemplate
Submitting workflows from the template
Executing the DAG successfully
Verifying artifact passing between tasks
Producing the final report artifact
Confirming the workflow is correctly identified as a live-processing example via metadata

@davehadley davehadley left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hazdl this was a good learning exercise to learn how to write Argo Workflows Templates. But I think that what is included in this PR goes beyond the scope of https://jira.diamond.ac.uk/browse/AP-1097. The point of the ClusterWorkflowTemplates in the examples directory is to provide minimal examples that demonstrate how to do something. In this case, our aim is simply to show the users how to run live-processing. I think that the CT analysis is distracting from the point of the template, to demo the "workflows.diamond.ac.uk/type: "live"" annotation.

Let's discuss a more minimal example that satisfies the requirements JIRA ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants