Repository navigation
MMIF describe as a part of mmif_utils module #326
Description
New Feature Summary
As a sibling to rewind command, it'd be nice to have a command line tool to describe/summarize input MMIF . This thread to discuss and brainstorm such a feature.
Related
Maybe we can start by exploring possibility of merging https://github.com/clamsproject/mmif-summarizer .
Alternatives
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
keighrim commentedon Nov 23, 2025
#339 currently provides a very limited set of statistics for the input MMIF. Although this
describecommand is not aiming to fully replicate all that complex graph building and html rendering from themmif-summarizerimplementation, I think we could benefit from having a little more features to thedescribecommand.To fully address and close this issue, I have two key items on my wishlist:
Naming the Annotation Process Identifier
We need to decide on a proper and consistent name for the identifier that tracks the sequence of CLAMS apps used to generate a MMIF's views.
mmif describecommand #339 usespipeline_id, inherited from the oldermmif-storageimplementation.wfe-repositories). There is currently no official consensus or glossary term to my understanding.recipeId, which I find very appealing as a clear alternative.Handling Collections of MMIFs
The tool should be able to accept a directory containing multiple MMIFs as input. It should then:
I am happy to hear any additional features or wishlists from others. Please leave comments below!
marcverhagen commentedon Nov 24, 2025
To me "workflow" was always a more general and high-level term focused at the methodology around running pipelines (like using chowda) whereas "pipeline" was more concrete and pointed at the pipeline itself.
Both pipeline_id and recipe_id (or recipeId) are fine with me.
There now is overlap between summarize and describe. If the goal is not to fully replicate the summarizer it does seem to be to replicate some of the summarizer and add a couple of new things (mostly the pipeline_id, which is very useful). So in that case we should be clear on what kind of information is in the result of one and what is in the result of the other. I do not have a clear dividing line. For example, I see the suggestion to have collection-level statistics for MMIF files. After a suggestion from Owen I was working on that for the summarizer and was including collection-level error/warning statistics (since the summarizer does retain the warnings), it was also including annotation statistics (which is not envisioned for describe I think); on the other hand, I had no plans to put in how many MMIFs came from the same recipe.
The describe command also has some overlap with the mmif-storage status API. This brings up something that I had thought about before which is to integrate summarize (and now describe) into the data server.
keighrim commentedon Nov 24, 2025
Regarding
I worked on this clamsproject/aapb-brandeis-datahousing@b1cdb4b Haven't merged yet, waiting for some further decisions from this discussion (like the naming).
The summarizer should also be updated to declutter the duplication of code, and I have thought that was the plan: splitting the summarizer into a CLI and more "advance" features like HTML generation and merge CLI to the main
mmifcmd. So there shouldn't be any "overlap" ideally.marcverhagen commentedon Nov 24, 2025
Nice.
Agreed. I am hoping to have that done by the time we meet this afternoon. To me, that's the easy part. What I was trying to say was that it is unclear to me how much the CLI part of the summarizer overlaps with describe and how best to separate them. The one thing that sticks out a bit to me is that the summary is always going to be more about the annotation content rather than any pipeline-related statistics, which make the pipeine_id one of those things that should be done by describe.
keighrim commentedon Nov 27, 2025
After giving some thoughts, I decided to go with more generic term "workflow" (I really liked "recipe", though!)
Also noticed that the
describeand the "ID" both don't have information about the source file. Assuming the MMIF file name is encoding the asset ID (AAPB-GIUD for AAPB data for example), we should at least show how many source files were used in a given MMIF. Then, since the source documents always come at the top-level, source info must be the first prefix in the ID construction. For example, IDs forVideoDocument-1/app-swt-detection/v8.1/xxxxTextDcoument-1-VideoDocument-1/app-nfa-wrapper/v0.8/xxxxSome notes on the first prefix
type-number-type-number-.... I think for thetypeparts, they MUST BE the canonical short names for document type in the vocabulary (unlike in themmif sourceorclams sourcecommand, where we use non-canonical short names likevideotext, ..)-for everywhere in that part of the string, assuming there's no negative number of documents used.VideoDocument-1-TextDcoument-1)keighrim commentedon Nov 27, 2025
#341 adds
pipeline_id-->workflowIdkeighrim commentedon Dec 19, 2025
v1.2.1 (#343) introduced workflow_id generation using "source information" as the initial prefix. however, following a recent discussion with Marc, we agreed that there is no significant need for this information, and agreed that its utility does not outweigh the risk of breaking backward compatibility. So that prefix will be reverted in the next iteration.
removed "source" media counts from workflow ID prefix ...
removed "source" media counts from workflow ID prefix ...
keighrim commentedon Apr 28, 2026
At 4/34/2026 meeting with Marc, we decided to bring back the source composition element, for better disambiguation. Also see clamsproject/mmif-storage#10. PR #386 simply brings back the old source syntax described three comments above.