feat: add enrichment origins metadata field - #16
anurag6569201 wants to merge 1 commit into
Conversation
Source PR: Unstructured-IO#4370 Source head: 7a03b8c
✅ Shipwright · ApproveRecommendation: approve PR #16 · Tier
Findings (4)
Fireworks usage: 13,110 input · 537 output · 13,647 total tokens · $0.0032 · 11s · 0 fix iteration(s) Open the Shipwright check for full evidence and the audit bundle. Use |
![shipwright-agent[bot]](https://avatars.githubusercontent.com/u/140162738?s=60&v=4){kind=small}
| yield field_name, " ".join(val.strip() for val in values) | ||
| # -- merge dict-of-list values: union keys, per key concatenate then dedupe | ||
| # -- records, preserving first-seen order -- | ||
| elif strategy is CS.DICT_LIST_UNIQUE: |
There was a problem hiding this comment.
Shipwright · HIGH
The dedupe identity 'tuple(sorted(record.items()))' is computed twice per record (once for 'seen_ids', once for 'record_id') and rebuilds 'seen_ids' from scratch on every record vi
Impact: The dedupe identity 'tuple(sorted(record.items()))' is computed twice per record (once for 'seen_ids', once for 'record_id') and rebuilds 'seen_ids' from scratch on every record via a set comprehension over all previously seen records. For a chunk consolidating many elements with long origin lists, this is O(n^2) in the number of records and will degrade noticeably on large documents.
Suggested fix: Review the cited evidence, fix the risk if confirmed, and rerun Shipwright.
| yield field_name, list(ordered_unique_keys.keys()) | ||
| elif strategy is CS.STRING_CONCATENATE: | ||
| yield field_name, " ".join(val.strip() for val in values) | ||
| # -- merge dict-of-list values: union keys, per key concatenate then dedupe |
There was a problem hiding this comment.
Shipwright · HIGH
The DICT_LIST_UNIQUE dedupe uses 'tuple(sorted(record.items()))' as the record identity.
Impact: The DICT_LIST_UNIQUE dedupe uses 'tuple(sorted(record.items()))' as the record identity. This breaks if any record value is itself a dict or list (unhashable after tuple conversion), and silently treats records with non-string values or nested structures as distinct even when semantically equal. The type hint restricts to 'dict[str, str]', but nothing enforces it at runtime; a producer passing a nested dict will rai…
Suggested fix: Review the cited evidence, fix the risk if confirmed, and rerun Shipwright.
| The same metadata as the first text-split chunk but includes `.is_continuation = True`. | ||
| Unused for non-oversized pre-chunks since those are not subject to text-splitting. | ||
|
|
||
| A new object is produced on each call (not cached) because each continuation chunk needs |
There was a problem hiding this comment.
Shipwright · HIGH
'_chunk_metadata()' deep-copies only 'enrichment_origins', but other mutable metadata fields (e.g.
Impact: '_chunk_metadata()' deep-copies only 'enrichment_origins', but other mutable metadata fields (e.g. 'emphasized_text_contents', 'emphasized_text_tags', 'languages') remain shared across split chunks via the shallow copy. If any downstream code mutates those lists in place on one chunk, the mutation leaks to sibling chunks — the exact bug class this change fixes for 'enrichment_origins' but leaves open for other list…
Suggested fix: Review the cited evidence, fix the risk if confirmed, and rerun Shipwright.
1 participant
Summary
Adds a new
enrichment_originsfield toElementMetadatathat tracks which model wrote (or contributed to) each enriched attribute.What's new
ElementMetadata.enrichment_origins— a serializeddict[str, list[dict[str, str]]]mapping a written attribute name (e.g.text,text_as_html,embeddings) to a list of{"type", "provider", "model"}records, in application order. Authoring enrichments overwrite the list; additive enrichments append, preserving the prior author.ConsolidationStrategy.DICT_LIST_UNIQUE— a new chunking consolidation strategy fordict[str, list]fields. It unions keys across elements and, per key, concatenates then dedupes records while preserving first-seen order. Wired up as the strategy forenrichment_origins.Testing
test_elements.py).DICT_LIST_UNIQUEmerge behavior during chunking (test_base.py).Summary by cubic
Adds per-attribute enrichment provenance via
ElementMetadata.enrichment_originsand a dict-list merge strategy, and isolates per-chunk metadata so provenance updates can’t leak across split chunks.New Features
ElementMetadata.enrichment_origins: maps an attribute to a list of{"type","provider","model"}in application order. Authoring overwrites; additive appends.ConsolidationStrategy.DICT_LIST_UNIQUE: unions keys and per-key concat+dedupes while preserving order; used forenrichment_originsin chunking.Bug Fixes
enrichment_originsper chunk and setsis_continuationon later chunks, preventing any in-place updates from leaking to siblings or lazy continuations.Written for commit 7a03b8c. Summary will update on new commits.
Source merge-base:
dedf144192347a89bb70ddc8a896eeeaa7440be0Source head:
7a03b8c9886b6cbdbfcfe647984482f5daaa6724