Skip to content

fix(harvester): handle legacy CDS recid checks before create - #931

Open
TahaKhan998 wants to merge 1 commit into
CERNDocumentServer:masterfrom
TahaKhan998:fix/issue-907-harvester-legacy-duplicates
Open

fix(harvester): handle legacy CDS recid checks before create#931
TahaKhan998 wants to merge 1 commit into
CERNDocumentServer:masterfrom
TahaKhan998:fix/issue-907-harvester-legacy-duplicates

Conversation

@TahaKhan998

@TahaKhan998 TahaKhan998 commented Aug 18, 2026

Copy link
Copy Markdown

Closes #907
This PR changes how the INSPIRE harvester handles records with a legacy CDS recid and avoids duplicates between old CDS and CDS-RDM. If the incoming record has an integer legacy recid, we first check pidstore and update when it is found. If it is not found, we check old CDS directly. We raise an error when it redirects to repository.cern or returns 404. We skip with a warning when the record is still only in legacy. If there is no legacy recid in the incoming record, the existing matching flow stays the same. If INSPIRE sends more than one CDS recid (merged records keep both), we check each in pidstore and on old CDS. Two different pidstore hits are treated as multiple records match.

@TahaKhan998
TahaKhan998 force-pushed the fix/issue-907-harvester-legacy-duplicates branch 16 times, most recently from f21a51c to 7b3fef1 Compare August 19, 2026 11:23
Comment thread site/cds_rdm/inspire_harvester/writer.py Outdated
Comment thread site/cds_rdm/inspire_harvester/writer.py Outdated
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
Comment thread site/cds_rdm/inspire_harvester/writer.py
Comment thread site/cds_rdm/inspire_harvester/writer.py Outdated
Comment thread site/cds_rdm/inspire_harvester/writer.py Outdated
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py
@TahaKhan998
TahaKhan998 force-pushed the fix/issue-907-harvester-legacy-duplicates branch 4 times, most recently from 00012a8 to c657a94 Compare August 21, 2026 08:30
Comment thread site/cds_rdm/inspire_harvester/load/validator.py Outdated
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
@TahaKhan998
TahaKhan998 force-pushed the fix/issue-907-harvester-legacy-duplicates branch from c657a94 to 7a31dab Compare August 21, 2026 14:24
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
@TahaKhan998
TahaKhan998 force-pushed the fix/issue-907-harvester-legacy-duplicates branch from 7a31dab to 6eea851 Compare August 21, 2026 14:42
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
@TahaKhan998
TahaKhan998 force-pushed the fix/issue-907-harvester-legacy-duplicates branch 3 times, most recently from 490ad97 to a3d82f6 Compare August 21, 2026 15:47
Comment thread site/cds_rdm/inspire_harvester/load/matcher.py Outdated
@TahaKhan998
TahaKhan998 force-pushed the fix/issue-907-harvester-legacy-duplicates branch from a3d82f6 to 58f8a06 Compare August 21, 2026 15:52
"title": "Fragmentation through Heavy and Light-flavor Measurements with the LHC ALICE Experiment",
"publication_date": "2024",
"languages": [{"id": "eng", "title": {"en": "English", "da": "Engelsk"}}],
"identifiers": [

@kpsherva kpsherva Aug 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not fully sure why this is removed from expected result - the results should still have CDS legacy recid inside the metadata, if this not how the records are created in the end then it might be incorrect - if record had legacy id and is ingested, it should be harvested with the legacy id in the metadata

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we still map the cds recid into metadata.identifiers when it’s on the inspire record, this test is a create test though and after this pr if the fixture still has a numeric cds recid the matcher thinks it’s still on old cds and skips so nothing gets created and the test fails, that’s why we strip them from the mock inspire payload with drop_legacy_cds_ids and why they’re gone from expected_result, the input doesn’t have them anymore. migrated records still keep the legacy id, that’s what add_legacy_recid is for in the update tests.

@TahaKhan998
TahaKhan998 force-pushed the fix/issue-907-harvester-legacy-duplicates branch from 58f8a06 to b402a30 Compare August 25, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harvester: avoiding duplicates between systems

2 participants