fix(harvester): handle legacy CDS recid checks before create - #931
fix(harvester): handle legacy CDS recid checks before create#931TahaKhan998 wants to merge 1 commit into
Conversation
f21a51c to
7b3fef1
Compare
00012a8 to
c657a94
Compare
c657a94 to
7a31dab
Compare
7a31dab to
6eea851
Compare
490ad97 to
a3d82f6
Compare
a3d82f6 to
58f8a06
Compare
| "title": "Fragmentation through Heavy and Light-flavor Measurements with the LHC ALICE Experiment", | ||
| "publication_date": "2024", | ||
| "languages": [{"id": "eng", "title": {"en": "English", "da": "Engelsk"}}], | ||
| "identifiers": [ |
There was a problem hiding this comment.
I am not fully sure why this is removed from expected result - the results should still have CDS legacy recid inside the metadata, if this not how the records are created in the end then it might be incorrect - if record had legacy id and is ingested, it should be harvested with the legacy id in the metadata
There was a problem hiding this comment.
we still map the cds recid into metadata.identifiers when it’s on the inspire record, this test is a create test though and after this pr if the fixture still has a numeric cds recid the matcher thinks it’s still on old cds and skips so nothing gets created and the test fails, that’s why we strip them from the mock inspire payload with drop_legacy_cds_ids and why they’re gone from expected_result, the input doesn’t have them anymore. migrated records still keep the legacy id, that’s what add_legacy_recid is for in the update tests.
58f8a06 to
b402a30
Compare
Closes #907
This PR changes how the INSPIRE harvester handles records with a legacy CDS recid and avoids duplicates between old CDS and CDS-RDM. If the incoming record has an integer legacy recid, we first check pidstore and update when it is found. If it is not found, we check old CDS directly. We raise an error when it redirects to repository.cern or returns 404. We skip with a warning when the record is still only in legacy. If there is no legacy recid in the incoming record, the existing matching flow stays the same. If INSPIRE sends more than one CDS recid (merged records keep both), we check each in pidstore and on old CDS. Two different pidstore hits are treated as multiple records match.