PIR: Forward extras on extracted profiles - #9353
Open
noisysocks wants to merge 1 commit into
Open
Conversation
8 tasks
noisysocks
marked this pull request as draft
July 31, 2026 01:22
Adds an open extras map to extracted profiles and to their addresses. Native never looks inside it: it reads extras off the extract message, stores them, and hands them back verbatim when an opt-out form is filled. A new field can then be added in the broker config alone, rather than needing a change and a release on all three platforms. - Parse profile-level and address-level extras from the extract response, strictly as Map<String, String>, so malformed values fail loudly through the existing message-parsing error path. - Persist them: new extras column on pir_extracted_profiles via a v17 auto migration, with address extras carried inside the existing addresses JSON. - Send addresses, extras, age, phoneNumbers, relatives and identifier to fillForm. Previously only name, profileUrl, fullName and email were sent, so addresses were dropped entirely. - Refresh stored profiles on re-scan rather than ignoring them. Extras are merged, so a key missing from a later scrape keeps the value already stored and a pending opt-out doesn't lose fields when a broker changes its markup. - Exclude extras from profile identity comparisons, so shipping a config that starts scraping a new field cannot make stored records look removed. Tech design: https://app.asana.com/1/137249556945/project/481882893211075/task/1216773212741757 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
noisysocks
force-pushed
the
randerson/pir-extras-passthrough
branch
from
August 3, 2026 05:03
5b4c506 to
38f5db8
Compare
noisysocks
changed the title
noisysocks
marked this pull request as ready for review
August 3, 2026 05:10
This comment was marked as low quality.
This comment was marked as low quality.
This comment was marked as low quality.
This comment was marked as low quality.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task/Issue URL:
https://app.asana.com/1/137249556945/task/1216773212741753
Tech Design URL (if applicable):
https://app.asana.com/1/137249556945/project/481882893211075/task/1216773212741757
API Proposals URL(s) (if applicable):
Description
Support
extrasin extracted profiles and addresses, pass them along tofillForm, and update stored profiles with the freshly scraped profile. This lessens the need for an app release when an opt out form requires a new field we don't have.Steps to test this PR
cyberbackgroundchecks.comand replace the JSON with: https://dub.duckduckgo.com/duckduckgo/dbp-api/blob/2fb2bb29f5d3f67bc6c70098ca36db7ca4a100b9/dbp-json/data/json/cyberbackgroundchecks.com.jsonNote
Medium Risk
Touches encrypted PIR local storage and changes scan persistence from insert-only to insert-or-update, which affects opt-out data continuity; migration is auto-generated with a dedicated test.
Overview
Adds extras maps on extracted profiles and addresses so broker configs can scrape fields outside the native schema and pass them into opt-out fillForm without an app release.
Persistence: Room v17 adds an
extrascolumn (default{}) onpir_extracted_profiles, with Moshi converters forMap<String, String>.saveExtractedProfilesreplacessaveNewExtractedProfiles: new rows insert as before; matching rows are updated viarefreshedWith, keeping local id, date added, and deprecated flag while merging profile and per-address extras (scraped keys win; missing keys retain stored values).Pipeline: Scan success maps script
extrasonto models;toParamsforwards age, addresses, phones, relatives, identifier, and extras to C-S-S. Profile identity comparisons (matches, job-record keys) ignore address extras so new scraped fields do not create duplicate records.Tests: Room migration 16→17, repository refresh/merge behavior, and PirUtils helpers; dev results UI shows extras.
Reviewed by Cursor Bugbot for commit 38f5db8. Bugbot is set up for automated code reviews on this repo. Configure here.