fix: allow more flexibility in choosing models used for PII removal

[dennyabrain]

Summary

This PR allows configuring the model used for by recognizer within Presidio. This removed the issue with false positives in hinglish text messages.

We have added 2 new parameters to the config for PII validator - nlp_engine_type and model_name. A sample initialization of pii validator now looks like this -

pii_validator = PIIRemover(
    entity_types=[
        "EMAIL_ADDRESS",
        "PHONE_NUMBER",
        "IN_AADHAAR",
        "PERSON",
        "IN_PAN",
        "LOCATION",
        "IN_VOTER",
        "IN_PASSPORT",
        "IN_VEHICLE_REGISTRATION",
        "URL",
    ],
    nlp_engine_type="transformers",
    model_name="dslim/bert-base-NER-uncased"
)

Since it helps to pre-download the models used by the validator, we are doing so in the Dockerfile. For now I have downloaded both models - one used by nlp_engine_type="spacy" and nlp_engine_type="transformers", we can probably choose one or a few that suits us in production. The changes to Dockerfile are indicative of what sort of changes are needed to support this PR and can use further scrutiny and discussion with the larger team.

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

Please add here if any other information is required for the reviewer.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 64d26e03-55d8-452a-a444-ef2c76a9c277

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Prajna1999]

@dennyabrain is this PR ready for review?

[dennyabrain]

@Prajna1999 You can start reviewing now.