Skip to content

About

VoiceShield AI - NBF-Lite anchored forensic audio identity

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

πŸ›‘οΈ VoiceShield AI

Zero-trust, real-time audio deepfake detection and mitigation platform for the Indian telecom context β€” built as a fully on-device Android app.

VoiceShield ships as an Android app that runs the whole pipeline on the phone: live calls are captured and scored by an on-device engine (acoustic model + prosodic XAI + enterprise watermark), and each incident is sealed into an on-device evidence block that an operator server anchors to a mandatory Hyperledger Fabric + IPFS (NBF) ledger β€” fail-closed, so a report is only ever handed off with a provable on-chain anchor. It is built for use by telecom providers, banks, contact centers, and parents (Child Shield) β€” everything runs on a free, open-licensed stack (attribution licenses retained; full inventory in backend/SOURCES_AND_TECHNOLOGY.md).

πŸ”΄ SIH-ready β€” all training data is from open licenses (FLEURS CC-BY-4.0, Common Voice CC-BY-4.0 β€” CC0 for v13 and earlier, self-generated TTS + real voice-clone impersonation via XTTS-v2/RVC/FreeVC). See backend/SOURCES_AND_TECHNOLOGY.md for full data sources, licenses, and compliance.

⚠️ This repository is the working web prototype of that Android app. It runs the identical pipeline in the browser/server so the architecture can be seen and tested end-to-end β€” browser mic β†’ the same engine β†’ the same ledger anchor. Every page you see here is a preview of how the app will behave on a phone.


πŸ€– VoiceShield on Android β€” architecture (three planes)

Detection is fully on-device; raw audio never leaves the phone. Only scalar evidence + hashes reach the operator server, which anchors every report to the mandatory NBF ledger.

β”Œβ”€ 1 Β· DATA PLANE  (Android, Kotlin) ────────────────────────────────┐
β”‚  AudioRecord 16 kHz mono int16 β†’ MPSC ring buffer (3 s / 1 s)      β”‚
β”‚  VAD + silence gate β†’ Kotlin DSP: FFT Β· autocorrelation pitch Β·    β”‚
β”‚  jitter/shimmer Β· phase continuity Β· watermark band 7.0–7.5 kHz    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                               β–Ό  (no network in the hot loop)
β”Œβ”€ 2 Β· MAIN ENGINE  (Android, on-device) ────────────────────────────┐
β”‚  onnxruntime-android + NNAPI Β· bundled AASIST-L INT8 (< 2 MB)      β”‚
β”‚  prosodic XAI Β· speaker voice-print Β· watermark short-circuit β†’ 0.0β”‚
β”‚  fusion: score = 0.7 Γ— model + 0.3 Γ— XAI β†’ risk band + mitigation  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                               β–Ό  scalar evidence + hashes only
β”Œβ”€ 3 Β· EVIDENCE LEDGER ──────────────────────────────────────────────┐
β”‚  on-device: Room + AES-256-GCM (Android Keystore) + local PoW blockβ”‚
β”‚  β†’ WorkManager retry β†’ POST /api/mobile/evidence β†’ server forensic β”‚
β”‚  PDF β†’ mandatory NBF-Fabric + IPFS anchor β†’ status = anchored      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

This website is the prototype surface. frontend/ streams browser-mic audio over WebSocket into backend/, which runs the same engine (backend/app/engine) and mirrors the same ledger β€” so the Data Plane boxes above become "browser mic β†’ WebSocket", while the engine and the fabric/IPFS anchor stay byte-identical.


🧠 What it does (in one picture)

Real Person OR AI Clone? ──▢ listen live / upload file
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Detection Pipeline (backend/app/engine)    β”‚
β”‚  β€’ Acoustic DL models:                       β”‚
β”‚      - Dhwani (Wav2Vec2 XLS-R + AASIST)      β”‚   ← trained on Indian languages
β”‚      - AASIST-L official pretrained          β”‚   ← ensemble
β”‚      - local AASIST-L (retrained, fallback)  β”‚
β”‚  β€’ XAI prosody layer:                        β”‚
β”‚      jitter / shimmer / phase continuity     β”‚
β”‚  β€’ Cross-session speaker consistency         β”‚
β”‚  β€’ Enterprise watermark (VoLTE-band pilot)   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                     β–Ό
   Risk Score (0–1)  ─▢  verdict + risk band
                     β–Ό
   Real-time alerts:  Telegram ・ Gmail ・ ntfy.sh
                      Fast2SMS ・ Webhook
                     β–Ό
Incident log  ─▢  I4C-ready forensic PDF
                      └── anchored on a proof-of-work
                          PoW block + NBF-Fabric (Vishvasya)
                          external anchor (hash + IPFS CID)

Key ideas

  • Multi-layer detection, not one model. Raw audio is scored by ensemble acoustic models AND by explainable prosodic features (jitter/shimmer/phase continuity). Both are combined: score = 0.7 Γ— model_prob + 0.3 Γ— xai_risk.
  • On-device by design (Android). The app scores every window on the phone with onnxruntime-android + NNAPI β€” no cloud round-trip, no raw audio leaving the device (DPDP-safe). The web prototype streams over WebSocket purely to demonstrate the same engine; the app itself uses no WebSocket.
  • Free alerts everywhere. Telegram, Gmail SMTP, ntfy.sh, Fast2SMS, and webhooks.
  • Tamper-evident reports. Every forensic PDF is hashed and anchored into a local proof-of-work blockchain ledger, then mirrored to a Hyperledger Fabric (MeitY NBF-Lite) external anchor with an AES-256-GCM ciphertext copy pinned on IPFS (raw forensic content never leaves operator custody β€” DPDP-safe).

✨ Feature list

Area What's included
Detection Dhwani (multilingual Indian deepfake model) + AASIST-L official ensemble, INT8 ONNX inference on CPU
Explainability Jitter %, Shimmer %, Phase Continuity, Pitch Stability, noise-floor dropouts
Real-time On-device by default (Android): the ring buffer feeds the engine locally. Web prototype: live analysis over WebSocket, 3 s windows / 1 s hop (300 ms/100 ms fallback). A latency profile governs the live hot loop: latency_profile=realtime (default) scores each window with the trained AASIST-L official + XAI prosody (VOICESHIELD_LATENCY_BUDGET_MS-gated, measured ~1.5-1.6 s/window on reference CPU); latency_profile=ensemble runs the full Dhwani ensemble live (seconds/window)
Risk Role-aware thresholds β€” Adult β‰₯ 0.85 critical, Child β‰₯ 0.70 critical (ADULT_THRESHOLD / CHILD_THRESHOLD in .env)
Alerting Telegram, Gmail SMTP, ntfy.sh, Fast2SMS, webhook β€” all optional
Escalation Suspected fraud β†’ optional DoT conduct hand-off (Sanchar Saathi / Chakshu / DIP) via CHAKSHU_DIP_WEBHOOK_URL; confirmed fraud β†’ I4C/1930 flow in the forensic PDF
Context Optional call-context enrichment (opt-in, CONTEXTUAL_ENRICHMENT=true): known-contact, caller reputation (escalation count), call-origin and transaction value adjust the risk before thresholds β€” fail-open when no context is provided
Verification Configurable mitigation workflows (backend/workflows.json, JSON β€” no code deploy): per-role Γ— risk-band actions + channels (pre-transaction call-back, MFA, supervisor escalation) with POST /api/incidents/{iid}/escalate feeding back caller reputation
SDK Official Python SDK (backend/sdk/voiceshield_sdk, sync + async) on an API contract defined in backend/sdk/voiceshield.proto (proto3) β€” REST + live WebSocket call sessions
On-device Android app (native Kotlin) β€” detection fully on-device via onnxruntime-android + NNAPI; deploy/edge/vs_edge.py CLI worker demonstrates the same on-device inference server-side
Android app Kotlin Β· Jetpack Compose Β· Material 3 Β· onnxruntime-android + NNAPI Β· Room Β· WorkManager Β· Android Keystore
Child Shield Lower threshold, auto-mute, protective overlay
Forensics I4C-ready PDF reports (IST timestamps, legal next steps)
Integrity SHA-256 + Merkle-root anchored to a PoW blockchain ledger
NBF anchor MANDATORY (fail-closed). Every block is ALSO cryptographically anchored on a MeitY National Blockchain Framework (Hyperledger Fabric) ledger with IPFS-pinned AES-256-GCM ciphertext. A forensic report is committed to the local ledger only after a real on-chain anchor succeeded (BLOCKCHAIN_EXTERNAL_ANCHOR=true + BLOCKCHAIN_ANCHOR_REQUIRED=true by default). Any gateway/provider failure raises BlockAnchorError and the report is refused β€” no unvalidated demo/pending report can be handed off as anchored. Explicitly disabling both flags restores the offline DemoAnchor shadow for laptop demos only
Speaker ID Enrollment + cross-session voice-print consistency (one-way, DPDP-safe)
Privacy Ephemeral RAM processing, zero disk for raw audio, scalar-only telemetry
i18n Dashboard UI in English / ΰ€Ήΰ€Ώΰ€¨ΰ₯ΰ€¦ΰ₯€ / ಕನ್ನ಑ (context-aware translations)
Languages Detection tuned for Hindi, English, Kannada (+ Tamil/Telugu/Malayalam via Dhwani)

πŸš€ Quick start (5 minutes)

Prerequisites

  • Python 3.10+ and Node.js 18+
  • ffmpeg (optional β€” only needed for AMR codec augmentation during training)

1. Backend

cd backend
python -m venv venv
venv\Scripts\activate            # Windows
# source venv/bin/activate       # Linux / Mac

pip install -r requirements.txt

2. Create the environment file and add your keys

copy .env.example .env           # Windows
# cp .env.example .env           # Linux / Mac

πŸ“„ Where do API keys go? β†’ backend/.env (this exact file). It is read by backend/app/config.py on startup, and it is already git-ignored so secrets are never committed.

The .env has clearly-marked sections. Here is exactly what goes where:

What Variable in backend/.env Where to get it (all FREE)
Telegram alerts TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID https://t.me/BotFather
Email alerts SMTP_USER, SMTP_PASSWORD, ALERT_EMAIL_TO Gmail App Password (2FA β†’ Security β†’ App Passwords)
Push alerts NTFY_TOPIC https://ntfy.sh (no signup)
SMS (India) FAST2SMS_API_KEY, FAST2SMS_TO_NUMBER https://www.fast2sms.com
Webhook WEBHOOK_URL any endpoint you control
Email API (alt) RESEND_API_KEY https://resend.com
Error tracking SENTRY_DSN https://sentry.io
Training data HF_TOKEN https://huggingface.co/settings/tokens (needed only for private gated repos; FLEURS is public)
Telemetry store DATABASE_URL reserved for optional Neon PostgreSQL β€” detection telemetry persists to local SQLite (backend/voiceshield.db) regardless

Blockchain API keys β†’ the same file, section β›“ BLOCKCHAIN REPORT LEDGER:

  • The report ledger is local and needs NO external key (BLOCKCHAIN_DIFFICULTY is just the PoW difficulty, default 4).
  • BLOCKCHAIN_RPC_URL, BLOCKCHAIN_EXPLORER_API_KEY, BLOCKCHAIN_ANCHOR_ADDRESS are optional anchors if you later pin block hashes to a public chain.
  • NBF-Lite external anchor β€” MANDATORY by default (fail-closed): BLOCKCHAIN_EXTERNAL_ANCHOR=true (default) selects the real Hyperledger Fabric + IPFS provider through the gateway in deploy/nbf-fabric/ β€” see the NBF anchor section. BLOCKCHAIN_ANCHOR_REQUIRED=true (default) means a forensic report is only written to the local ledger after the on-chain anchor succeeded: any gateway/provider failure raises BlockAnchorError and the report endpoint refuses (503), and verification of an unanchored report reports it as NOT blockchain-anchored. Configure NBF_GATEWAY_URL=http://<host>:4000 + the NBF_* settings. Offline dev/demo only β€” set BLOCKCHAIN_EXTERNAL_ANCHOR=false and BLOCKCHAIN_ANCHOR_REQUIRED=false to use the local DemoAnchor shadow (status demo, never anchored) so the repo works on a laptop with no Fabric node.
  • Exactly one variable per line, restart the backend after editing.

3. Start the backend

python -m uvicorn app.main:app --host 0.0.0.0 --port 8000
# API docs at  http://localhost:8000/docs

4. Start the frontend

cd frontend
npm install
npm run dev
# Dashboard at  http://localhost:5173

5. One-command start (Windows)

Double-click start_all.bat (backend :8000 + frontend :5173).


πŸ”Œ REST API (all endpoints)

Transport contract. The Android app uses REST + gRPC only β€” no WebSocket in the app. The /ws/stream/{call_id} live-stream surface exists for the web prototype (this site) and the Python SDK. The app syncs evidence blocks and verifications through /api/mobile/evidence, /api/blockchain/verify/... and the report endpoints.

Method Route What it does
POST /api/analyze Analyze an uploaded audio file (full windowed sweep)
POST /api/detect Quick detection on a single audio payload
WS /ws/stream/{call_id} Live streaming detection during a call
GET /api/sessions List sessions
GET /api/sessions/{sid} Session detail
GET /api/sessions/{sid}/windows Per-window telemetry
GET /api/incidents List incidents
GET /api/incidents/{iid} Incident detail
POST /api/incidents/{iid}/ack Acknowledge an incident
GET /api/incidents/{iid}/report Incident forensic PDF
GET /api/report/{call_id} Forensic PDF by call id
GET /api/blockchain Ledger status + all blocks
GET /api/blockchain/{index} Single block
GET /api/blockchain/verify/call/{call_id} Full tamper-evidence check
GET /api/blockchain/verify/block/{index} PoW + hash integrity check
GET /api/blockchain/onchain NBF anchor summary (count + per-block link)
GET /api/blockchain/onchain/{call_id} External anchor status/verification for one call
POST /api/blockchain/retry/{call_id} Re-attempt a pending NBF-Fabric/IPFS anchor
POST /api/speakers/register Enroll a speaker voice-print
POST /api/incidents/{iid}/escalate Confirm fraud β†’ escalate + bump caller reputation for future calls
GET /api/workflows Current mitigation workflow rules (adult/child Γ— risk band)
POST /api/mobile/evidence (planned) Android app evidence-block sync β†’ server forensic PDF + NBF anchor (WorkManager retry)
GET /health, /info Health / model status

/api/analyze also accepts optional context fields (multipart form: caller, origin, txn_value, txn_category, known_contact); the live WebSocket accepts the same as query params. See backend/sdk/SDK_README.md for the SDK + proto contract.


πŸ“ Project structure

voiceshield/
β”œβ”€β”€ backend/
β”‚   β”œβ”€β”€ app/
β”‚   β”‚   β”œβ”€β”€ main.py              # FastAPI entry point (all REST endpoints)
β”‚   β”‚   β”œβ”€β”€ config.py            # Pydantic settings ↔ backend/.env
β”‚   β”‚   β”œβ”€β”€ grpc/                # gRPC servicer, proto pb2, server (port 50051)
β”‚   β”‚   β”œβ”€β”€ models/              # AASIST-L architecture + ONNX inference + export
β”‚   β”‚   β”œβ”€β”€ engine/              # dhwani, aasist_official, prosody, fusion,
β”‚   β”‚   β”‚                        #   ring_buffer, watermark, speaker, risk,
β”‚   β”‚   β”‚                        #   pipeline
β”‚   β”‚   β”œβ”€β”€ context/             # call-context enrichment (opt-in)
β”‚   β”‚   β”œβ”€β”€ streaming/           # WebSocket live-call endpoint
β”‚   β”‚   β”œβ”€β”€ mitigation/          # Role-aware alerts + configurable workflows
β”‚   β”‚   β”‚                        #   (workflows.py + ../workflows.json), escalate
β”‚   β”‚   β”œβ”€β”€ forensics/           # I4C-ready PDF reports
β”‚   β”‚   β”œβ”€β”€ blockchain/          # PoW report ledger (local) + external NBF anchor
β”‚   β”‚   β”‚   β”‚                    #   external_anchor.py (AES-256-GCM, IPFS, gateway)
β”‚   β”‚   β”œβ”€β”€ compliance/          # DPDP Act posture helpers
β”‚   β”‚   └── store.py             # In-memory / Neon persistence
β”‚   β”œβ”€β”€ sdk/                     # voiceshield_sdk (sync/async) + voiceshield.proto
β”‚   β”‚   β”‚                        #   + grpc.py client (SDK ↔ app/grpc servicer)
β”‚   β”œβ”€β”€ workflows.json           # editable mitigation workflow rules (bank/enterprise/gov)
β”‚   β”œβ”€β”€ training/                # augmentations, dataset, train, evaluate,
β”‚   β”‚   β”‚                        #   fetch_indic (FLEURS + TTS spoof), merge_protocols
β”‚   β”œβ”€β”€ models/                  # ONNX + .onnx.data (aasist_l, dhwani β€” git-ignored, regenerate via export_onnx)
β”‚   β”œβ”€β”€ hf_models/aasist/        # official AASIST-L pretrained repo (MIT)
β”‚   β”œβ”€β”€ data/                    # training + evaluation audio (git-ignored)
β”‚   β”œβ”€β”€ checkpoints/             # trained weights + training_log.txt
β”‚   β”œβ”€β”€ .env.example             # ← copy to .env
β”‚   β”œβ”€β”€ .env                     # ← ADD YOUR API KEYS HERE
β”‚   β”œβ”€β”€ SOURCES_AND_TECHNOLOGY.md  # data licenses + tech inventory (SIH)
β”‚   └── requirements.txt
β”œβ”€β”€ deploy/
β”‚   └── edge/                    # vs_edge.py β€” on-device inference worker (CLI)
β”œβ”€β”€ mobile/                      # Android app (Kotlin β€” planned): data plane + engine + ledger sync
β”œβ”€β”€ frontend/
β”‚   β”œβ”€β”€ src/
β”‚   β”‚   β”œβ”€β”€ pages/               # Dashboard, Incidents, Reports
β”‚   β”‚   β”œβ”€β”€ components/          # RiskGauge, Radar, SessionCard, ShieldBanner, ui
β”‚   β”‚   β”œβ”€β”€ i18n.tsx             # en / hi / kn translations
β”‚   β”‚   └── ...
└── start_all.bat                # starts backend + frontend

πŸŽ“ Training the model (SIH-compliant)

Training uses only open-license data:

  • Bonafide (real human voices) β€” Google FLEURS (CC-BY-4.0) for Hindi / English / Kannada (+ Common Voice CC-BY-4.0 in the cloud pipeline).
  • Spoof β€” REAL A.I. voice impersonation β€” the cloud pipeline clones the actual bonafide speakers with Coqui XTTS-v2 (CPML-1.0, research/non-commercial hackathon use) so the "attacker" speaks in the victim's cloned voice. RVC / FreeVC (MIT) add conversion-style clones. This targets the actual SIH problem (voice impersonation), not just robocall TTS. ⚠️ XTTS is non-commercial-only; a commercial production retrain should regenerate the spoof pool with MIT cloners (RVC/FreeVC/Chatterbox).
  • Auxiliary pure-TTS class β€” edge-tts / gTTS (minority, ~9 Indian languages).
  • Augmentations β€” G.711, AMR-NB/WB, packet-loss, bandwidth, and noise that simulate Indian telecom networks.

Option A β€” Cloud (recommended, free GPU, big corpus)

See cloud/ β€” ready-to-run Google Colab + Kaggle notebooks:

colab_01_build_dataset.ipynb   # real voices + REAL XTTS/RVC/FreeVC clones β†’ train/val CSVs
colab_02_train_wav2vec2.ipynb  # wav2vec2-base audio classifier (fits free T4)
colab_03_train_xlsr.ipynb      # XLSR-300m for best Indic cross-lingual accuracy
kaggle_train_wav2vec2.py       # Kaggle equivalent

Option B β€” On-device (CPU, small set, AASIST-L fallback)

cd backend
# 1. (optional) fetch more data
python -m training.fetch_indic --language hindi   --bonafide 200 --spoof 200 --out data/indic
python -m training.fetch_indic --language english --bonafide 200 --spoof 200 --out data/indic
python -m training.fetch_indic --language kannada --bonafide 200 --spoof 200 --out data/indic

# 2. build the expanded SIH dataset (real speech + cloned/TTS spoofs)
python -m training.build_sih_dataset --out data/indic --bonafide 200 --spoof 150
python -m training.merge_protocols --data_dir data/indic --output data/indic/merged

# 3. train AASIST-L
python -m training.train --train_dir data/indic --train_protocol data/indic/merged_train.txt \
                         --val_dir data/indic --val_protocol data/indic/merged_val.txt \
                         --save_dir checkpoints --epochs 30

# 4. export to ONNX + INT8
python -m app.models.export_onnx checkpoints/best_model.pth

⚠️ The retrained local AASIST-L is the fallback detector. The primary production path uses the Dhwani and official AASIST-L pretrained models (loaded automatically when present).

πŸ“¦ Trained weights (checkpoints/*.pth, models/aasist_l.onnx) are git-ignored artifacts. On a fresh clone, regenerate the ONNX fallback with python -m app.models.export_onnx checkpoints/best_model.pth; until then inference returns a neutral 0.5.


⚠️ Suspected vs confirmed fraud (two flows): when the risk score crosses the threshold, flagged-call metadata can be routed to the DoT side (Chakshu / Sanchar Saathi / Digital Intelligence Platform) as suspected fraud for network-level action (CHAKSHU_DIP_WEBHOOK_URL). If the user has actually lost money / is a cyber-crime victim, the forensic PDF directs them to the confirmed post-fraud reporting path β€” 1930 helpline / cybercrime.gov.in / RBI Sachet (https://sachet.rbi.org.in). Chakshu/DIP are citizen-facing portals (no public API) β€” the webhook is your documented integration point.


πŸ”’ Privacy & compliance (DPDP Act 2023)

  • βœ… Ephemeral processing β€” audio buffers live in RAM, overwritten every window
  • βœ… Zero disk storage β€” raw audio is never written to disk by the live path
  • βœ… Minimized telemetry β€” only scalar sub-scores cross the wire/WebSocket
  • βœ… One-way speaker embeddings β€” audio can never be reconstructed
  • βœ… Audit trail β€” non-PII scalar data only

πŸ“Š Latency budget (per window) β€” measured, two profiles

The hot loop is app.engine.pipeline.analyze_window β€” the exact callable the WebSocket live driver, the gRPC servicer, and /api/analyze run per window. It self-reports its latency_ms, and tests/test_latency_gate.py enforces the real-time budget fail-hard on the real hot loop (median of β‰₯20 post-warmup runs; VOICESHIELD_LATENCY_BUDGET_MS is the only way to raise it, and it is printed when used).

Profile Models per window Measured (3 s window, reference CPU) Gate
realtime (default) trained AASIST-L official + XAI prosody (legacy ONNX only as last resort) XAI ~0.77 s, model ~0.78 s β†’ total ~1.5-1.6 s median latency_budget_ms=2000 (default) fail-hard
ensemble (opt-in streaming; always for /api/analyze + gRPC Analyze) Dhwani + AASIST-L official (+ XLS-R in file analyze) Dhwani ~1.9 s, ensemble ~4.1 s, +XLS-R ~5.7 s (cold load ~72 s) No streaming contract β€” bounded batch, sized by the SDK RPC timeout (default 120 s)

The original README table (ingest ≀7 / prosody ≀20 / model ≀35 / fusion ≀16 = ≀78 ms total) was retired because it is physically unreachable on CPU: librosa.pyin-based prosody alone measures ~0.77 s and the trained AASIST-L official ~0.78 s. This section now documents the real numbers instead of a fabricated budget. Windows are 3 s / 1 s hop by default (300 ms / 100 ms fallback when no 3 s model is present). latency_profile is set in backend/.env (LATENCY_PROFILE) or VOICESHIELD_-prefixed env; the forensic /api/analyze surface always uses the full ensemble regardless of profile.


β›“ NBF anchor network (MeitY Vishvasya)

Every locally-mined report block is also committed to a permissioned Hyperledger Fabric ledger through an NBF-Lite-compatible REST gateway, with the report PDF stored on IPFS as ciphertext.

Forensic PDF ──▢ AES-256-GCM ──▢ IPFS /store ──▢ CID
                                      β”‚
Local PoW block ──▢ NBF gateway ──▢ Fabric chaincode AnchorReport
                  (block_hash, merkle_root, file_sha256,
                   ipfs_cid, enc_alg, key_fp)
  • Private by design: only hashes / CID / key fingerprint go on the ledger; the raw PDF stays encrypted under an operator-held master key (reports/keys/org_master.key). Ciphertext is pinned to IPFS so the full record is recoverable and tamper-evident without exposing voice content.
  • Fail-closed (MANDATORY by default): BLOCKCHAIN_EXTERNAL_ANCHOR=true + BLOCKCHAIN_ANCHOR_REQUIRED=true mean a forensic report is committed to the local PoW ledger only after the on-chain anchor succeeded. If the Fabric network / gateway is unreachable, anchor_report raises BlockAnchorError, the report endpoints return 503 ("not blockchain-anchored"), the block is NOT inserted, and verify_report reports the call as not anchored. No unanchored evidence can be handed off as forensic. The DemoAnchor shadow and 'pending'/retryable degrade are available only when the operator explicitly disables both flags for offline dev/demo (BLOCKCHAIN_EXTERNAL_ANCHOR=false + BLOCKCHAIN_ANCHOR_REQUIRED=false); even then the demo status is reported truthfully (demo, never anchored). Production I4C / judiciary hand-off runs with the mandatory posture on.
  • Deploy (free tier): see deploy/nbf-fabric/ β€” docker compose up -d on Oracle Cloud ARM A1 free tier (IPFS + Fabric 2.2 peer/orderer (cryptogen identities, no CA) + trimmed nbf-samplerest gateway + voiceshield-report.go chaincode) or WSL2 Ubuntu on a laptop. Zero cloud spend either way.
  • Ministry alignment: MeitY's National Blockchain Framework (Vishvasya, CDAC/MeitY) is the same substrate NBF-Lite kits teach; anchors at NIC DCs (Bhubaneswar/Pune/Hyderabad) have already authenticated 34 Cr+ documents, and the same framework is being used for telecom blockchain (SMS/Spam enforcement across 1.13 L entities with RBI/SEBI/NIC/C-DAC).
  • Live-verified: anchored end-to-end on 2026-09-19 (Fabric AnchorReport tx-id tx-test-1, IPFS CID QmZimdoSUNhXCrt…48kBa, GET /api/blockchain/onchain/e2e-NBF-LIVE-01 β†’ verified:true, chain VoiceShieldAIV1 height 14, 78 tests passing) β€” see docs/nbf-live-demo.md.

πŸ“œ License

The VoiceShield AI code is MIT. Third-party components used and their licenses are documented in backend/SOURCES_AND_TECHNOLOGY.md.

πŸ™ Acknowledgments

  • AASIST: Jung et al., Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks (ICASSP 2022)
  • Dhwani: Ayush2635, Multilingual Deepfake Audio Detection (Wav2Vec2 XLS-R + AASIST)
  • FLEURS: Google, CC-BY-4.0
  • Indian Cyber Crime Coordination Centre (I4C) forensic reporting standards
  • Dept. of Telecommunications β€” Sanchar Saathi / Chakshu / Digital Intelligence Platform (DIP): downstream "suspected fraud" escalation reference (citizen-facing; we integrate via our own documented webhook, no public API)
  • RBI Sachet (https://sachet.rbi.org.in): reporting of suspected unauthorised financial entities, referenced in the forensic PDF
  • IndiaAI Mission (MeitY): compute/dataset-grant program for public-interest AI β€” referenced as a training-capacity pathway (no API)

About

VoiceShield AI - NBF-Lite anchored forensic audio identity

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages