Skip to content

About

ML security firewall detecting Visual Prompt Injections & typographic jailbreaks in multimodal VLMs

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Ar-Rasheed Smart University Logo

Visual Prompt Injection Detection in Large Vision-Language Models

An end-to-end Machine Learning security firewall detecting visual prompt injection, typographic camouflage, and adversarial instruction overrides targeting Multimodal Large Language Models (MLLMs).

Python Version Scikit-Learn OpenCV Tesseract Benchmark Status


📖 About The Project

As Multimodal Large Language Models (MLLMs such as GPT-4V, Claude 3.5 Sonnet, and Gemini) become widely integrated into enterprise systems, they introduce a critical attack surface: Visual Prompt Injection (VPI).

Attackers can embed covert text instructions inside images—ranging from visible memos and signage to extreme low-contrast typographic camouflage ($\Delta\text{RGB} \le 10$) invisible to human operators. When processed by a downstream VLM, these instructions hijack the agent's behavior, leading to system prompt exfiltration, unauthorized API execution, and safety guardrail bypasses.

This project implements an independent, lightweight ML security gateway deployed upstream of the VLM. The firewall:

  1. Normalizes and enhances image contrast via Computer Vision (adaptive Otsu binarization and thresholding) to defeat typographic camouflage.
  2. Extracts textual payloads using Optical Character Recognition (OCR).
  3. Analyzes semantic intent with n-gram TF-IDF vectorization.
  4. Classifies threats with a Linear Support Vector Machine (LinearSVC) tuned specifically to maximize Recall, preventing zero-day adversarial payloads from compromising the multimodal model.

🏛️ System Architecture

Input Image ──► [CV Preprocessing & Binarization] ──► [Tesseract OCR] ──► [TF-IDF Vectorizer] ──► [Linear SVM] ──► SAFE (Forward to VLM)
                                                                                                    │
                                                                                                    └──► MALICIOUS (Block & Alert)

System Architecture Diagram


📊 Key Evaluation Results

The pipeline is validated against both In-Distribution synthetic datasets (5,000 images incorporating real-world AdvBench jailbreaks) and Out-of-Distribution real-world benchmarks (Meta's official CyberSecEval 3 dataset).

Quantitative Performance

Dataset / Benchmark Accuracy Precision Recall F1-Score ROC-AUC
Synthetic Test Set (In-Distribution) 100.0% 100.0% 100.0% 100.0% 1.000
Meta CyberSecEval 3 (Out-of-Distribution) 86.7% 100.0% 80.0% 88.9% 1.000

Note: Recall is strictly prioritized over precision to minimize False Negatives, ensuring malicious payloads never leak through to the target VLM.


📈 Visualizations & Diagnostics

1. Confusion Matrix & ROC Curve

Evaluated on the synthetic test partition with zero false negatives and zero false positives.

Confusion Matrix ROC Curve

2. Semantic K-Means Clustering (2D PCA)

Unsupervised clustering ($k=3$) over the TF-IDF feature space naturally discovers three distinct semantic attack profiles:

  • Cluster 0: Benign Notices & Signage (meeting, invoice, exit, caution)
  • Cluster 1: System Prompt Exfiltration (system, instructions, verbatim, print, reveal)
  • Cluster 2: Command Injection & Jailbreaks (dan, ignore, bypass, curl, bash, override)

K-Means Clusters PCA

3. Top Malicious Feature Weights

Linear SVM coefficient analysis highlights discriminative keywords driving adversarial detection:

Top Malicious Words


⚙️ Installation & Setup

Prerequisites

  • Python 3.10 or higher
  • System Tesseract-OCR engine:
    # Debian/Ubuntu
    sudo apt-get install tesseract-ocr
    
    # macOS
    brew install tesseract

Environment Setup

# Clone the repository
git clone https://github.com/odai-dev/visual_prompt_injection_detection.git
cd visual_prompt_injection_detection

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

💻 Usage

The CLI tool src/main.py provides complete control over training, live inference, and evaluation:

1. Train the Entire Pipeline

Generates synthetic data, executes OCR extraction, fits TF-IDF vectorizers, trains Linear SVM & Random Forest classifiers, performs K-Means clustering, and renders all diagnostic figures:

python src/main.py --train

2. Live Image Prediction (Firewall Gateway)

Pass any candidate image through the preprocessing, OCR, and classification pipeline:

python src/main.py --predict path/to/sample_image.png

Output: Returns safety verdict (SAFE vs MALICIOUS), confidence probability, and extracted payload.

3. Benchmark Evaluation

Re-evaluates the serialized model (models/best_supervised_model.joblib) against the synthetic test set and Meta's CyberSecEval 3:

python src/main.py --evaluate

4. Run Test Suite

pytest tests/

📁 Repository Structure

├── Ar-rasheed_uni_logo.png                  # University insignia
├── Visual_Prompt_Injection_Detection_Report.pdf  # Final compiled academic thesis/report
├── Visual_Prompt_Injection_Detection.ipynb  # Interactive demonstration notebook
├── report.md                                # Markdown source for the academic paper
├── requirements.txt                         # Python dependencies
├── src/
│   ├── main.py                              # CLI interface entry point
│   ├── data_collection.py                   # Automated synthetic data generation engine
│   ├── preprocessing.py                    # Computer Vision filtering & OCR pipeline
│   ├── feature_extraction.py               # TF-IDF n-gram vectorization
│   ├── modeling.py                          # Supervised Linear SVM & Random Forest
│   └── clustering.py                        # Unsupervised K-Means semantic grouping
├── models/
│   ├── best_supervised_model.joblib         # Serialized Linear SVM classifier
│   ├── tfidf_vectorizer.joblib              # Fitted TF-IDF feature extractor
│   └── kmeans_model.joblib                  # Serialized K-Means clustering model
├── results/
│   ├── figures/                             # Generated charts (ROC, Confusion Matrix, PCA)
│   ├── predictions/                         # Inference outputs & CyberSecEval 3 predictions
│   └── reports/                             # Detailed metrics JSON and final summary markdown
└── tests/                                   # Unit and integration test suites

🎓 Academic Attribution

  • Author: Odai Gubran
  • Institution: Ar-Rasheed Smart University
  • Supervisor: Eng. Sondos Saif
  • Subject: Visual Prompt Injection Detection in Large Vision-Language Models

For comprehensive methodology details and formal literature review, see Visual_Prompt_Injection_Detection_Report.pdf and report.md.

About

ML security firewall detecting Visual Prompt Injections & typographic jailbreaks in multimodal VLMs

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages