Skip to content
AikyamLabPublic

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

243 Commits

Folders and files

Repository files navigation

MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations

arXiv Dataset

Official code for MEA (Multimodal Explainability Agent), a multi-agent framework that turns post-hoc explanation tools into faithful natural-language explanations across tabular, text, and vision models. Paper: arXiv:2610.02480 · Benchmark: MEA-Benchmark on Hugging Face.

Overview

MEA consists of two collaborating agents in a Proposer → Actor pipeline:

  • Proposer: Selects and configures explanation tools based on the question and modality
  • Actor: Turns the tool outputs into a natural-language explanation grounded in model behavior; it is optimized end-to-end against faithfulness

Faithfulness is evaluated by a perturbation-based protocol (not an agent). The benchmark has 10 question types (Q1–Q10) spanning feature attribution, counterfactual reasoning, and spurious feature detection, across tabular, text, and vision. Optimizing against faithfulness rewards with a modality-adaptive penalty improves faithfulness over the untrained backbone by +28% (tabular), +21% (text), and +34% (vision), and MEA outperforms post-hoc explainers, agentic, and closed-source baselines across six datasets.

Question Types

Type Description
Q1 Which part of the input was most responsible for the prediction?
Q2 Which part of the input was least responsible for the prediction?
Q3 Which parts distinguish the prediction from the next-best alternative?
Q4 Why are instances A and B given different predictions?
Q5 If a certain part is masked, would the prediction change?
Q6 How should the instance change to flip the prediction to a target class?
Q7 If one important part is removed/changed, how would the prediction change?
Q8 Is there an irrelevant (spurious) part causing the wrong prediction?
Q9 What shared feature makes multiple misclassified inputs difficult?
Q10 Why are two similar instances given different predictions (one correct, one wrong)?

XAI Tools

LIME, SHAP, SmoothGrad, GradCAM, Guided Backpropagation, Integrated Gradients, Sensitivity Analysis


Setup

1. Install dependencies

python -m venv ~/.venvs/mea
source ~/.venvs/mea/bin/activate
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt

2. Environment variables

export HF_HOME=/path/to/hf_cache
export TRANSFORMERS_CACHE=/path/to/hf_cache
export TORCH_HOME=/path/to/torch_cache

export TINKER_API_KEY=...       # required for Qwen3.6-35B-A3B (primary model)
export GEMINI_API_KEY=...       # optional, for Gemini API baseline
export ANTHROPIC_API_KEY=...    # optional, for Claude API baseline
export OPENAI_API_KEY=...       # optional, for GPT API baseline

Datasets

Dataset Modality Model
Adult Census (adult_tabnn, adult_2layernn) tabular 2-layer NN / TabNN
Breast Cancer (cancer_tabnn, cancer_2layernn) tabular 2-layer NN / TabNN
SNLI (snli_cnn, snli_2layernn) text CNN / 2-layer NN
IMDb (imdb_cnn, imdb_2layernn) text CNN / 2-layer NN
CUB-200 (cub_resnet, cub_densenet) vision ResNet / DenseNet
STL-10 (stl10_resnet, stl10_densenet) vision ResNet / DenseNet

The benchmark is hosted on Hugging Face; download it into dataset/ so that JSONs are in dataset/{train,test}/{modality}/. Model checkpoints are under models_to_read/.


Running the pipeline

Single question

python MEA_pipeline.py \
    --dataset  dataset/test/vision/stl10_resnet_q1.json \
    --question_id 0 \
    --model_url  models_to_read/vision/stl10_resnet.pth \
    --vlm  tinker/Qwen/Qwen3.6-35B-A3B \
    --mode test \
    --output_dir outputs/

Key flags:

Flag Default Description
--vlm — VLM backend. Primary: tinker/Qwen/Qwen3.6-35B-A3B. API: gemini-2.5-pro, claude-sonnet-4-5, gpt-4o. Local: Qwen/Qwen3-VL-8B-Instruct
--mode test Dataset split (train or test)
--no-eval off Skip faithfulness evaluation
--no-improvement off Skip the improvement loop
--temperature 0.0 Sampling temperature
--tinker_checkpoint — Tinker LoRA checkpoint to evaluate (tinker/<run_id>--<step>)
--tinker_lora_rank 32 LoRA rank (must match training)
--dataset_dir / --models_dir ./dataset / ./models_to_read Root overrides for benchmark JSONs and model files

Batch runner

Shell interface:

./run_pipeline_batch.sh \
    --vlm tinker/Qwen/Qwen3.6-35B-A3B \
    --mode test \
    --datasets "stl10_resnet cub_resnet" \
    --q_types "1 2 3 4" \
    --question_ids "0-9" \
    --output_dir outputs/ \
    --parallel --max_jobs 4

Python interface:

python run_pipeline_batch.py \
    --vlm tinker/Qwen/Qwen3.6-35B-A3B \
    --mode test \
    --datasets stl10_resnet cub_resnet \
    --q_types 1 2 3 4 \
    --question_ids 0-9 \
    --output_dir outputs/ \
    --parallel --max_workers 4

Use --modality {vision,text,tabular,all} instead of --datasets to run all datasets for a modality. Use --dry_run to preview commands.

Config file:

python run_pipeline_batch.py --config batch_config_example.json

Training

MEA is trained with GRPO (Group Relative Policy Optimization) on Qwen3.6-35B-A3B via the Tinker SDK. The pipeline itself serves as the RL environment: faithfulness scores are used as rewards to guide the Proposer and Actor.

Training uses LoRA (rank 32).

python -m training.rl.train \
    --dataset_name stl10_resnet \
    --mode train \
    --q_types 1 2 3 \
    --model_name Qwen/Qwen3.6-35B-A3B \
    --output_dir checkpoints/grpo_vision \
    --num_rollouts 4 \
    --no-improvement

Faithfulness Evaluation

Faithfulness is measured by a perturb-and-measure protocol: the evaluator masks or replaces the top-ranked features identified by the explanation and checks whether the model's prediction changes accordingly. A soft faithfulness score in [0, 1] is computed per question type.

Evaluators for Q1–Q10 are in evaluation/explanation_faithfulness/.


Output structure

outputs/
  {dataset}_{qtype}_{question_id}/
    results.json            # full pipeline output (strategy, explanation, evaluation)
  training_data/            # GRPO training datapoints (passed faithfulness threshold)
    {modality}/{dataset}/{qtype}/{row_no}/training_datapoint.json
  logs/<timestamp>/         # per-job logs from batch runner

Code structure

MEA_pipeline.py             # Main pipeline orchestrator (MEAPipeline class)
MEA_agent_system.py         # Agent system (Proposer, Actor)
run_pipeline_batch.py       # Python batch runner
run_pipeline_batch.sh       # Shell batch runner
agents/                     # Agent base classes
prompts/                    # Per-question-type prompt builders (Q1–Q10)
evaluation/                 # Faithfulness evaluators (Q1–Q10)
training/rl/                # GRPO training (env, trainer, dataset, rollout)
models_to_read/             # Model checkpoints and loader scripts

Citation

@article{cheng2026mea,
  title         = {{MEA}: A Reward-Driven Multi-Agent System for Faithful Model Explanations},
  author        = {Cheng, Yuyang and Ravi, Raghav Kaushik and Sridhar, Srivarshinee and Saha, Sriparna and Ghosh, Akash and Agarwal, Chirag},
  journal       = {arXiv preprint arXiv:2610.02480},
  year          = {2026},
  eprint        = {2610.02480},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI}
}

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages