Search results

[BUG] Task : wikitext:103:document_level does not work

bug

Status: Open.

#1299 In huggingface/lighteval;

· debasisdwivedy opened on Jul 15, 2026

[BUG] LLM judge silently scores API error strings: litellm backend sends max_tokens as a list, provider returns 400

bug

Status: Open.

#1296 In huggingface/lighteval;

· AbhilakshSinghReen opened on Jul 14, 2026

hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset

Status: Open.

#1293 In huggingface/lighteval;

· ErenAta16 opened on Jul 11, 2026

[BUG] transformers backend overrides the model's generation_config.eos_token_id, so chat models whose turn terminator differs from tokenizer.eos never stop (Gemma pads to max_new_tokens)

Status: Open.

#1278 In huggingface/lighteval;

· waqaskhan137 opened on Jul 2, 2026

[FT] Introduce a static 'lint' command to validate custom tasks before execution

Status: Open.

#1266 In huggingface/lighteval;

· Kesav2k04 opened on Jun 18, 2026

MMLU-Pro incorrect prompt format

bug

Status: Open.

#1265 In huggingface/lighteval;

· RasmusHoier opened on Jun 17, 2026

Lighteval's LCB grader fails correct solutions that read sys.stdin.buffer

bug

Status: Open.

#1255 In huggingface/lighteval;

· RasmusHoier opened on Jun 12, 2026

Guidance on adding REFUTE benchmark task (scientific critique & calibration)

Status: Open.

#1253 In huggingface/lighteval;

· connerlambden opened on Jun 5, 2026

[FT] IRT-based scoring as alternative ranking method for fairer model comparison

Status: Open.

#1240 In huggingface/lighteval;

· testofschool opened on May 20, 2026

[BUG] [vLLM] Batch truncation bug: context length check uses first prompt instead of longest

bug

Status: Open.

#1204 In huggingface/lighteval;

· paulovsantanas opened on Apr 6, 2026

Custom task: RAIL Score responsible AI evaluation (8-dimension scoring)

Status: Open.

#1202 In huggingface/lighteval;

· SumitVermakgp opened on Apr 2, 2026

[META] Is this repo still maintained? Community PRs going unreviewed for months

Status: Open.

#1200 In huggingface/lighteval;

· kryvokhyzha opened on Mar 29, 2026