Skip to content

M2: the benchmark that decides whether this ships #3

Description

@heydryft

STOP GATE. Nothing proceeds past this until it passes.

Replay the real Arc corpus: 21,619 lines, 449 in-place retraction markers, known-dead numbers, two divergent copies of the same FACTS.md.

Baselines: full-context, and plain-file-plus-grep.

Done when Datum beats both by >=10 points, scores 100% on a stale-fact-resistance set where the correct answer requires honouring a supersession, and surfaces zero dead numbers in a default read.

Context: file+grep scores 74.0% on LOCOMO vs Mem0's 68.5%; MemoryBench found no memory system consistently beats using all task context. If this fails, stop building.

See HANDOFF.md §1, §13.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions