A logic program answers owns(alphabet_inc, X) only from clauses whose head says owns. Wikidata never says "owns". It records that YouTube is owned by Google and that XXVI Holdings is owner of Google, in two of the 1,790 properties it uses to link one entity to another. The semantic rewriter closes that gap without changing how the program is evaluated. When evaluation first reaches a predicate, it asks an entailment model which other predicates imply it and in which argument order, admits the implications the model trusts as ordinary rules, such as owns(X, Y) :- owned_by(Y, X), and lets the engine carry on. Each admitted rule keeps the model's opinion of it, so every answer can also say how probable it is: the chance that every rule of one of its derivations is right.
The rewriter runs on any logic engine that implements a small protocol. Two ship with it: a pure-Python engine with no dependencies, and PyMeTTa (MeTTa on SWI-Prolog). With tabling, every query terminates and gets exactly the answers its admitted rules imply; DESIGN.md states the conditions and the protocols, and RESULTS.md the experiment behind the defaults.
Each question below is a MeTTa file written in its own words, with its own rules. Beside each, examples/wikidata holds a knowledge base of Wikidata's statements about the question's subject, in Wikidata's 1,790 properties that link two entities, each glossed with its English label. None of the questions names one of those properties.
; Who descends from Queen Victoria? Wikidata has no descendant property, and none of its 1,790 entity-valued
; properties is called "parent of"; the question uses its own words and its own rules.
(gloss parent_of "{x} is a parent of {y}")
(gloss descendant_of "{x} is a descendant of {y}")
(= (descendant_of $X $Y) (parent_of $Y $X))
(= (descendant_of $X $Z) (let True (parent_of $Y $X) (descendant_of $Y $Z)))
(query (descendant_of $X queen_victoria))
; What did Leonardo da Vinci create? Wikidata files a work's maker under many properties, recorded on the work.
(gloss created "{x} created {y}")
(query (created leonardo_da_vinci $W))
; What does Alphabet own, directly or through what it owns?
(gloss owns "{x} owns {y}")
(= (owns $X $Z) (let True (owns $X $Y) (owns $Y $Z)))
(query (owns alphabet_inc $C))--trace prints each step on standard error as it happens. Here PyMeTTa evaluates whether Alphabet owns YouTube, the cross-encoder nli-deberta-large narrows the candidates, and Jev, TypeSafe's judge model, decides which of them mean ownership:
$ semrewrite ask examples/wikidata/alphabet.metta 'owns(alphabet_inc, youtube_q866)' --judge jev --threshold 0.65 --strategy rewrite-depth-tabled --depth 1 --retriever judge --k 50 --engine pymetta --work 50000000 --seconds inf --trace --explain
pymetta starts on owns(alphabet_inc, youtube_q866), over 18,841 facts and 1 rule, with no rewrites yet
evaluation reached owns, which no fact states, and asks for the rewrites concluding it
the nli-deberta-large retriever ranked the 1,790 predicates of the same arity as premises of "A owns B" and kept the 50 it ranks highest
jev was asked 101 questions, whether each premise sentence entails "A owns B", and admits a score of 0.650 or more:
admit 0.990 "B is owned by A" owned_by_p127, ranked 2
0.010 "A is owned by B" owned_by_p127, ranked 2
admit 0.990 "A is owner of B" owner_of_p1830, ranked 3
0.010 "B is owner of A" owner_of_p1830, ranked 3
admit 0.860 "B is afterward owned by A" afterward_owned_by_p11812, ranked 11
0.010 "A is afterward owned by B" afterward_owned_by_p11812, ranked 11
95 more rejected, the highest at 0.620: "B is beforehand owned by A" (beforehand_owned_by_p11811)
admitted as rules:
owns(X0, X1) ⇐ owned_by_p127(X1, X0)
owns(X0, X1) ⇐ owner_of_p1830(X0, X1)
owns(X0, X1) ⇐ afterward_owned_by_p11812(X1, X0)
pymetta finished: ok, 1 answer, 2,293,164 work
the memo answered 5,699 more asks for owns, without asking the judge again
the claim check reads statements about applications' individuals with nli-deberta-large: jev has no exception trust
owns(alphabet_inc, youtube_q866)
owns(alphabet_inc, youtube_q866) [by owns(X, Z) :- owns(X, Y), owns(Y, Z)]
owns(alphabet_inc, google_q95) [by owns(X, Z) :- owns(X, Y), owns(Y, Z)]
owns(alphabet_inc, xxvi_holdings) [by owns(X0, X1) ⇐ owned_by_p127(X1, X0), score 0.990]
owned_by_p127(xxvi_holdings, alphabet_inc) [fact]
owns(xxvi_holdings, google_q95) [by owns(X0, X1) ⇐ owner_of_p1830(X0, X1), score 0.990]
owner_of_p1830(xxvi_holdings, google_q95) [fact]
owns(google_q95, youtube_q866) [by owns(X0, X1) ⇐ owned_by_p127(X1, X0), score 0.990]
owned_by_p127(youtube_q866, google_q95) [fact]
rewrites admitted (3):
owns(X0, X1) ⇐ afterward_owned_by_p11812(X1, X0)
owns(X0, X1) ⇐ owned_by_p127(X1, X0)
owns(X0, X1) ⇐ owner_of_p1830(X0, X1)
statements about applications' individuals read by nli-deberta-large: jev has no exception trust
ok: 1 answer, 2,293,164 work, 101 judge questions, 4 pairs reading 3 claims in 89.04 s, 0 rewrites withdrawn, 1 application read, 0 defeated, 0 work on exceptions, 191.95 s
No statement says owns, so the engine called back for rewrites in the middle of evaluating. The cross-encoder read all 1,790 properties as premises of "A owns B", in both argument orders, and Jev judged its 50 best, each both ways: "B is owned by A" entails "A owns B" at 0.990, and "A is owned by B" at 0.010. Three of Jev's 101 answers reached the threshold, and the engine carried on with those three as rules. The question's own transitive rule chained them: Alphabet owns XXVI Holdings, which owns Google, which owns YouTube. Wikidata states the first and last of those links on what is owned, as owned by, and the middle one on the owner, as owner of. As the rule recursed, evaluation asked for owns 5,699 more times, and the memo answered each, so Jev was asked once. The claim check then read four sentence pairs for the three rewrites and one application of them, and withdrew and defeated nothing. Jev has no trust fitted for reading an application, so nli-deberta-large read that one.
Wikidata files the properties that hold each relation in its own subproperty hierarchy, so its own answer to each question follows from the same statements (questions.py). Each run allows one rewrite on any path, and both judges see the same 50 candidates; cosine-bge is an embedding-similarity judge:
| question | Wikidata's answer | read literally | judged by cosine-bge | judged by jev |
|---|---|---|---|---|
victoria: descendant_of(X, queen_victoria) |
1,164 | 0 | 1,165: 1,164 in Wikidata's answer, 1 not | 1,164, all in Wikidata's answer |
leonardo: created(leonardo_da_vinci, W) |
854 | 0 | 0 | 851: 849 in Wikidata's answer, 2 not |
alphabet: owns(alphabet_inc, C) |
3,225 | 0 | none, the work budget stopped it | 2,849, all in Wikidata's answer |
Read literally, no question finds anything. Jev returned exactly Victoria's 1,164 descendants. Its two answers outside Wikidata's for Leonardo are works his own entry lists as notable. The 376 holdings it missed for Alphabet hang from companies that only three properties connect to the rest: copyright holder, which Jev scored 0.22, and parent and child organization, which the cross-encoder ranked 68th and 555th, so Jev never saw them. The embedding judge can't tell which argument is which. On Alphabet's question it admitted owned by in both directions, which the transitive rule turns into every company owning every other, and the work budget stopped it. examples/README.md has every rewrite each run admitted and what each missed.
$ semrewrite ask examples/wikidata/victoria.metta 'descendant_of(frederik_x_of_denmark, queen_victoria)' --judge jev --threshold 0.65 --strategy rewrite-depth-tabled --depth 1 --retriever judge --k 50 --engine pymetta --work 50000000 --seconds inf --explain
descendant_of(frederik_x_of_denmark, queen_victoria)
descendant_of(frederik_x_of_denmark, queen_victoria) [by descendant_of(X, Z) :- parent_of(Y, X), descendant_of(Y, Z)]
parent_of(margrethe_ii_of_denmark, frederik_x_of_denmark) [by parent_of(X0, X1) ⇐ child_p40(X0, X1), score 0.970]
child_p40(margrethe_ii_of_denmark, frederik_x_of_denmark) [fact]
descendant_of(margrethe_ii_of_denmark, queen_victoria) [by descendant_of(X, Z) :- parent_of(Y, X), descendant_of(Y, Z)]
parent_of(ingrid_of_sweden, margrethe_ii_of_denmark) [by parent_of(X0, X1) ⇐ child_p40(X0, X1), score 0.970]
child_p40(ingrid_of_sweden, margrethe_ii_of_denmark) [fact]
descendant_of(ingrid_of_sweden, queen_victoria) [by descendant_of(X, Z) :- parent_of(Y, X), descendant_of(Y, Z)]
parent_of(princess_margaret_of_connaught, ingrid_of_sweden) [by parent_of(X0, X1) ⇐ child_p40(X0, X1), score 0.970]
child_p40(princess_margaret_of_connaught, ingrid_of_sweden) [fact]
descendant_of(princess_margaret_of_connaught, queen_victoria) [by descendant_of(X, Z) :- parent_of(Y, X), descendant_of(Y, Z)]
parent_of(prince_arthur_duke_of_connaught_and_strathearn, princess_margaret_of_connaught) [by parent_of(X0, X1) ⇐ child_p40(X0, X1), score 0.970]
child_p40(prince_arthur_duke_of_connaught_and_strathearn, princess_margaret_of_connaught) [fact]
descendant_of(prince_arthur_duke_of_connaught_and_strathearn, queen_victoria) [by descendant_of(X, Y) :- parent_of(Y, X)]
parent_of(queen_victoria, prince_arthur_duke_of_connaught_and_strathearn) [by parent_of(X0, X1) ⇐ child_p40(X0, X1), score 0.970]
child_p40(queen_victoria, prince_arthur_duke_of_connaught_and_strathearn) [fact]
rewrites admitted (18):
descendant_of(X0, X1) ⇐ derived_from_base_unit_p12571(X0, X1)
descendant_of(X0, X1) ⇐ filial_church_p12476(X1, X0)
descendant_of(X0, X1) ⇐ gens_p5025(X0, X1)
descendant_of(X0, X1) ⇐ has_spin_off_p2512(X1, X0)
descendant_of(X0, X1) ⇐ is_proceedings_from_p4745(X0, X1)
descendant_of(X0, X1) ⇐ parent_club_p831(X0, X1)
descendant_of(X0, X1) ⇐ parent_peak_p3137(X0, X1)
parent_of(X0, X1) ⇐ child_p40(X0, X1)
parent_of(X0, X1) ⇐ father_p22(X1, X0)
parent_of(X0, X1) ⇐ mother_house_p612(X1, X0)
parent_of(X0, X1) ⇐ mother_p25(X1, X0)
parent_of(X0, X1) ⇐ parent_astronomical_body_p397(X1, X0)
parent_of(X0, X1) ⇐ parent_cell_line_p3432(X1, X0)
parent_of(X0, X1) ⇐ parent_club_p831(X1, X0)
parent_of(X0, X1) ⇐ parent_organization_or_unit_p749(X1, X0)
parent_of(X0, X1) ⇐ parent_p8810(X1, X0)
parent_of(X0, X1) ⇐ parent_peak_p3137(X1, X0)
parent_of(X0, X1) ⇐ subdivision_of_this_unit_p9059(X0, X1)
ok: 1 answer, 34,386 work, 202 judge questions, 343.95 s
King Frederik X of Denmark descends from Victoria through his mother Margrethe II, her mother Ingrid of Sweden, Ingrid's mother Princess Margaret of Connaught, and Margaret's father Prince Arthur, Victoria's son. Each step is a Wikidata statement read through one of Jev's rewrites, shown with its score. Jev also admitted premises such as parent peak and parent club, which add nothing here because the knowledge base holds no statement in them.
$ semrewrite ask examples/wikidata/victoria.metta 'descendant_of(queen_victoria, queen_victoria)' --judge cosine-bge --strategy rewrite-depth-tabled --depth 1 --retriever judge --k 50 --engine pymetta --work 50000000 --seconds inf --explain
descendant_of(queen_victoria, queen_victoria)
descendant_of(queen_victoria, queen_victoria) [by descendant_of(X, Z) :- parent_of(Y, X), descendant_of(Y, Z)]
parent_of(alfred_duke_of_saxe_coburg_and_gotha, queen_victoria) [by parent_of(X0, X1) ⇐ child_p40(X1, X0), score 0.968]
child_p40(queen_victoria, alfred_duke_of_saxe_coburg_and_gotha) [fact]
descendant_of(alfred_duke_of_saxe_coburg_and_gotha, queen_victoria) [by descendant_of(X, Y) :- parent_of(Y, X)]
parent_of(queen_victoria, alfred_duke_of_saxe_coburg_and_gotha) [by parent_of(X0, X1) ⇐ child_p40(X0, X1), score 0.959]
child_p40(queen_victoria, alfred_duke_of_saxe_coburg_and_gotha) [fact]
rewrites admitted (8):
descendant_of(X0, X1) ⇐ descendant_of(X1, X0)
parent_of(X0, X1) ⇐ child_p40(X0, X1)
parent_of(X0, X1) ⇐ child_p40(X1, X0)
parent_of(X0, X1) ⇐ father_p22(X1, X0)
parent_of(X0, X1) ⇐ mother_p25(X1, X0)
parent_of(X0, X1) ⇐ parent_of(X1, X0)
parent_of(X0, X1) ⇐ parent_p8810(X0, X1)
parent_of(X0, X1) ⇐ parent_p8810(X1, X0)
ok: 1 answer, 2,584,921 work, 202 judge questions, 223.69 s
Embedding similarity scores how alike two sentences are, whichever way they run. As premises of "A is a parent of B", the embedding judge scored "B is the child of A" at 0.96 and "A is the child of B" at 0.97, and admitted both. So "Prince Alfred is the child of Queen Victoria" also made Alfred a parent of Victoria, and Victoria a descendant of herself.
--beliefs gives every answer an opinion that it holds, in subjective logic's terms: a probability P, and the belief b, disbelief d and uncertainty u it comes from. A rewrite is one claim, right or wrong for every use of it, so an answer is as probable as the chance that every rewrite of one of its derivations is right. Each rewrite is right as Jev's opinion of it says, drawn toward the rate at which labelled pairs hold by how far Jev proved right on them; DESIGN.md says how the probabilities combine.
That makes a low rewrite threshold cheap. At 0.1 instead of 0.65, Jev admits 110 rewrites into Victoria's descendants instead of 18, and 21 of the answers they add are not in Wikidata's record. Each of those 21 ranks below each of Wikidata's 1,164, so an answer threshold anywhere between them keeps exactly Wikidata's answer:
| answer | rank | in Wikidata's answer | P | b | d | u |
|---|---|---|---|---|---|---|
sophia_hermann |
1,160 of 1,185 | yes | 0.938 | 0.929 | 0.029 | 0.042 |
sophia_wise_glossinger |
1,161 of 1,185 | yes | 0.938 | 0.929 | 0.029 | 0.042 |
stephan_albert_bernadotte_count_bernadotte |
1,162 of 1,185 | yes | 0.938 | 0.929 | 0.029 | 0.042 |
stephan_straehl |
1,163 of 1,185 | yes | 0.938 | 0.929 | 0.029 | 0.042 |
victoria_bayard_romanov_ilyinsky |
1,164 of 1,185 | yes | 0.938 | 0.929 | 0.029 | 0.042 |
british_army |
1,165 of 1,185 | no | 0.324 | 0.321 | 0.637 | 0.042 |
hellenic_armed_forces |
1,166 of 1,185 | no | 0.324 | 0.321 | 0.637 | 0.042 |
imperial_german_army |
1,167 of 1,185 | no | 0.324 | 0.321 | 0.637 | 0.042 |
luftwaffe |
1,168 of 1,185 | no | 0.324 | 0.321 | 0.637 | 0.042 |
royal_air_force |
1,169 of 1,185 | no | 0.324 | 0.321 | 0.637 | 0.042 |
Asked about one of them, the rewriter shows where its doubt comes from:
$ semrewrite ask examples/wikidata/victoria.metta 'descendant_of(british_army, queen_victoria)' --judge jev --threshold 0.1 --strategy rewrite-depth-tabled --depth 1 --retriever judge --k 50 --engine pymetta --work 50000000 --seconds inf --beliefs --explain 2>/dev/null
descendant_of(british_army, queen_victoria) P 0.324 b 0.321 d 0.637 u 0.042
descendant_of(british_army, queen_victoria) [by descendant_of(X, Z) :- parent_of(Y, X), descendant_of(Y, Z)]
parent_of(edward_vii, british_army) [by parent_of(X0, X1) ⇐ military_branch_p241(X0, X1), score 0.330]
military_branch_p241(edward_vii, british_army) [fact]
descendant_of(edward_vii, queen_victoria) [by descendant_of(X, Y) :- parent_of(Y, X)]
parent_of(queen_victoria, edward_vii) [by parent_of(X0, X1) ⇐ child_p40(X0, X1), score 0.970]
child_p40(queen_victoria, edward_vii) [fact]
The British Army descends from Victoria because one of her sons served in it, read through a rewrite that takes a person's military branch for their child. Jev admitted that rewrite at 0.330, so the answer is more probably wrong than right. RESULTS.md has the same measurement on all three questions.
The rewriter reads each rewrite the judge admits against what the knowledge base states, so a belief you state can withdraw a match it contradicts. examples/actions_first_person.kb states eight things I do, such as "I watch TV" and "I play video games", and asks whether I waste time. nli-deberta-large matches two of them:
semrewrite rewrites examples/actions_first_person.kb wastes_time --retriever allexamples/actions_first_person_believed.kb is the same file with the belief added, as a gloss and a fact with no arguments:
gloss gaming_productive "I consider gaming to be productive".
gaming_productive.
Asked again, the rewriter withdraws the gaming match:
$ semrewrite rewrites examples/actions_first_person_believed.kb wastes_time --retriever all
admit 0.968 wastes_time() ⇐ watch_tv()
defeat 0.887 wastes_time() ⇐ play_games()
contradicts d 1.000 >= 0.659 "I consider gaming to be productive." similarity 0.764
9 judge questions at threshold 0.659
$ semrewrite ask examples/actions_first_person_believed.kb wastes_time --retriever all --beliefs --explain
wastes_time() P 0.478 b 0.337 d 0.000 u 0.663
wastes_time() [by wastes_time() ⇐ watch_tv(), score 0.968]
watch_tv() [fact]
rewrites admitted (1):
wastes_time() ⇐ watch_tv() P 0.478 b 0.337 d 0.000 u 0.663
rewrites withdrawn by what the knowledge base states (1):
wastes_time() ⇐ play_games()
contradicts d 1.000 >= 0.659 "I consider gaming to be productive." similarity 0.764
ok: 1 answer, 4 work, 18 judge questions, 1 pair reading 2 claims in 1.04 s, 1 rewrite withdrawn, 3.04 s
The judge reads the belief as contradicting the gaming match's claim, "If I play video games, I waste time.", more surely than it was calibrated to read a match to admit it, so the match is withdrawn: ask no longer answers through it, and "I waste time" stands on watching TV alone. A statement the judge reads as contradicting a claim less surely than that only lowers the match's P, discounted by the trust the judge earned reading delta-ATOMIC's updates against the inferences they weaken or strengthen. ask --explain names the statements that withdrew or weighed on each match. The check reads rewrites, not answers: the same model reads the belief as contradicting "sam wastes time." as surely as "kim wastes time.", though only kim plays games. And it reads only the statements that stand out in similarity to a rewrite, because the judge reads most unrelated statements as contradictions too; DESIGN.md says how, and RESULTS.md gives the measurements.
A belief about one person changes that person's answers alone. examples/actions_people.kb is examples/actions.kb with amy also playing games and kim also watching TV, and examples/actions_people_believed.kb states a belief about kim, as a gloss and a fact naming kim:
gloss gaming_productive "{x} considers gaming to be productive".
gaming_productive(kim).
nli-deberta-large admits the gaming match only below its calibrated threshold, so this asks at 0.03:
$ semrewrite ask examples/actions_people_believed.kb 'wastes_time(X)' --retriever all --threshold 0.03 --beliefs --explain
wastes_time(kim) P 0.420 b 0.263 d 0.000 u 0.737
wastes_time(kim) [by wastes_time(X0) ⇐ watch_tv(X0), score 0.756]
watch_tv(kim) [fact]
wastes_time(sam) P 0.420 b 0.263 d 0.000 u 0.737
wastes_time(sam) [by wastes_time(X0) ⇐ watch_tv(X0), score 0.756]
watch_tv(sam) [fact]
wastes_time(amy) P 0.224 b 0.014 d 0.001 u 0.985
wastes_time(amy) [by wastes_time(X0) ⇐ play_games(X0), score 0.039]
play_games(amy) [fact]
rewrites admitted (3):
play_games(X0) ⇐ gaming_productive(X0) P 0.528 b 0.400 d 0.000 u 0.600
wastes_time(X0) ⇐ play_games(X0) P 0.224 b 0.014 d 0.001 u 0.985
wastes_time(X0) ⇐ watch_tv(X0) P 0.420 b 0.263 d 0.000 u 0.737
applications defeated by what the knowledge base states about their individuals (1):
wastes_time(kim) ⇐ play_games(kim), given "kim plays games."
contradicts d 0.985 >= 0.659 "kim considers gaming to be productive." similarity 0.676
ok: 3 answers, 20 work, 36 judge questions, 1 pair reading 3 claims in 1.10 s, 0 rewrites withdrawn, 1 application read, 1 defeated, 12 work on exceptions, 4.90 s
The statement names kim, so it bears on each application of a match to kim, not on the match. Each application reads the statements that name its individuals and stand out against its match's claim, as delta-NLI reads an update: the application's premise and the statement together, "kim plays games. kim considers gaming to be productive.", against its conclusion, "kim wastes time.". The judge reads that as a clear contradiction, so kim's application of the gaming match derives nothing, as an abnormality atom ab(play_games, kim) blocks one instance of a default in answer-set programming. kim still wastes time through watching TV, at that match's P, and the derivation shown is that one; amy still wastes time through the gaming match, which stays admitted, at the P it had with nothing stated about kim. A reading short of a clear contradiction only lowers the answers derived through that application, at a trust fitted on delta-ATOMIC's updates read the same way with names filled in.
examples/parent_of_tom.kb states Tom's family in words that never say "parent", in the rewriter's plain-text format. Each predicate has a gloss, a sentence with a placeholder per argument, which is what the entailment model reads, and the gold lines say which implications a careful reader accepts:
% Tom's family, stated in words that never say "parent".
gloss mum "{x} is the mum of {y}"
gloss mother "{x} is the mother of {y}"
gloss dad "{x} is the dad of {y}"
gloss parent "{x} is a parent of {y}"
gloss child_of "{x} is a child of {y}"
gloss guardian "{x} is the legal guardian of {y}"
gloss grandparent "{x} is a grandparent of {y}"
mum(alice, tom).
dad(bob, tom).
mother(carol, alice).
child_of(bob, dave).
guardian(erin, tom).
grandparent(X, Z) :- parent(X, Y), parent(Y, Z).
query parent(X, tom).
query grandparent(X, tom).
% The implications a careful reader accepts; erin, a legal guardian, is not a parent.
gold mum => mother.
gold mother => mum.
gold mother => parent.
gold dad => parent.
gold child_of => parent swap.
Asking for Tom's parents with the NLI cross-encoder nli-deberta-large as the judge, at the threshold it was calibrated to on labelled predicate pairs:
$ semrewrite ask examples/parent_of_tom.kb 'parent(X, tom)' --judge nli-deberta-large --retriever all --explain
parent(alice, tom)
parent(alice, tom) [by parent(X0, X1) ⇐ mum(X0, X1), score 0.998]
mum(alice, tom) [fact]
parent(bob, tom)
parent(bob, tom) [by parent(X0, X1) ⇐ dad(X0, X1), score 0.997]
dad(bob, tom) [fact]
parent(erin, tom)
parent(erin, tom) [by parent(X0, X1) ⇐ guardian(X0, X1), score 0.823]
guardian(erin, tom) [fact]
rewrites admitted (14):
child_of(X0, X1) ⇐ dad(X1, X0)
child_of(X0, X1) ⇐ mother(X1, X0)
child_of(X0, X1) ⇐ mum(X1, X0)
child_of(X0, X1) ⇐ parent(X1, X0)
mother(X0, X1) ⇐ child_of(X1, X0)
mother(X0, X1) ⇐ mum(X0, X1)
mum(X0, X1) ⇐ child_of(X1, X0)
mum(X0, X1) ⇐ mother(X0, X1)
mum(X0, X1) ⇐ parent(X0, X1)
parent(X0, X1) ⇐ child_of(X1, X0)
parent(X0, X1) ⇐ dad(X0, X1)
parent(X0, X1) ⇐ guardian(X0, X1)
parent(X0, X1) ⇐ mother(X0, X1)
parent(X0, X1) ⇐ mum(X0, X1)
ok: 3 answers, 63 work, 78 judge questions, 4.58 s
Every answer comes with its derivation, and each step names the rule or the admitted rewrite it used, with the judge's score for a rewrite. The third answer is wrong: a legal guardian need not be a parent, and at this threshold the judge admits guardian ⇒ parent along with other false rewrites such as child_of(y, x) ⇒ mother(x, y). Raising the threshold to 0.995, which calibration for answer F1 chose for lazy-tabled evaluation on three of the experiment's four held-out benchmarks, leaves the answers alice and bob:
$ semrewrite ask examples/parent_of_tom.kb 'parent(X, tom)' --judge nli-deberta-large --retriever all --threshold 0.995
parent(alice, tom)
parent(bob, tom)
ok: 2 answers, 14 work, 65 judge questions, 5.54 s
At that threshold the judge admits six rewrites, all true, and the grandparent query composes them with the knowledge base's own rule:
$ semrewrite ask examples/parent_of_tom.kb 'grandparent(X, tom)' --judge nli-deberta-large --retriever all --threshold 0.995 --explain
grandparent(carol, tom)
grandparent(carol, tom) [by grandparent(X, Z) :- parent(X, Y), parent(Y, Z)]
parent(carol, alice) [by parent(X0, X1) ⇐ mother(X0, X1), score 0.996]
mother(carol, alice) [fact]
parent(alice, tom) [by parent(X0, X1) ⇐ mum(X0, X1), score 0.998]
mum(alice, tom) [fact]
grandparent(dave, tom)
grandparent(dave, tom) [by grandparent(X, Z) :- parent(X, Y), parent(Y, Z)]
parent(dave, bob) [by parent(X0, X1) ⇐ child_of(X1, X0), score 0.998]
child_of(bob, dave) [fact]
parent(bob, tom) [by parent(X0, X1) ⇐ dad(X0, X1), score 0.997]
dad(bob, tom) [fact]
rewrites admitted (6):
mother(X0, X1) ⇐ mum(X0, X1)
mum(X0, X1) ⇐ mother(X0, X1)
parent(X0, X1) ⇐ child_of(X1, X0)
parent(X0, X1) ⇐ dad(X0, X1)
parent(X0, X1) ⇐ mother(X0, X1)
parent(X0, X1) ⇐ mum(X0, X1)
ok: 2 answers, 24 work, 78 judge questions, 7.93 s
That trade-off between admitting enough and admitting only what is true, and how to calibrate a judge for it, is most of what RESULTS.md measures.
Clone the repository and install the parts you need:
git clone https://github.com/MesTTo/semantic-rewriter
cd semantic-rewriter
pip install . # the core: pure-Python engine, no dependencies
pip install ".[models]" # NLI, reranker and embedding judges, embedding retrieval (torch)
pip install ".[pymetta]" # the PyMeTTa engine
pip install ".[all]" # everything, plus the conformance kit and the Jev judgeThe core runs on Python 3.11 or later; the PyMeTTa engine needs 3.12 or later. uv sync --all-extras builds the environment the experiment used, with CUDA 12.8 torch on Linux and Windows. A command that needs a library you have not installed names the extra that installs it. The claim check needs the models extra once a knowledge base states something it reads: a fact with no arguments, or a fact naming an individual a rewrite is applied to, as nearly every knowledge base's facts do. It chooses what to read with an embedding model, and where the run's judge has no calibrated trust for a reading, as Jev has none, it reads with nli-deberta-large; with Jev's extra alone, a run with something to read stops with an error naming the extra.
semrewrite ask KB QUERY [--judge NAME] [--reader NAME] [--engine python|pymetta] [--strategy NAME] [--depth N] [--explain] [--trace] [--json]
semrewrite ask KB QUERY --beliefs [--answer-threshold P] # each answer with its opinion, ranked by P
semrewrite rewrites KB PREDICATE [--judge NAME] [--all] # what discovery admits, what KB's statements withdraw, and with --all what it rejects
semrewrite rewrites KB PREDICATE --snars # the admitted rewrites as SNARS beliefs and rules for GoalChainer
semrewrite judge [--judge NAME] [--trust T] [--base-rate A] < PAIRS # each sentence pair's opinion, as JSON lines
semrewrite engines # installed engines and their capabilities
semrewrite strategies # the strategies and their knobs
semrewrite conformance --engine NAME # check an engine against the protocol
semrewrite convert family.metta family.kb # between the text, JSON and MeTTa formats--judge oracle uses the gold lines of the knowledge base instead of a model, which is how tests and the experiment separate engine behaviour from judge error. --retriever all judges every predicate of the right arity instead of the 8 nearest by gloss embedding, which suits small catalogues and loads no embedding model. --retriever judge ranks every predicate with the --prefilter judge, the cross-encoder nli-deberta-large by default, and the admitting judge sees only the --k it ranks highest, which suits a catalogue too large to judge whole, such as Wikidata's 1,790 properties above. --depth N replaces the strategy's depth bound, counted the way the strategy counts it, and --depth 0 removes it. --explain prints each answer's derivation and the rewrites admitted, and --trace prints each step on standard error as it happens, as in the Alphabet example above. --beliefs prints each answer with its opinion, highest P first, as in the Victoria example above, and --answer-threshold P drops the answers below P. Every run checks each rewrite the judge admits against the knowledge base's statements read by meaning: a statement the judge reads as contradicting the rewrite's claim at its calibrated threshold, which --threshold does not move, withdraws the rewrite, and vaguer readings weigh on its opinion. Each application of a rewrite to the row it concludes is checked too, against the statements naming the row's individuals: one the judge reads as contradicting the application's conclusion, given its premise, at that threshold derives nothing, and an answer every derivation of which applies a defeated application is no answer; vaguer readings weigh on the answers derived through the application alone. --explain lists under each admitted rewrite the statements read against it, then the rewrites withdrawn, each with the statements that withdrew it, then the applications defeated and weighed, each with its statements, and the answers a defeat left no derivation for; --json writes them with the check's cost; rewrites prints each withdrawn rewrite as a defeat line and each admitted one's opinion after the check where something was read against it; and rewrites --snars exports that opinion. The check reads with --reader where given, else with the run's judge wherever it carries the calibrated trust a reading weighs at, else with nli-deberta-large, and --trace, --explain and --json name a reader the run's judge handed a reading to, with why; a --reader without that trust is refused. --judge oracle, which reads no sentences, has no check unless a --reader is given: without one it refuses a knowledge base that states something without arguments and leaves facts naming individuals to the rows. The check uses the run's embedding model, loading the default one, only once there is something to read, when --retriever is all or judge. rewrites --snars prints each admitted rewrite as two lines of the live world of mettabase's SNARS, the subjective-logic reasoner GoalChainer uses: its validity, a statement (valid rw-owns-owned_by_p127-10) believed with the rewrite's opinion as evidence, and a row of SNARS's rule kit deriving owns(X0, X1) from owned_by_p127(X1, X0) and that statement. The row is of SNARS's certain-rule class, a rule that holds whenever its premises do, so the rewrite's doubt lives in its validity alone: SNARS's sn prove gives the conclusion the product of the premise's and the validity's probabilities, every application of the rewrite shares the one validity statement, and a proof that uses it twice counts it once, as every answer through it does here. A knowledge base seeds each of a question's N candidate answers at 1/N (sn marginal!), the probability SNARS leaves an answer no derivation reaches. judge reads sentence pairs from standard input, one JSON object a line with a premise and a conclusion, and writes each pair's opinion as a JSON line, calibrated at the judge's trust and base rate unless --trust and --base-rate give others, for pairs drawn from another population; GoalChainer uses it to ask whether a request states an element of a problem. The exit status is 0 when the query finished, 2 when a budget stopped it and 1 on an error.
from semrewrite import Rewriter, formats
from semrewrite.engines import make as engine
from semrewrite.judges import make as judge
kb = formats.load("examples/parent_of_tom.kb")
rewriter = Rewriter(kb, judge("nli-deberta-large"), engine("python"))
outcome = rewriter.ask("grandparent(X, tom)", "lazy-tabled")
print(outcome.answers) # {('carol', 'tom'), ('dave', 'tom')}
print(outcome.admitted) # the rewrites admitted, as (premise, conclusion, argument mapping)Rewriter checks every rewrite it admits, and every application of one, with its judge, and outcome.defeated holds the rewrites the knowledge base's statements withdrew, each with the readings that withdrew it; outcome.applications holds each application read, by (rewrite key, row), with its readings and its leaf's opinion that the rewrite applies normally there, outcome.fallen the answers a defeated application left no derivation for, and outcome.exception_work the work those exceptions added, apart from outcome.work; with beliefs=16, outcome.readings holds the statements read against each admitted rewrite. reader= gives the check its reader, and claims=ClaimCheck(kb, judge, embedder, reader=...), ClaimCheck from semrewrite, its embedding model or other settings.
A knowledge base can also be built in code with KB, atom and Rule, and predicates may have any arity: gave(x, y, z) with the gloss {x} gave {y} to {z} implies received(z, y, x), and discovery proposes every argument mapping.
| engine | needs | tables | depth bound | bound on rewrites | lazy discovery | untabled | proofs |
|---|---|---|---|---|---|---|---|
python |
nothing | yes | yes | yes | yes | no | yes |
pymetta |
pymetta |
yes | yes | yes | yes | yes | yes |
An engine is anything with a solve(program, query, budget, expand) method and a capabilities record; DESIGN.md gives the six rules it must keep. semrewrite conformance --engine NAME checks them with property tests against a reference evaluator: exact answers with tables, one meaning of depth, lazy expansion equal to eager, hard budgets, engine keywords accepted as ordinary names, and the same solution, proofs included, whatever ran before in the process and in processes that hash strings differently. A Prolog through janus, a Datalog engine or SQL with recursive queries can implement the same protocol; add it to semrewrite/engines/__init__.py and run the kit.
| judge | what it is | extra |
|---|---|---|
nli-deberta-base, nli-deberta-large, nli-deberta-large-wanli |
NLI cross-encoders reading premise and conclusion together | models |
reranker-qwen3 |
Qwen3-Reranker asked yes or no | models |
typical-qwen3-4b |
Qwen3-4B-Instruct asked whether the conclusion typically follows, answering yes, no or unsure | models |
jev |
TypeSafe's hosted System One model, with answers cached on disk; reads a TypeSafe API key from TYPESAFE_API_KEY, or from the file JEV_KEY_FILE names |
jev |
cosine-bge, cosine-qwen3 |
embedding similarity, the baseline that cannot tell direction | models |
oracle |
admits exactly a given set of implications | nothing |
Each judge's default threshold, trust and base rate in semrewrite/judges/calibration.json were calibrated on the labelled pairs of the kind of question it is asked: Wikidata's property pairs for the judges of what must follow, δ-ATOMIC's for typical-qwen3-4b. For the rewriter, calibrate it on your own knowledge bases instead (see RESULTS.md): the pair-set thresholds favour recall, and the rewriter needs precision because it applies every admitted rule everywhere. A judge is anything with name, threshold, trust, base_rate, usage, opinions(proposals) and close(), which DESIGN.md describes.
The pair sets the judges were measured on live under data/pairs: a hand-labelled hard set, and pairs from Wikidata's property hierarchy (CC0), whose query results are under data/raw/wikidata. Two more sets hold implications that are true typically rather than by meaning. δ-ATOMIC's defeasible inferences (Rudinger et al. 2020, MIT, under data/raw/delta-atomic) pair an everyday event about PersonX and a human-written update with an inference the update strengthens or defeats. Balanced COPA's causes and effects (Kavumba et al. 2019, CC BY 4.0 on its author's card) give each question's two alternatives, the more plausible one holding. The Levy/Holt directional pairs come from entgraph_eval, which states no license, and COPA's own questions state none either, so this repository includes neither. These commands download them at the revisions the experiment used, check each file's SHA-256, and build the pair files:
python -m semrewrite.pairs fetch levyholt
python -m semrewrite.pairs fetch copa
python -m semrewrite.pairs build levyholt copaThe Jev judge's cached answers are not included either, so rerunning it needs a TypeSafe API key.
The experiment ran twelve strategies with a perfect judge, and six of them with each of seven model judges, over four generated knowledge bases (75 predicates, 58 queries) on PyMeTTa:
- With tables and a per-predicate memo, every query terminated, on the benchmarks and on 400 generated catalogues per property; without tables, rewriting looped exactly when the query reached a cycle of rewrites.
- Jev and the NLI cross-encoders tell which way an implication runs (direction accuracy 0.77 to 0.89 on two external pair sets); embedding similarity sits at chance.
- Admission precision decides answer quality, so the judge's threshold has to be calibrated for answer F1, not for pair accuracy.
- With an imperfect judge, a depth bound of 3 limits how far a false rule spreads. At calibrated thresholds, tabled evaluation bounded at three steps (
depth-tabled) raised answer F1 for six of the seven judges, nli-deberta-large's from 0.68 to 0.91, and cost the perfect judge three answers that need more than three steps. Bounding three rewrites alone (rewrite-depth-tabled) keeps those answers and gave Jev its best F1, 0.92, but scored below the step bound with every other judge. - Asked what typically follows rather than what is entailed, Qwen3-4B-Instruct-2507 (
typical-qwen3-4b) scores held-out log loss 0.588 on δ-ATOMIC's defeasible inferences and 0.485 on Balanced COPA, against nli-deberta-large's 0.661 and 0.637, and on a knowledge base of actions ranks the three that waste time first, where the entailment judge finds one.
RESULTS.md has every number, the method and the sixteen PyMeTTa defects found on the way, fourteen of them with a script under repros/ that reproduces them.
Every query terminates when the catalogue of predicates is finite, rewrites only rename and permute arguments, user rules have no function symbols, discovery is memoised, and evaluation is tabled or depth-bounded. Answers are exact with respect to the admitted rules; whether they are true depends on the judge. Function symbols, negation and discovery of facts from text are outside the design. DESIGN.md explains each condition and what breaks without it.
uv sync --all-extras
uv run pytest # unit, property and conformance tests, 400 cases per property
uv run pytest --hypothesis-profile=ci -n auto # the same with 100 cases per property on every CPU, as CI runs PyMeTTa
uv run pytest --hypothesis-profile=ci -n 4 --store-durations --clean-durations # refresh .test_durations, which balances CI's PyMeTTa runners
uv run ruff check . # lint, as CI does
uv run ty check semrewrite # type check, with every extra installed
uv run semrewrite conformance --engine pymetta
uv run python examples/walkthrough.py # rerun every command this README and examples/README.md showEvery transcript in this README and in examples/README.md sits between BEGIN GENERATED and END GENERATED markers and is what the walkthrough printed; tests/test_examples.py checks that it covers every block. The experiment's scripts are under experiments/ and its results under results/; RESULTS.md says how to rerun them.
Apache License 2.0; see LICENSE.