Skip to content

Stop tokenizing user content as special - track trust in the jinja template VM - #281

Open
llitz wants to merge 1 commit into
NVIDIA:mainfrom
llitz:2026-11-12-vm-trust-segments
Open

llitz wants to merge 1 commit into
NVIDIA:mainfrom
llitz:2026-11-12-vm-trust-segments

Conversation

@llitz

@llitz llitz commented Nov 13, 2026 •

Copy link
Copy Markdown

Rendered chat templates interleave template-emitted text with user content. Special tokens arriving in user content were tokenized with the same tokenizer as template output, so a prompt quoting a marker such as <|vision_start|> could inject a real special-token id into the model stream.

The renderer now records, per output span, whether the template VM produced it or user content flowed through it:

  • TokenSegment carries a trust bit; jinja emits trusted spans for template literals and untrusted spans for injected values, including values reaching the output through namespace containers, function results, and macro captures (the bit recurses with the value).
  • Adjacent same-trust spans merge, so runs are maximal and token ids are identical to single-segment encoding for untrusted prompts.
  • Untrusted spans tokenize through a separate ordinary tokenizer that cannot emit special tokens; trusted spans keep the template tokenizer.
  • expand_prompt counts and splices image placeholder ids from prompt_token_ids instead of re-splitting the text, which keeps placeholder accounting correct across trust boundaries and lets ImagePrompt drop its write-only text field (the field had no runtime readers; its single reader sat in a test driver this patch fixes in the same motion).

Testing:

  • new unit tests: pasted markers render into untrusted spans while template-emitted envelopes stay trusted, including through captures and macros; maximal-run merging keeps encoding identical to single-segment for ordinary prompts;
  • the full flare-helios fixture suite (FLARE_TEST_TEMPLATE_DIR prepared via tests/frontend/prepare_templates.py): 580 passed;
  • the test_images fixture driver builds again (it had been left unbuildable by the signature change this patch supersedes).

@original-el8 original-el8 added area:api Serving API, protocol encoding, tool parsing, and client-visible behavior. bug Something isn't working priority:P1 High attention: unresolved correctness, serving failure, or model-data deletion risk. triage:design-review Needs a maintainer scope, architecture, or contract decision before implementation or integration. labels Nov 13, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

Prioritized P1 frontend correctness and categorized for design review. The report shows that special-token text supplied as user content can become protocol token IDs. The proposed repair carries trust provenance through the template VM and uses ordinary tokenization for untrusted spans.

Review provenance through containers, macros, captures and function results, ordinary-text tokenization across span boundaries, and multimodal placeholder accounting. The reported 580 fixture tests support that review; preserve legitimate template-emitted protocol markers while ensuring quoted user markers cannot create them. No specific affected model is established by the PR description.

@original-el8 original-el8 self-assigned this Nov 13, 2026

@original-el8 original-el8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the shared frontend is the appropriate place for this repair, but the implementation still allows request content to become trusted and introduces regressions in ordinary tokenization and image prompts. The four inline findings were reproduced at 6b44d78. They violate the prompt-renderer requirements to preserve ordinary token IDs, distinguish literal request markers from protocol markers, and retain image placeholders.

Validation: all 21 existing Jinja unit tests passed with the pinned external templates prepared by tests/frontend/prepare_templates.py. Additional temporary regression tests exercised the actual VM and Frontend::render_chat; Qwen tokenization checks used the recipe's pinned nvidia/Qwen3.8-Flash-Next-FP4-Local template and tokenizer at b9ce18a6887e18c089fdea8509a33e63429a3300. The Gemma provenance reproduction used the pinned chat_template.google fixture. These checks reproduced the failures described inline; the existing Jinja tests do not catch them. I did not run the full reported 580-test suite, image preprocessing, or GPU serving.

Please preserve provenance through indexing, containers, calls, and string transformations, and preserve ordinary BPE tokenization across provenance boundaries. Add regression tests for the concrete cases below before merging.

Comment thread helios/src/frontend/jinja/vm.rs Outdated
}
let value = self.hinted(v.index(&key));
self.stack.push(value);
self.stack_trust.push(trust || key_trust);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve request provenance when indexing

trust || key_trust makes an untrusted container's value trusted whenever the index is a template literal. For example, rendering {{ messages[0]['content'] }} with request content <|image_pad|> produces [(13, true)]. This also affects the pinned Gemma template: a user message containing <turn|> is emitted in a trusted run through message['content'], so it still reaches the tokenizer that recognizes protocol tokens. A trusted lookup key must not confer trust on the selected request value. Add both the minimal indexing case and an actual recipe-template regression.

Comment on lines +493 to +508
let (v, trust) = self.pop_trusted();
input_trust = trust;
filter(name, v, args, &kw_named, !self.hint_texts.is_empty())?
}
_ => {
let v = self.pop();
method(name, v, args, &kwargs)?
let (v, trust) = self.pop_trusted();
input_trust = trust;
method(name, v, args, &kw_named)?
}
};
let trusted = match op {
I::Test(_) => true,
I::Call(_) => !result.carries_untrusted(),
_ => arguments_trusted && input_trust && !result.carries_untrusted(),
};
let result = Self::force_trust(result, trusted);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve mixed trust runs through trimming

The trim implementation reconstructs a changed string with Value::marked, which drops its trust runs. This filter path then makes the whole result untrusted when its input contained request text, including legitimate template-emitted image markers. Reproduced with the pinned Qwen template and tokenizer: a user message containing one image followed by text hello retains one image placeholder token, but changing the text to hello produces zero placeholder tokens, while the rendered text still contains the image envelope. expand_prompt consequently rejects the placeholder count for that valid image request. Trim must slice and remap provenance alongside text and checkpoint hints instead of discarding it.

Comment thread helios/src/frontend/mod.rs Outdated
Comment on lines +724 to +733
let mut segments = Vec::with_capacity(runs.len());
let mut cursor = 0usize;
for (end, allow_special) in runs {
if end > cursor {
segments.push(TokenSegment {
text: rendered[cursor..end].to_owned(),
allow_special,
image_index: None,
});
cursor = end;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve ordinary BPE merges across trust boundaries

Turning every trust run into an independently encoded segment changes token IDs even when the prompt contains no user-supplied protocol markers. With the pinned Qwen template/tokenizer, render a user message followed by an assistant lookup tool call with arguments {"x":"hello"} and options enable_thinking=false, add_generation_prompt=true. In the resulting <parameter=x> text, full-text encoding uses token 24582 for =x, while this path emits [28, 87] because = is trusted and x is untrusted. A minimal BPE vocabulary containing a, b, and ab reproduces the same problem with template text a followed by request text b. Merging only adjacent runs with equal trust cannot fix this: special-token eligibility must be enforced without introducing ordinary-text tokenization boundaries.

Comment on lines +85 to +101
pub fn carries_untrusted(&self) -> bool {
match self {
Self::String(s) => s.2.iter().any(|(_, trusted)| !trusted),
_ => false,
}
}
pub fn baked_untrusted(value: Self) -> Self {
match value {
Self::Str(s) => {
let runs = vec![(s.len(), false)];
Self::marked_runs(s.to_owned(), Vec::new(), runs)
}
Self::String(s) => Self::String(Rc::new((s.0.clone(), s.1.clone(), vec![(
s.0.len(),
false,
)]))),
other => other,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Carry provenance through containers and function results

Both helpers handle strings only, and I::Call derives result trust solely from carries_untrusted() without preserving argument provenance. As a result, {% set ns = namespace(body=m) %}{{ ns.body }} renders request value m='<|image_pad|>' as trusted. A separate reproduction assigns the request messages array to ns.body and iterates it; each message's content is also emitted as trusted because baked_untrusted leaves the array unchanged. This contradicts the claimed recursive provenance and leaves special-token injection possible through normal template operations. Preserve provenance through container construction, assignment, traversal, and builtin results; fixing literal indexing alone will not resolve these cases.

@llitz
llitz force-pushed the 2026-11-12-vm-trust-segments branch from 6b44d78 to 2118c84 Compare November 14, 2026 00:05
@llitz

llitz commented Nov 14, 2026

Copy link
Copy Markdown
Author

All four findings are fixed in 2118c846 (force-pushed), each with the regression test you asked for:

P1 indexing (vm.rs) — I::Index now pushes only the container's trust; the key's trust is discarded (selection never taints). I::Slice verified to already follow container trust. Regression: indexing_key_trust_does_not_taint_value covers the minimal messages[0]['content'] shape and the Gemma message['content'] shape.

P2 trim (vm.rs/value.rs) — trim/strip/lstrip/rstrip and the string branch of slice now rebuild via Value::sliced_annotations, remapping trust runs and checkpoint hints to the surviving byte range. Regression: trim_remaps_trust_runs covers your hello/hello ␣ placeholder-retention shape in both directions, plus rstrip and multibyte boundaries.

P3 ordinary tokenization (mod.rs) — implemented your proposed shape: the full rendered text is encoded in one pass with the template tokenizer (ordinary prompts are byte-identical to pre-patch encoding, merges across trust boundaries preserved), then tokens whose ids are in the added-token decoder and whose byte offsets overlap an untrusted span are grouped into maximal windows; each window's byte range is re-encoded with the ordinary tokenizer and spliced with byte-offset accounting. Regression: ordinary_bpe_merges_across_boundaries covers the minimal a/b/ab vocab, the <parameter=x> split, a boundary-straddling marker formed from user bytes, and multibyte UTF-8.

P4 provenance (value.rs/vm.rs) — baked_untrusted/carries_untrusted recurse through List/Map/Namespace; call results combine argument provenance with result trust (macro calls keep capture-derived run trust); filter/method results combine input, argument, and result trust; concat/add and string-append concatenate runs. Regression: namespace_provenance_recursive covers namespace(body=m), ns['body'], assignment, dict(), array iteration, and the |upper leak.

Validation: fixture suite with FLARE_TEST_TEMPLATE_DIR (583 passed; the only failure is the pre-existing torch-gated native_pixels_match_canonical_torchvision_processor, which fails identically on main in this environment), all four new regression groups green, test_images builds.

Two minor follow-ups found in our own second-pass review, disclosed for completeness (both over-taint — fail toward untrusted, no injection path): Map.get(key) currently taints the selected value with the key's trust (asymmetric with the P1 fix; safe direction, but a dynamic-key lookup loses a template-emitted special id), and the replace/capitalize/lower/upper/join/string/tojson filters rebuild without remapping runs (conservative bake; no shipped template routes an image placeholder through them). Happy to fold either into this PR or post a follow-up — your call.

…mplate VM

Rendered chat templates interleave template-emitted text with user
content. Special tokens arriving in user content were tokenized with the
same tokenizer as template output, so a prompt quoting a marker such as
<|vision_start|> could inject a real special-token id into the model
stream.

The renderer records, per output span, whether the template VM produced
it or user content flowed through it:

- The jinja VM emits trust runs: trusted spans for template literals and
  untrusted spans for injected values. Provenance travels inside values,
  so namespace and dict construction, attribute assignment, list and map
  literals, macro captures, iteration, and slicing preserve it, and an
  indexing or lookup key only selects a value; it never confers trust on
  it.
- Text-transform filters that shrink text (trim and the strip family)
  slice both the checkpoint hints and the trust runs to the surviving
  byte range instead of discarding them, so a template marker emitted
  next to user content keeps its special-token id across a trim.
- The whole rendered prompt encodes in one pass with the template
  tokenizer, preserving the pre-patch token ids for ordinary prompts,
  including BPE merges that cross trust boundaries. Tokens whose id
  comes from the added vocabulary and whose byte range overlaps an
  untrusted span are then re-encoded with an ordinary tokenizer that has
  no added vocabulary, and the replacement ids are spliced in with byte
  offset accounting, so user-quoted markers cannot produce protocol ids.
- expand_prompt counts and splices image placeholder ids from
  prompt_token_ids instead of re-splitting the text, which keeps
  placeholder accounting correct across trust boundaries and lets
  ImagePrompt drop its write-only text field (the field had no
  runtime readers; its single reader sat in a test driver this patch
  fixes in the same motion).

The fixture suite passes; ordinary prompts render byte-identical token
ids to full-text encoding while prompts carrying real image placeholders
keep their expansion through trim and mixed-trust paths.

Signed-off-by: Carlos Augusto <mb@lab.how>
@llitz
llitz force-pushed the 2026-11-12-vm-trust-segments branch from 2118c84 to 33d2dff Compare November 17, 2026 02:18
@llitz
llitz marked this pull request as ready for review November 17, 2026 02:19
@llitz

llitz commented Nov 17, 2026

Copy link
Copy Markdown
Author

Rebased onto current main (5fd4070d) — the conflicts with the Inkling 2 changes are resolved: the new PaddedGrid/ClassToken layouts keep working through the reduced expand_prompt signature (neither reads the removed field), tml.rs uses the lazy ordinary tokenizer, and the trust machinery is unchanged. Full fixture suite: 622 passed, 1 pre-existing environmental (torch-gated driver). All four P1 fixes and their regression tests are unchanged and green. Ready for re-review.

@llitz

llitz commented Nov 17, 2026

Copy link
Copy Markdown
Author

@original-el8 rebased onto current main — the Inkling 2 conflicts are resolved (new layouts keep working through the reduced expand_prompt signature; tml.rs uses the lazy ordinary tokenizer). Full fixture suite: 622 passed, 1 pre-existing environmental. The four P1 fixes and their regression tests are unchanged. Ready for re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:api Serving API, protocol encoding, tool parsing, and client-visible behavior. bug Something isn't working priority:P1 High attention: unresolved correctness, serving failure, or model-data deletion risk. triage:design-review Needs a maintainer scope, architecture, or contract decision before implementation or integration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants