Raw citation control markers such as `citeturn1view0` leak into Codex TUI output

[mushikingh]

Summary

When Codex CLI is routed through OpenCodex to the GitHub Copilot provider, assistant messages containing web citations can display OpenAI/ChatGPT-style citation control markers literally:

The setting is supported. citeturn1view0turn1view1

The same raw markers are retained in the Codex transcript. They are not rendered as source links, converted to a readable citation, or removed.

The delimiters are Unicode private-use characters. The example above is:

\uE200cite\uE202turn1view0\uE202turn1view1\uE201

• U+E200 begins the citation marker.
• U+E202 separates cite and each source reference.
• U+E201 ends the marker.
• Values such as turn1view0 are opaque, turn-scoped source identifiers and are not useful when displayed directly to the user.

This appears to be a response-compatibility issue: citation presentation syntax reaches the TUI as ordinary assistant text, but the TUI does not understand or render it.

Environment

Observed on September 1, 2026 with:

• OpenCodex package: @bitkyc08/opencodex@2.10.1-preview.20260805
• Installed package artifact: 2.10.1-preview.20260805-4f47f002f0b85fef
• Codex CLI: 0.151.0
• Provider: github-copilot
• Model route: github-copilot/gpt-5.6-sol
• OS: macOS 26.6.2, Apple Silicon (arm64)
• Terminal: TERM=xterm-256color, COLORTERM=truecolor
• API mode: streaming OpenAI Responses API through the local OpenCodex proxy

Minimal relevant Codex configuration:

model = "github-copilot/gpt-5.6-sol"
openai_base_url = "http://127.0.0.1:10100/v1"

Reproduction

1. Start OpenCodex:

   ocx start --port 10100

2. Configure Codex CLI to use the local OpenCodex Responses endpoint and a GitHub Copilot-backed model:

   model = "github-copilot/gpt-5.6-sol"
   openai_base_url = "http://127.0.0.1:10100/v1"

3. Start Codex CLI.

4. Ask a question that causes a web lookup and requires source citations. For example:

   Verify the current Codex configuration value for automatic approval review using official documentation.
   

5. Observe a citation-bearing commentary or final assistant message.

6. Open the Codex transcript with Ctrl+T.

Actual behavior

The TUI and transcript contain literal private-use citation syntax:

The valid value is auto_review. citeturn1view0turn1view1

Consequences:

• The user sees unusual glyphs and internal source IDs.
• The references are not clickable or human-readable.
• Copying the answer also copies the protocol/presentation markers.
• The persisted transcript contains client-specific rendering artifacts.
• Multiple source IDs make the leaked text especially noisy.

The issue occurs in both intermediate commentary and the final answer, so it does not appear limited to one Codex message phase.

Expected behavior

Raw citation control markers should never be visible in user-facing assistant text.

Depending on available source metadata and client capability, OpenCodex should do one of the following:

1. Preserve/emit structured url_citation annotations that the client can render.
2. Convert citations to a readable TUI fallback, such as Markdown links or a short Sources list.
3. If the opaque reference cannot be resolved, remove the presentation marker cleanly rather than exposing private-use delimiters and internal IDs.

The desktop citation/Sources-chip behavior should remain intact.

Relevant OpenCodex implementation details

The installed source already has structured web-source support:

• src/types.ts

  • OcxUrlCitation represents a web source.
  • Its comment explicitly notes that the desktop app reads url_citation annotations while the TUI ignores annotations.

• src/web-search/loop.ts

  • Collects and deduplicates web-search sources.
  • Sends them through web_search_call_end.

• src/bridge.ts

  • Accumulates pendingWebSources.
  • takeWebAnnotations() converts them into url_citation annotations.
  • closeCurrentMessage() emits response.output_text.done, response.content_part.done, and response.output_item.done.
  • The non-streaming flushText() path also attaches annotations.

• src/web-search/parse.ts

  • Parses structured url_citation annotations.
  • Also extracts a trailing Markdown Sources: block.

I could not find handling for the private-use marker grammar (U+E200, U+E202, and U+E201) in the installed OpenCodex source.

The observed turnNviewN references may originate from a Codex-hosted web tool rather than OpenCodex's synthetic web-search sidecar. Therefore, the first diagnostic step should be to determine whether the markers are:

1. Already present as literal text in the upstream GitHub Copilot response.
2. Produced while OpenCodex translates an upstream response into Responses API events.
3. Present only after Codex CLI processes an otherwise structured downstream response.

This report does not assume which component originally creates the marker. The actionable compatibility problem is that the OpenCodex-routed path allows it to reach user-visible TUI text without a usable fallback.

Suggested investigation

Capture one sanitized failing streaming turn at both sides of the proxy:

1. The upstream provider's assistant text/events.
2. OpenCodex's downstream Responses API SSE events.

Inspect these downstream event fields in particular:

response.output_text.delta
response.output_text.done
response.content_part.done
response.output_item.done
response.completed

Questions to answer:

• Do the private-use characters first appear upstream or downstream?
• Are url_citation annotations present at the same time as literal markers?
• Does the final completed response differ from the streamed deltas?
• Does the behavior reproduce in non-streaming mode?
• Does it reproduce with another OpenCodex provider?
• Does the same request render correctly when Codex uses an official provider directly?
• Does the Codex desktop app render the response correctly while the TUI leaks the marker?

Possible fix direction

If OpenCodex receives these markers as literal provider text, add a narrowly scoped citation normalizer at the assistant-output bridge boundary.

Important implementation constraints:

• The parser must be streaming-safe. A marker can be split across arbitrary response.output_text.delta boundaries.
• Normalize both streaming deltas and authoritative final text so .delta, .done, content-part, output-item, and completed snapshots remain consistent.
• Match only the exact citation grammar; do not remove arbitrary Unicode private-use characters globally.
• Apply normalization only to assistant output, never user messages or tool-result payloads.
• Avoid emitting both a normalized textual fallback and duplicate structured annotations.
• Preserve existing desktop url_citation/Sources-chip behavior.
• If source IDs can be mapped to URLs from the associated tool result, prefer proper annotations or readable Markdown links.
• If no mapping exists, fail closed by removing the control syntax and opaque IDs from display rather than leaking them.

A stateful stream normalizer may need to retain a small suffix beginning at U+E200 until it sees U+E201 or can determine that the sequence is not a valid citation marker.

Suggested tests

Add unit/integration coverage for both streaming and non-streaming response paths:

1. One citation with one source:

   before \uE200cite\uE202turn1view0\uE201 after
   

2. One citation with multiple sources.

3. Multiple citations in one assistant message.

4. A citation adjacent to punctuation and at end-of-message.

5. Every possible application-level split between Unicode scalar values within the marker. If normalization happens below SSE decoding, also test transport-byte splits within a multibyte UTF-8 delimiter.

6. A marker split across the final delta and .done event.

7. Existing structured url_citation annotations without marker text.

8. Marker text plus structured annotations, ensuring sources are not duplicated.

9. A malformed/unclosed marker, ensuring output is deterministic and no stream data is lost.

10. Ordinary Unicode and unrelated private-use characters, ensuring they remain unchanged.

11. Commentary-phase and final-answer messages.

12. Transcript/snapshot output, ensuring raw markers are not persisted.

Acceptance criteria

• No literal U+E200 cite U+E202 ... U+E201 sequence is displayed in Codex TUI output.
• No raw citation sequence is retained in the transcript.
• Citation-bearing responses remain readable when the TUI cannot render annotations.
• Resolvable sources remain attributable through annotations, Markdown links, or a readable source list.
• Streaming and non-streaming outputs are equivalent.
• Citation markers split across SSE chunks are handled correctly.
• Existing desktop Sources-chip behavior is not regressed.
• User/tool content containing similar text is not modified.

[github-actions]

_{Automated translation bookkeeping — detected language: English.}

[Ingwannu]

Current dev confirms one important part of this report: there is no handler for the exact U+E200 / U+E202 / U+E201 citation grammar, and the GitHub Copilot Responses repair currently relays ordinary assistant response.output_text unchanged. If Copilot sends this syntax as literal text, OpenCodex will expose it to Codex exactly as reported.

I am keeping this open as a real compatibility candidate, but I am not adding a global private-use-character scrubber from the report alone. The reproduction uses OpenCodex 2.10.1-preview while current dev has a newer provider-scoped Copilot stream repair, and the first component that creates the marker is still unproven. A global or bridge-wide strip could also corrupt ordinary assistant output that is intentionally discussing this syntax.

Please attach one sanitized reproduction from the current release or current dev containing:

• the upstream Copilot SSE blocks for the affected assistant text;
• the corresponding downstream OpenCodex SSE blocks;
• response.output_text.delta, .done, content_part.done, output_item.done, and response.completed when present;
• the non-streaming response for the same prompt if it also reproduces.

Remove Authorization, account ids, request prompts, and unrelated content; keep only event types and the citation-bearing text fields. Do not post credentials.

If the marker is already literal upstream, the scoped fix will live in the GitHub Copilot Responses repair: an exact-grammar, stateful per-output normalizer for split deltas plus the authoritative done/completed snapshots, with raw user/tool content untouched and existing structured url_citation annotations preserved. If it first appears downstream, the fix belongs at that exact translation step instead. This evidence decides the correct boundary rather than only the visible symptom.

[lidge-jun]

리뷰 · 우선순위 56 / 80

설명

이 이슈는 Codex CLI가 OpenCodex를 통해 GitHub Copilot 제공자를 쓸 때, 웹 인용이 OpenAI/ChatGPT 스타일 citation control marker 그대로 TUI와 트랜스크립트에 남는다는 호환 버그입니다. 예시는 citeturn1view0turn1view1 이고, 유니코드 private-use U+E200/U+E202/U+E201 로 감싼 cite 토큰입니다. turn-scoped opaque id라 사용자에게 보여줄 값이 아닙니다. 보고 환경은 오래된 preview 패키지 @bitkyc08/opencodex@2.10.1-preview... 와 Codex CLI 0.151.0 입니다.

지금 HEAD를 보면 src/web-search 과 src/bridge.ts 는 url_citation 어노테이션을 Sources 칩으로 붙이는 경로가 있습니다. 하지만 Copilot/업스트림이 assistant 텍스트 안에 private-use cite 마커를 인라인으로 넣는 경우, TUI는 그것을 렌더하지 않고 글자 그대로 보여 줍니다. OpenCodex가 응답 텍스트를 클라이언트로 전달하기 전에 마커를 제거하거나 url_citation 으로 정규화하는 계층이 있는지는 이 이슈만으로 단정할 수 없습니다. rg 기준 U+E200/stripCit 전용 처리는 src에 보이지 않았습니다.

버전 괴리가 큽니다. 보고는 2.10.1-preview 이고 현재 패키지는 2.40.0, HEAD는 remote hub 이후입니다. 먼저 2.39+/2.40 에서 Copilot+web citation 재현이 되는지 확인해야 합니다. 재현되면 Responses/Chat 정규화 한곳(예: adapter 또는 bridge 출구)에서 private-use cite 블록을 strip 또는 변환하는 패치가 맞습니다. types/config 분할 무관합니다. 점수는 56입니다. 보기 흉한 UX이지만 라우팅/인증 사고는 아닙니다.

경로 유니코드 U+E200–E202 cite 블록 - TUI가 모르는 presentation syntax입니다. 링크 URL이 마커 안에 없고 turn id만 있으면 strip이 안전합니다
경로 src/bridge.ts url_citation - desktop Sources 칩 경로와는 별개일 수 있습니다. Copilot 인라인 마커는 다른 입구입니다
경로 보고 버전 2.10.1-preview - 현재 2.40.0과 너무 멉니다. 재현 요청이 먼저입니다
경로 GitHub Copilot provider - #3156/#3163 context window와 같은 제공자 축이지만 증상이 다릅니다. 한 PR로 묶지 마십시오

메인테이너의 판단이 필요한 지점

• strip-only vs url_citation 변환 중 무엇인지. URL이 없으면 strip이 맞습니다
• Codex 쪽 렌더 수정에 맡길지, 프록시가 항상 정규화할지
• 최신 버전 재현 전까지 needs-repro 로 둘지

너의 추천
최신 dev/2.40.0 에서 Copilot 인용 재현을 요청하는 댓글을 남기고, 재현되면 private-use cite 블록 strip 패치를 adapter/응답 출구에 둡니다. #3163과 묶지 않습니다. 라벨은 바꾸지 않습니다.

이 댓글은 grok-bot이 작성했습니다

Translated Message

Original language: Korean

Review · Priority 56 / 80

Description

This issue is a compatibility bug where, when Codex CLI uses the GitHub Copilot provider through OpenCodex, web citations remain as OpenAI/ChatGPT-style citation control markers in the TUI and transcripts. An example is citeturn1view0turn1view1, which is a cite token wrapped in Unicode private-use characters U+E200/U+E202/U+E201. It's a turn-scoped opaque ID and should not be shown to users. The reported environment uses an older preview package @bitkyc08/opencodex@2.10.1-preview... and Codex CLI 0.151.0.

Looking at HEAD now, src/web-search and src/bridge.ts have paths that attach url_citation annotations as Sources chips. However, when Copilot/upstream inlines private-use cite markers inside assistant text, the TUI doesn't render them and displays them as literal characters. It cannot be determined from this issue alone whether there is a layer in OpenCodex that removes markers or normalizes them to url_citation before passing response text to the client. Based on rg searches, no dedicated handling for U+E200/stripCit is visible in src.

There is a large version gap. The report uses 2.10.1-preview, the current package is 2.40.0, and HEAD is post-remote hub. First, we need to confirm whether Copilot+web citation can be reproduced in 2.39+/2.40. If reproduced, a patch that strips or converts private-use cite blocks in one place for Responses/Chat normalization (e.g., adapter or bridge exit) would be appropriate. Separation of types/config is irrelevant. The score is 56. It's an ugly UX but not a routing/authentication concern.

Paths:

• Unicode U+E200–E202 cite blocks - presentation syntax unknown to TUI. If there's no link URL inside the marker and only a turn ID, stripping is safe
• src/bridge.ts url_citation - may be separate from the desktop Sources chip path. Copilot inline markers are a different entry point
• Report version 2.10.1-preview - too far behind current 2.40.0. Repro request comes first
• GitHub Copilot provider - same provider axis as Opencodex doesnt recognise context window from the github-copilot provider #3156/fix(catalog): read GitHub Copilot context window metadata #3163 context window but different symptoms. Do not bundle into one PR

Points requiring maintainer judgment:

• Whether to do strip-only or url_citation conversion. If there's no URL, stripping is correct
• Whether to leave render fixes to the Codex side or have the proxy always normalize
• Whether to keep it as needs-repro until reproduction with the latest version

Recommendation:

Leave a comment requesting reproduction of Copilot citations in the latest dev/2.40.0, and if reproduced, place a private-use cite block stripping patch at the adapter/response exit. Do not bundle with #3163. Do not change labels.

This comment was written by grok-bot

[lidge-jun]

Fixed by #3195, landed on dev as f3bcc67.

Your diagnosis was right on the first hypothesis. Searching the source for U+E200/U+E201/U+E202 and citeturn returns nothing: OpenCodex neither emits nor recognizes that grammar, and its citation support is entirely the structured url_citation path you found. The markers are already literal text in the upstream response, because the Copilot backend is ChatGPT-derived. The proxy is simply the last place that can see them before a client that cannot render them, so that is where the strip now happens.

Two things about the implementation worth knowing, since you clearly read the bridge:

A span can straddle a delta boundary — U+E200cite in one chunk and the rest in the next — so the streaming path uses a stateful filter rather than a per-delta strip. It withholds an unterminated tail and releases it at close, so a stream dying mid-marker still delivers the words the model produced instead of swallowing them.

The accumulated text is stripped separately, because closeCurrentMessage re-sends it in output_text.done, content_part.done and output_item.done. Filtering only the deltas would have produced a clean-looking stream and a transcript that still contained the markers — which is the half of your report about the saved transcript.

On your three expected behaviors: this implements option 3. Options 1 and 2 are not reachable here — turnNviewN ids are turn-scoped and opaque, and the response carries no mapping from them to a URL, so there is nothing to convert them into. The structured url_citation path is untouched, so desktop Sources chips are unaffected.

One caveat: I could not obtain a live Copilot citation response, so this is verified against synthesized text matching the exact grammar you documented plus the real bridge end-to-end. If you see a variant shape after upgrading, please reopen with the raw codepoints.

Thank you for the report — naming the exact codepoints and proposing the three candidate origins turned this from an investigation into a confirmation.