Skip to content

Comparing an Opaque with Opaque | null no longer overflows the stack - #536

Merged
ASDAlexander77 merged 3 commits into
mainfrom
fix-opaque-nullable-compare-recursion
Oct 11, 2026
Merged

ASDAlexander77 merged 3 commits into
mainfrom
fix-opaque-nullable-compare-recursion

Conversation

@ASDAlexander77

Copy link
Copy Markdown
Owner

Problem

Comparing an Opaque with an Opaque | null crashed the compiler with a stack overflow (0xC00000FD) and printed nothing:

class State {
    current: Opaque | null;
    set(next: Opaque): boolean {
        if (next != this.current) { ... }   // tslang: 0xC00000FD, no diagnostic

Reported in docs/bugs/native-ts-graphics-findings.md, issue 1 (docs/bugs/ntg_opaque_nullable_compare.ts).

Cause

LogicOp (UnaryBinLogicalOrHelper.h), in the branch for a nullable left operand (isNullableTypeNoUnion) and a union on the right that needs no tag, called itself again with left, baseType, right, rightType: the union's base type went in as the left type, and the right type was still the union. That reaches the same branch again, forever. A WinDbg stack of the debug build showed LogicOp<CmpIOp, ..., ne, ...> calling itself until the stack ran out.

The minimal trigger is any reachable opaque != (opaque | null). A function that is never called is dropped before lowering, so it doesn't crash.

Fix

Pass the base type as the right type: left, leftType, right, baseType. The mirrored branch for a union on the left already did it this way.

Testing

  • New 00opaque_nullable_compare.ts (compile + JIT): the issue's class, Opaque vs Opaque | null with the union on either side, and a class vs Class | null with ===/!==. The current release crashes with 0xC00000FD.
  • Full ctest: 3915/3915 pass.

🤖 Generated with Claude Code

ASDAlexander77 and others added 3 commits October 11, 2026 12:58
`next != this.current`, with `current: Opaque | null`, crashed the
compiler with 0xC00000FD and no message. LogicOp, for a nullable left
operand and a union without a tag on the right, called itself again with
the union's base type as the LEFT type and the union still as the right
one - the same branch, forever. The base type now replaces the right
type, as for a union on the left.

Reported in docs/bugs/native-ts-graphics-findings.md (issue 1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ompare-recursion

# Conflicts:
#	tslang/test/tester/CMakeLists.txt
@ASDAlexander77
ASDAlexander77 merged commit e13766b into main Oct 11, 2026
2 checks passed
@ASDAlexander77
ASDAlexander77 deleted the fix-opaque-nullable-compare-recursion branch October 11, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant