unified: Output source code skeletons in AST dump by tausbn · Pull Request #22624 · github/codeql

tausbn · 2026-09-18T11:32:30Z

This PR makes it so that output AST nodes (with locations that span part of the source code) now get an additional source=... annotation that compactly expresses how much of the source code is included in the location.

For example, a variable declaration might look like this:

variable_declaration source="⟨modifier⟩ ⟨pattern⟩ = ⟨value⟩"

Here, the bits inside Unicode angle brackets correspond to sub-ranges of the input that are owned by direct children of the node in question. The string inside the brackets gives the name of the corresponding field. (I chose Unicode angle brackets because they are unlikely to occur naturally in source code.)

This provides us with (hopefully) an intuitive and compact way of representing locations, without needing to list particular row and column offsets (which are impossible to inspect manually anyway).

First commit contains the AST dumper changes, the second updates the corpus with all of the new source skeletons.

Copilot

Copilot review overview

🟡 Changes recommended

Skeleton generation currently hides unnamed delimiters and fails to validate malformed zero-width child ranges.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2

Open (2)

Skip hidden unnamed tokens when building skeletons · New
Validate zero-width child ranges before ignoring them · New

What changed in this PR

Adds opt-in source skeleton annotations to Yeast AST dumps and enables them for generated Swift corpus expectations.

Changes:

Adds source-range skeleton generation, validation, and unit tests.
Enables skeletons in unified Swift corpus tests.
Documents the feature and regenerates Swift outputs.

Swift corpus paths below are relative to unified/extractor/tests/corpus/.

File	Description
`shared/yeast/src/dump.rs`	Implements source skeleton dumping and tests.
`shared/yeast/doc/yeast.md`	Documents source skeleton output.
`unified/extractor/tests/corpus_tests.rs`	Enables skeletons for mapped Swift AST dumps.
`swift/variables/var-without-initialiser.output`	Updates generated variable output.
`swift/variables/var-binding.output`	Updates generated variable output.
`swift/variables/tuple-destructuring-binding.output`	Updates generated variable output.
`swift/variables/property-with-willset-and-didset-observers.output`	Updates generated observer output.
`swift/variables/multiple-bindings-on-one-line.output`	Updates generated binding output.
`swift/variables/let-with-type-annotation.output`	Updates generated variable output.
`swift/variables/let-binding.output`	Updates generated variable output.
`swift/variables/compound-assignment.output`	Updates generated assignment output.
`swift/variables/binding-modifier-does-not-leak-into-initializer.output`	Updates generated binding output.
`swift/variables/assignment.output`	Updates generated assignment output.
`swift/types/actor-declaration.output`	Updates generated type output.
`swift/types/binding-modifier-does-not-leak-into-accessor-body.output`	Updates generated accessor output.
`swift/types/class-function.output`	Updates generated class output.
`swift/types/class-inheritance.output`	Updates generated inheritance output.
`swift/types/class-with-initializer.output`	Updates generated class output.
`swift/types/class-with-method.output`	Updates generated class output.
`swift/types/class-with-multiple-base-types.output`	Updates generated inheritance output.
`swift/types/class-with-stored-properties.output`	Updates generated class output.
`swift/types/computed-property.output`	Updates generated property output.
`swift/types/conditional-compilation-in-class-body.output`	Updates generated conditional output.
`swift/types/constructor-with-parameters.output`	Updates generated constructor output.
`swift/types/empty-class.output`	Updates generated class output.
`swift/types/enum-with-associated-values.output`	Updates generated enum output.
`swift/types/enum-with-cases.output`	Updates generated enum output.
`swift/types/enum-with-comma-separated-cases-chained-declaration.output`	Updates generated enum output.
`swift/types/extension.output`	Updates generated extension output.
`swift/types/function-type-with-convention-attribute.output`	Updates generated function-type output.
`swift/types/function-type-with-sendable-attribute.output`	Updates generated function-type output.
`swift/types/generic-class-parameters-and-constraints.output`	Updates generated generic-type output.
`swift/types/generic-type-arguments.output`	Updates generated generic-type output.
`swift/types/inline-array-type.output`	Updates generated array-type output.
`swift/types/noncopyable-type.output`	Updates generated noncopyable-type output.
`swift/types/property-with-getter-and-setter.output`	Updates generated property output.
`swift/types/protocol-declaration.output`	Updates generated protocol output.
`swift/types/protocol-with-read-only-and-read-write-property-requirements.output`	Updates generated protocol output.
`swift/types/qualified-type.output`	Updates generated qualified-type output.
`swift/types/static-function.output`	Updates generated static-function output.
`swift/types/struct.output`	Updates generated struct output.
`swift/optionals-and-errors/catch-where-clauses.output`	Updates generated error-handling output.
`swift/optionals-and-errors/do-catch.output`	Updates generated error-handling output.
`swift/optionals-and-errors/force-unwrap.output`	Updates generated optional output.
`swift/optionals-and-errors/nil-coalescing.output`	Updates generated optional output.
`swift/optionals-and-errors/optional-chaining.output`	Updates generated optional output.
`swift/optionals-and-errors/optional-enum-case-binding.output`	Updates generated optional-pattern output.
`swift/optionals-and-errors/optional-type-annotation.output`	Updates generated optional-type output.
`swift/optionals-and-errors/throwing-function.output`	Updates generated throwing-function output.
`swift/optionals-and-errors/try-expression.output`	Updates generated try-expression output.
`swift/optionals-and-errors/try-expression-2.output`	Updates generated try-expression output.
`swift/operators/addition.output`	Updates generated operator output.
`swift/operators/comparison.output`	Updates generated operator output.
`swift/operators/custom-postfix-operator.output`	Updates generated operator output.
`swift/operators/division.output`	Updates generated operator output.
`swift/operators/equality.output`	Updates generated operator output.
`swift/operators/logical-and.output`	Updates generated operator output.
`swift/operators/logical-not.output`	Updates generated operator output.
`swift/operators/logical-or.output`	Updates generated operator output.
`swift/operators/multiplication.output`	Updates generated operator output.
`swift/operators/operator-precedence-addition-and-multiplication.output`	Updates generated precedence output.
`swift/operators/parenthesised-expression.output`	Updates generated expression output.
`swift/operators/partial-range-from.output`	Updates generated range output.
`swift/operators/range-operator.output`	Updates generated range output.
`swift/operators/subtraction.output`	Updates generated operator output.
`swift/operators/unresolved-operator-sequence.output`	Updates generated operator output.
`swift/operators/unresolved-operator-sequence-with-casts.output`	Updates generated operator output.
`swift/operators/unresolved-operator-sequence-with-ternary.output`	Updates generated operator output.
`swift/loops/break-and-continue.output`	Updates generated loop output.
`swift/loops/for-in-over-array-literal.output`	Updates generated loop output.
`swift/loops/for-in-over-range.output`	Updates generated loop output.
`swift/loops/for-in-with-where-clause.output`	Updates generated loop output.
`swift/loops/repeat-while-loop.output`	Updates generated loop output.
`swift/loops/while-loop.output`	Updates generated loop output.
`swift/literals/boolean-literals.output`	Updates generated literal output.
`swift/literals/floating-point-literal.output`	Updates generated literal output.
`swift/literals/integer-literal.output`	Updates generated literal output.
`swift/literals/line-magic-literal.output`	Updates generated literal output.
`swift/literals/negative-integer-literal.output`	Updates generated literal output.
`swift/literals/nil-literal.output`	Updates generated literal output.
`swift/literals/string-literal.output`	Updates generated literal output.
`swift/literals/string-with-interpolation.output`	Updates generated interpolation output.
`swift/functions/call-with-inout-argument.output`	Updates generated call output.
`swift/functions/constructor-call-with-type-arguments.output`	Updates generated constructor-call output.
`swift/functions/function-call.output`	Updates generated call output.
`swift/functions/function-call-with-labelled-arguments.output`	Updates generated call output.
`swift/functions/function-with-default-parameter-value.output`	Updates generated function output.
`swift/functions/function-with-inout-parameter.output`	Updates generated function output.
`swift/functions/function-with-named-parameters.output`	Updates generated function output.
`swift/functions/function-with-no-parameters.output`	Updates generated function output.
`swift/functions/function-with-parameters-and-return-type.output`	Updates generated function output.
`swift/functions/generic-function.output`	Updates generated generic-function output.
`swift/functions/generic-type-alias.output`	Updates generated type-alias output.
`swift/functions/leading-dot-expression-call.output`	Updates generated call output.
`swift/functions/leading-dot-expression-value.output`	Updates generated expression output.
`swift/functions/method-call.output`	Updates generated method-call output.
`swift/functions/nested-function-type.output`	Updates generated function-type output.
`swift/functions/variadic-function.output`	Updates generated variadic-function output.
`swift/expressions/array-type-constructor.output`	Updates generated array-constructor output.
`swift/expressions/array-type-metatype.output`	Updates generated metatype output.
`swift/expressions/consume-expression.output`	Updates generated expression output.
`swift/expressions/copy-expression.output`	Updates generated expression output.
`swift/expressions/generic-specialization-expression.output`	Updates generated specialization output.
`swift/expressions/key-path-expression.output`	Updates generated key-path output.
`swift/expressions/unsafe-expression.output`	Updates generated expression output.
`swift/desugar/additive-expression-is-desugared.output`	Updates generated desugaring output.
`swift/desugar/another-additive-expression-is-desugared.output`	Updates generated desugaring output.
`swift/desugar/import-with-deeply-nested-path-three-parts.output`	Updates generated import output.
`swift/desugar/import-with-dotted-path-two-parts.output`	Updates generated import output.
`swift/desugar/scoped-import-uses-name-pattern.output`	Updates generated import output.
`swift/desugar/simple-import-with-single-name.output`	Updates generated import output.
`swift/control-flow/binding-modifier-does-not-leak-to-sibling.output`	Updates generated control-flow output.
`swift/control-flow/defer-statement.output`	Updates generated defer output.
`swift/control-flow/discard-statement.output`	Updates generated discard output.
`swift/control-flow/fallthrough.output`	Updates generated switch output.
`swift/control-flow/guard-let.output`	Updates generated guard output.
`swift/control-flow/if-case-let-with-shadowing-in-condition-value.output`	Updates generated conditional output.
`swift/control-flow/if-else-if-chain.output`	Updates generated conditional output.
`swift/control-flow/if-else.output`	Updates generated conditional output.
`swift/control-flow/if-let-optional-binding.output`	Updates generated conditional output.
`swift/control-flow/if-statement.output`	Updates generated conditional output.
`swift/control-flow/nested-enum-case-pattern.output`	Updates generated pattern output.
`swift/control-flow/switch-case-item-where-clauses.output`	Updates generated switch output.
`swift/control-flow/switch-expression-pattern.output`	Updates generated switch output.
`swift/control-flow/switch-statement.output`	Updates generated switch output.
`swift/control-flow/switch-with-binding-pattern.output`	Updates generated switch output.
`swift/control-flow/switch-with-labeled-case-pattern-arguments.output`	Updates generated switch output.
`swift/control-flow/ternary-expression.output`	Updates generated ternary output.
`swift/collections/array-literal.output`	Updates generated collection output.
`swift/collections/dictionary-literal.output`	Updates generated collection output.
`swift/collections/dictionary-subscript.output`	Updates generated subscript output.
`swift/collections/empty-array-literal-with-type.output`	Updates generated collection output.
`swift/collections/set-literal.output`	Updates generated collection output.
`swift/collections/subscript-access.output`	Updates generated subscript output.
`swift/collections/tuple-literal.output`	Updates generated tuple output.
`swift/collections/tuple-member-access.output`	Updates generated tuple output.
`swift/closures/closure-with-capture-list.output`	Updates generated closure output.
`swift/closures/closure-with-explicit-parameters.output`	Updates generated closure output.
`swift/closures/closure-with-shorthand-parameters.output`	Updates generated closure output.
`swift/closures/multi-statement-closure.output`	Updates generated closure output.
`swift/closures/trailing-closure.output`	Updates generated closure output.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

With this change, output AST nodes (with locations that span part of the source code) now get an additional `source=...` annotation that compactly expresses how much of the source code is included in the location. For example, a variable declaration might look like this: ``` variable_declaration source="⟨modifier⟩ ⟨pattern⟩ = ⟨value⟩" ``` Here, the bits inside Unicode angle brackets correspond to sub-ranges of the input that are owned by direct children of the node in question. The string inside the brackets gives the name of the corresponding field. (I chose Unicode angle brackets because they are unlikely to occur naturally in source code.) This provides us with (hopefully) an intuitive and compact way of representing locations, without needing to list particular row and column offsets (which are impossible to inspect manually anyway).

Regenerates all of the existing corpus output files to include source skeletons. Note that this already reveals some issues with our current locations: - arguments with trailing commas have a location that includes said comma, - switch_case nodes contain a `block` that encompasses the `case` statement itself. These will be fixed in due course.

asgerf

Looks great! I really like this way of visualising the source range

tausbn added the no-change-note-required This PR does not need a change note label Sep 18, 2026

tausbn changed the title ~~Tausbn/unified corpus source skeletons~~ unified: Output source code skeletons in AST dump Sep 18, 2026

tausbn requested a balanced review from Copilot September 18, 2026 11:36

Copilot started reviewing on behalf of tausbn September 18, 2026 11:36 View session

Copilot AI reviewed Sep 18, 2026

View reviewed changes

Comment thread shared/yeast/src/dump.rs

Comment thread shared/yeast/src/dump.rs

github-actions Bot added the documentation label Sep 18, 2026

tausbn force-pushed the tausbn/unified-corpus-source-skeletons branch from 1714793 to 2f0f02b Compare September 18, 2026 11:51

tausbn added 2 commits September 18, 2026 12:05

tausbn force-pushed the tausbn/unified-corpus-source-skeletons branch from 2f0f02b to a0f9f35 Compare September 18, 2026 12:06

tausbn marked this pull request as ready for review September 18, 2026 12:09

tausbn requested a review from a team as a code owner September 18, 2026 12:09

asgerf approved these changes Sep 18, 2026

View reviewed changes

tausbn merged commit 09d8d9a into main Sep 18, 2026
86 checks passed

tausbn deleted the tausbn/unified-corpus-source-skeletons branch September 18, 2026 12:42

Navigation Menu

unified: Output source code skeletons in AST dump - #22624

unified: Output source code skeletons in AST dump#22624
tausbn merged 2 commits into
mainfrom
tausbn/unified-corpus-source-skeletons

tausbn commented Sep 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

asgerf left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tausbn commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

asgerf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tausbn commented Sep 18, 2026 •

edited

Loading