Repository navigation
Conversation
Add a mapped hybrid reader, eager reference linking, field memoization and real Clava construction/codegen benchmarks. Preserve six-fixture fidelity checks, cache counters, memory probes, measurements and the recommendation to prefer eager integration with selective laziness.
…elds Generate typed Java bindings from the companion schema, map bounded blocks, and resolve all graph references before dropping construction indexes. Keep memoized scalar fields in existing Clava stores and retain immutable backing files. Add full-suite cache experiments and cross-format fidelity, mapping and lifetime checks.
Archive 27 baseline-equivalent runs, fidelity and memory checks, and excluded race/quota trials. Isolate experiment scratch files and stop matrices on unexpected failures. Document that the generated schema works but does not improve suite time or retained heap with blanket laziness.
Syntax validation now sends one native-only syntax-check option before compiler arguments. Add focused coverage for its argument placement.
Generate structural descriptors from official schema reflection and validate bounded offsets, vtables, vectors, strings, unions, scalar domains and required fields. Bound recursion and verification work, and reject references outside a single block. Validation: 15 isolated JUnit tests pass; templates, NAS LU and fidelity native streams verify across 191 blocks. Consumer generation and framing integrate this verifier in the following cutover change.
Exercise schema mismatch, header ordering, duplicate headers, missing and incorrect End records, empty blocks, trailing records and truncated framing. Verify mapped files can be deleted after both successful and failed reads. Validation: 25 structural and envelope tests pass in isolated JUnit runs on JDK 17 and JDK 26 against the current eager reader sources.
Reflect exact supplied Clava classes and their inherited public DataKeys into deterministic JSON with declaring constants and generic value types. Classify references, enums, optional values and lists without schema annotations or a naming alias map. Validation: Java 17 compilation succeeds; all 112 payload classes expose 2,186 keys and repeated inventory output is byte-identical.
Cover negative 64-bit IDs that would narrow to a valid 32-bit null sentinel, invalid zero and negative references, and scope isolation for translation-unit node IDs. Validation: 27 structural, envelope and ID tests pass against the current eager reader sources on JDK 17.
Verify schema-specific cache directories, reject invalid hashes and false CCACHE_DISABLE values, and check compressed cache invocation settings.
Cover exact release tag resolution, missing local manifests, canonical schema hashing, malformed archives and incompatible compiler versions.
Preserve experimental implementations on lazy-flatbuffers-experiment. Add fixed suite runners, structural corpus checks, batch consumer comparisons, cross-TU and round-trip probes, and source-identified repeated memory comparisons.
Keep nullable-node metadata beside the Java DataKey and reflect it into the consumer inventory, preserving VariableArrayType size expressions without producer annotations or binding aliases.
Resolve the exact selected schema and pinned compiler/runtime without sibling checkouts. Document schema evolution and add generated-source drift checks, including ARM compiler bootstrap and archive verification tests.
Find revision manifests stored directly in a frozen distribution and hash local native tools alongside the parser JAR.
Apply the same collection attempts and waiting period to every runtime, including historical controls that retain an AST. This avoids extra control-side GCs changing the peak measurement.
Run eager FlatBuffers, historical Text, and Protobuf runtimes sequentially against the same filtered Clava-JS suite. Record source and runtime identities, test counts, cache state, and the shared Vitest wall-time boundary. Remove ccache and its aliases from the runner PATH while setting CCACHE_DISABLE=true.
Use the shared filtered PATH for all transports and inspect live parser file descriptors alongside mappings, AST collection and temporary folders. Require every eager cycle to release those resources.
…anup Add a strict selected-corpus reparse mode with preserved compiler options, source hashes and stable generated bytes. Check mappings and open parser files across all previous parse iterations, and capture frozen runtime provenance from its own directory.
Render class arguments from each enclosing template parameter group, including nested primary and partial-specialization scopes. Keep enclosing headers before the member template header. Verify renamed, packed, non-type and mixed nested parameters through stable parse-generate-reparse tests.
Pin full corpus and audit hashes, preserve exact input options and record every excluded case before strict round-trip validation. Select all eager syntax-pass cases without manual exclusions or waivers.
Unwrap parenthesized pointees for cv/ref/noexcept suffix discovery, preserve method ref qualifiers, and globally qualify nonlocal member-pointer classes. Cover namespace shadowing, local classes, and template specializations through stable source round trips.
Prevent anonymous-record normalization from applying a later const typedef to the mutable first alias. Exercise assignment and stable reparsing, and update the member-pointer golden for global class qualification.
Ignore empty declaration output when joining a translation unit, preserve terminal newlines while adding inline comments, and avoid doubling trailing record newlines inside heterogeneous declaration statements. Verify byte-stable source regeneration.
Document hash-verified automatic selection of every compiler-accepted output, record exclusions without waivers, and distinguish the historical comparison filter from full integration-suite coverage.
Clang reports unsupported dependent friend methods in the lexical class. Preserve their original declaration and enclosing headers instead of inventing an owner or placing friend before template. Document that AST edits within these declarations are not emitted, and reject raw strings or continued literals rather than change their contents. Verify direct and nested friend round trips and explicit rejection cases.
Print Clang's GNU malloc and nothrow attributes with valid spellings, and stabilize the order of GNU attributes without moving inline or other attribute forms. Drop only empty OpenMP captured placeholders anchored at the standalone directive, preserving structured and macro-expanded bodies. Add a focused parse-generate-reparse fixture covering attribute semantics, standalone target directives, and structured OpenMP bodies.
Avoid adding an unintended lvalue qualifier to factory-created C functions. Verify generated C parses and remains stable across regeneration.
Prevent a record pointee and globally qualified owner from being parsed as one nested name. Normalize parentheses introduced by reparsing while preserving const qualifiers and pointer/array declarators.
Capture exact release and consumer artifact provenance, full suite and corpus results, strict roundtrip evidence, repeated-parse cleanup, and matched transport measurements. Document measured costs and outstanding CI/security findings for review.
lm-sousa
changed the title
Record nine matched diagnostic observations with explicit GC disabled equally. Qualify the earlier 18.9% penalty and reopen the speed adoption gate; the residual reader cost remains unattributed. Preserve production sources and the local dumper selector.
Record source-verified GC and StoreDefinition cache omissions, plus the narrower generated-reader specialization candidate. Separate existing equivalents, compression-policy tradeoffs and benchmark settings from portable runtime changes.
Reuse StoreDefinition metadata per Clava class with ConcurrentHashMap.computeIfAbsent, matching the Protobuf reader fast path. Record the GC and concurrency ports and accepted XStream incompatibility. The full parser check, coverage and generated drift verification pass with 234 tests and four expected skips; both enabled-cache probes also pass.
Record 27 serial observations with matched App workloads, source provenance and per-call evidence. Document the production GC improvement and remaining App construction penalty without claiming an individual-port speedup.
Expose the recorded Java and JS command durations, distinguish common benchmark workloads from unfiltered suites, and audit the earlier cache-state ranking without attributing the remaining regression to an unprofiled phase.
Run four rounds of both historical suite workloads in bypass, cold and warm states for cumulative App and separate command timing. Preserve historical control values and original chart renderers, retain same-day results, and correct the JS heap description.
Replace historical comparison controls with 144 accepted measurements of current builds in rotated serial rounds. Preserve original App and separate wall-time methodology, record matching workloads and cache counters, and publish current boxplots with complete provenance.
Use LF line endings for embedded CSV and trim trailing report whitespace. Keep all measured values unchanged and refresh renderer provenance.
lm-sousa
force-pushed
the
ast-flatbuffers
branch
from
October 7, 2026 16:04
a02653e to
907efcb
Compare
- Repair the cached dumper's execute permission in-process instead of forking chmod on every App call (~10 ms each). - Drop the executable digest from the ccache namespace. ccache's compiler check already covers the executable, and hashing the 77 MB dumper cost ~200 ms on the first cached parse of every JVM. Both match the Protobuf consumer so the two wire prototypes are compared on equal resource handling.
Lets Clava use pattern matching in switch. specs-java-libs stays on 17, which Java 21 modules can depend on. ClangAstParser and nightly CI follow in the next commit.
Clava no longer generates code for the FlatBuffers AST stream. The build used to download the dumper's schema, download or compile flatc, compile the FlatBuffers Java runtime from source, inventory ClavaAst DataKeys with a javac'd helper, run two Python generators and hash-check ~28k generated lines against a 416-line manifest. Now the schema's binary reflection (ast-wire.bfbs) is a resource read with the stock flatbuffers-java runtime, and NodeBindings maps payload tables onto DataKeys by the existing naming convention once per table. NodeBindingsTest binds every payload table, so a schema/key mismatch still fails in CI. Also: - Drop the generated structural verifier: buffer reads are bounds-checked and the stream comes from the selected release, whose schema hash is checked in the header. - Read one size-prefixed block at a time from a stream instead of mapping the file and unmapping it through sun.misc.Unsafe. - Drop End counter and per-node vector consistency checks, opt-in wire metrics and the zstd-jni dependency. - ClangAstParser targets Java 21; nightly CI runs on 21. The first parse in a JVM is ~195 ms faster (fewer classes to load) and steady-state parsing 2-4% faster on boolean.c, NAS BT and NAS LU.
clang-dumper v18.1.8_7-rc3 writes a node's children inside its Node record and drops the NodeClass, Children and Counter records. The reader builds a node when its Node record is read, so the pending payload/class bookkeeping is gone. Update the schema resource and select the release. Also drop ExecutableDigest: it only re-hashed local dumper builds against a manifest that CMake rewrites on every build.
The binary schema (ast-wire.bfbs) and its hash were committed resources that had to be regenerated by hand with flatc on every schema change. clang-dumper now publishes the schema as a release asset, so the wireSchema Gradle task fetches it for the release in clang-dumper-release.tag (or copies it from a local build), checks its sha256 against the manifest and writes both resources under build/. Selects v18.1.8_7-rc4, the first release that publishes the asset. The ClangAstParser check passes (209 tests).
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
lm-sousa
force-pushed
the
ast-flatbuffers
branch
from
October 18, 2026 03:33
ccf8872 to
aa09467
Compare
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clava now imports completed eager FlatBuffers v2 ASTs through the production path. It verifies the exact schema and binaries selected by
clang-dumper-release.tag, generates bindings through strict Java naming/type checks, and rejects malformed or incompatible streams. Lazy work is preserved onlazy-flatbuffers-experiment; there is no legacy selector or Text fallback.Warning: unresolved departures from intended behavior
These findings were identified after the earlier validation and review summaries below. They remain unfixed at the current PR head
907efcbbeand must be addressed before treating this implementation as production ready. The production direction has pivoted to Protobuf; this warning records what must not be forgotten if FlatBuffers work resumes.Unsupported node records can be silently discarded.
CompleteReader.readNoderecords an unsupported class without constructing its node, and the reader permits completion when the unsupported node is not needed by the retained graph.CompleteReaderUnsupportedNodeTest.ignoresAnUnsupportedUnreachableNodeexplicitly accepts this behavior. This can hide missing consumer support after a dumper upgrade. Passing corpus/closure checks therefore does not prove that every emitted node class is supported. The intended behavior is an explicit failure for an unsupported emitted node, including an unreachable one, with class, ID and source location where available. Intentional omissions must be explicit in the producer/schema contract; reachability alone must not justify silent dropping. Replace the acceptance test with rejection coverage. The historical Text reader was not uniformly strict, because it had generic-attribute and dummy-constructor fallbacks, but that does not authorize this silent-discard policy.Dependent friend member-template generation can silently lose AST edits. For affected declarations,
FriendDecl.getCode()reads the original source range instead of generating the declaration from the current AST. Changes inside that declaration are consequently not reflected in emitted code. This path also rejects raw-string literals and escaped-newline source. Unchanged parse → generate → reparse byte identity does not establish transformation correctness, so the earlier fidelity results must not be read as proof that these declarations are safely editable. Replace this workaround with structured owner/declaration metadata and printing from the current AST, and add tests that modify declarations and verify generated/reparsed output. Until this is implemented, explicitly reject the affected constructs rather than successfully emitting stale source. Do not carry the workaround into the Protobuf production port.These are correctness and support-detection gaps, not accepted compatibility exceptions or performance tradeoffs. They are distinct from the existing XStream warning and unresolved performance gate.
The eager reader now caches store definitions per Clava class with
ConcurrentHashMap.computeIfAbsent, avoiding the shared synchronized metadata lookup on every node. specs-java-libs #37 also removes the duplicate GC request and uses the Protobuf immutable ordered metadata construction while retaining the stronger FlatBuffers concurrency regression.Compatibility warning: the immutable metadata map in specs-java-libs #37 breaks existing XStream XML persistence. XStream removal is being handled in a separate PR. Until it lands, 17 legacy XML tests fail in that dependency; its jOptions check is not green.
Latest validation at Clava
09542632awith specs-java-libsae7194a7: full parser check passed 234 tests with four expected skips, coverage and generated-output drift verification. Three release-resource tests skip under the preserved local dumper selector, and one cache-hit test skips underCCACHE_DISABLE=true; the separate enabled-cache run passed both GeneratedParseRootTest tests without skips. SpecsUtils check and coverage passed 6,654 tests with two skips. jOptions passed 1,725 tests and failed the 17 XML tests described above. The parser used an isolated temporary CUDA cache after the shared cache failed validation. The matched optimization benchmark completed 27 accepted serial observations at these exact source revisions. Production Java test-body medians were 38.638 s previous FlatBuffers, 34.010 s updated FlatBuffers and 31.926 s Protobuf, a 12.0% before/after improvement. App medians with diagnostics disabled were 27.280/27.122/25.284 s for Java and 9.933/9.847/9.507 s for JS in previous/updated/Protobuf order. App improvements were under 1%; updated FlatBuffers remained 7.3% and 3.6% slower than Protobuf. Before/after captured workloads match in every round, all 170 JS App calls match Protobuf, and the 207 matching Java calls retain a 7.5% penalty. These measurements do not isolate each port or remeasure memory.Earlier RC3 validation used the authorized v18.1.8_5-rc3 prerelease. The validation report and evidence identify consumer
f03eb45d8, producer158080298, dependency revisions and artifact hashes. Full parser checks passed 237 tests with one expected disabled-cache skip; the enabled-cache run passed both tests. The full Linux JS suite passed 162 tests with two pending. The published producer passed all 1,062 native corpus streams. All 778 syntax-valid eager corpus outputs passed strict regeneration, with no Text-valid/eager-invalid paired case. The raw 125 Text/eager differences remain recorded, including 26 outputs invalid in both printers; they are not waived.Final measurements found a cost as well as retained-heap savings. NAS median peak RSS was 562 MiB eager versus 515 MiB Text and 587 MiB Protobuf, while retained heap was 14.9/37.8/15.6 MiB. Small C++ eager peak/retained memory exceeded Text. All 120 eager cleanup cycles passed. The matched Java median was 38.016 s eager versus 35.635 s Text and 31.986 s Protobuf; matched JS snapshots were 44.319/42.813/39.253 s. This does not claim a speed or universal memory advantage. These frozen measurements predate the current optimization ports. The GC diagnostic reduced the Java eager-versus-Protobuf gap from 18.9% to 6.8% when explicit GC was disabled equally; it does not isolate the reader cost or establish a current speed advantage. The speed adoption gate remains unresolved.
GPT-6.1-Sol medium reviewed the integrated implementation and final fixes; the findings from that review were resolved. That review did not establish the intended unsupported-node and AST-edit behavior described in the warning above. Field/node, regeneration, compatible-release and CI drift procedures are documented.
The follow-up CUDA test gate uses the manifest's host capability API. All three CUDA tests still pass on Linux and the TypeScript build passes; unsupported macOS hosts skip the CUDA group. Latest remote CI must be checked separately. SonarCloud findings remain open on this PR and native #56; overall CI is not claimed green.
Stacked on #258 and depends on specs-java-libs #37 and clang-dumper #56. Only testing RC publication is authorized; no master merge or stable publication is included.
Whole-command medians were also recorded for the common historical workloads, not unfiltered current-branch suites. With diagnostics disabled, Java was 33.819 s updated FlatBuffers versus 32.424 s Protobuf; JS was 34.827 versus 33.170 s. The remaining reversal survives the GC control. The earlier pilot did win App timing and Java command time; its JS whole-command ranking was narrow and Protobuf won the warm-cache median. Production additions have not been profiled individually.
The fresh comparison report now replaces historical Text and Protobuf controls with current builds and reruns FlatBuffers alongside them. It completed 144 accepted serial measurements on 7 October: four rotated rounds of three implementations, both common suite workloads and direct/cold/warm cache states, separately for cumulative App and uninstrumented whole-command timing. Protobuf uses the remotely verified branch heads Clava
ff5e58afa, specs-java-libs19c8e3e4and producerab9d0238. Text uses consumerbae332499, dependencyf49ca2c4and producerc549b4b2. All runtime JARs remained unchanged during measurement. Only Before-cache remains a clearly dated 2 October reference, excluded from paired statistics.FlatBuffers remained slower than both fresh controls in every suite/cache/metric cell. Median paired FlatBuffers-versus-Protobuf bypass differences were +4.01% JS and +7.83% Java for App creation, and +5.59% for each whole-command workload. Warm gaps were +10.74%/+16.44% for JS/Java App and +6.63%/+10.18% for JS/Java command time. All 170 JS App inputs match both controls; Java matches 207 of 216, with the matching-call diagnostic retaining a +7.91% bypass penalty versus Protobuf. Four observations are descriptive and do not establish significance of small differences. The committed report, evidence and SVGs retain exact revisions, all source/content/option differences, timer boundaries, original test-file orders, cache counters and renderer functions. Java uses 512 MiB; JS uses its original default heap observed at about 7.55 GiB. Diagnostics are disabled consistently in isolated benchmark builds. These compare branch implementations, not an isolated format switch; memory was not remeasured. The speed gate remains unresolved.