diff --git a/plans/2026-07-15_194452-collection-rebind-delayed-move-lowering.md b/plans/2026-07-15_194452-collection-rebind-delayed-move-lowering.md new file mode 100644 index 00000000..3cab470b --- /dev/null +++ b/plans/2026-07-15_194452-collection-rebind-delayed-move-lowering.md @@ -0,0 +1,329 @@ +# Collection Rebind Delayed-Move Lowering Implementation Plan + +> **For Hermes:** Use `subagent-driven-development` to implement this plan task-by-task, with `test-driven-development` for each behavior change. + +**Goal:** Lower same-local collection rebinds as an existing-bytecode ownership transfer so `set` / `array_push` receives the unique collection without introducing opcodes, runtime pattern matching, or deep COW copies. + +**Architecture:** Keep the target local readable while evaluating the key and RHS. Immediately before the existing builtin `Call`, emit the existing move-clear sequence `ldc null; stloc target`; the collection handle already on the stack then becomes the owned call argument. The existing post-call `stloc target` writes the result back. Alias legality remains owned by the current lifetime pass. + +**Tech Stack:** Rust, RustScript compiler IR/codegen, existing `Ldloc`/`Ldc`/`Stloc`/`Call`, Arc COW collections, interpreter/JIT/AOT/no-std compatibility tests. + +--- + +## Root cause + +The current pipeline loses move intent at lowering: + +1. `parse_index_assign_with_terminator` builds `set(Expr::Var(target), key, value)` and wraps it in `Stmt::Assign { index: target, ... }`. +2. The lifetime pass recognizes `Set` / `ArrayPush` collection mutation and calls `require_collection_mutation_permitted`, so aliases are already rejected. +3. `rewrite_local_source_move_on_rebind` only rewrites a whole RHS shaped as `Expr::Var(source)` and explicitly skips `source == target`; it never rewrites the nested call’s first argument. +4. Codegen compiles that first `Expr::Var(target)` through `emit_copy_ldloc`. +5. The target local remains populated at `Arc::make_mut`, so the stack handle and local handle force a deep COW detach despite lifetime analysis having proved the mutation legal. + +Correct existing-bytecode lowering: + +```text +ldloc target # temporary Arc-handle copy; target remains readable + + +ldc null +stloc target # delayed move: stack handle now owns the collection +call set/3 +stloc target # rebind result +``` + +For `ArrayPush`, omit the key. No collection contents are copied. Delaying the clear until after argument evaluation preserves `a[i] = a[j]` and side-effecting key/RHS behavior. + +## Constraints + +- Do not modify `OpCode`, opcode values, assembler/disassembler syntax, VMBC, AOT IR, native bridge op tables, or no-std opcode tables. +- Do not add runtime call-pattern recognition or Arc-pointer matching. +- Do not add hidden locals or stack-reordering instructions. +- Do not alter parser evaluation order or lifetime alias policy. +- Activate only when local move semantics are enabled, the assignment destination equals the collection source, and the call is built-in `Set` or `ArrayPush` with exact arity. +- Keep compatibility-language/plugin compilation on the existing generic path when local move semantics are disabled. +- The transient `Null` is a runtime ownership transfer; do not change compiler `type_state` before the final rebind. + +--- + +### Task 1: Add a RED bytecode-shape regression test + +**Objective:** Prove current codegen leaves the target local populated at the collection builtin call. + +**Files:** +- Modify: `tests/compiler/compiler_common_tests.rs` + +**Step 1: Extend the test decoder to expose `Call` operands** + +Update `DecodedInstr` / `decode_instructions` to record the existing call index and argc without changing production code. + +**Step 2: Add `same_local_collection_set_clears_target_immediately_before_call`** + +Compile: + +```rust +let mut a = [10, 20]; +a[0] = a[1]; +a[0]; +``` + +Resolve `a` through debug info and find `BuiltinFunction::Set.call_index()`. Assert the instruction window ending at the assignment call is: + +```text +Ldc +Stloc +Call +Stloc +``` + +Also assert an `Ldloc ` occurs before key/RHS evaluation, so the move remains delayed rather than clearing `a` before `a[1]` is read. + +**Step 3: Run the RED test** + +```bash +cargo test --locked --test compiler_tests \ + same_local_collection_set_clears_target_immediately_before_call -- --nocapture +``` + +Expected RED: current output contains `Call(Set)` followed by `Stloc(a)` but lacks the immediate `Ldc Null; Stloc(a)` pair before the call. + +**Step 4: Commit the RED test only after confirming the expected failure** + +Do not modify production code in this step. + +--- + +### Task 2: Implement delayed-move lowering in codegen + +**Objective:** Transfer the target local’s ownership immediately before the existing builtin call. + +**Files:** +- Modify: `src/compiler/codegen.rs` +- Test: `tests/compiler/compiler_common_tests.rs` + +**Step 1: Factor direct-call emission from argument compilation** + +Split the current `compile_direct_call` responsibilities: + +```rust +fn compile_direct_call(&mut self, index: u16, args: &[Expr]) -> Result<(), CompileError> { + for arg in args { + self.compile_scalar_expr(arg)?; + } + self.emit_direct_call(index, args) +} + +fn emit_direct_call(&mut self, index: u16, args: &[Expr]) -> Result<(), CompileError> { + let argc = u8::try_from(args.len()).map_err(|_| CompileError::CallArityOverflow)?; + if let Some(builtin) = BuiltinFunction::from_call_index(index) { + debug_assert!(builtin.accepts_arity(argc)); + self.record_builtin_call_operand_types(args); + self.assembler.call(index, argc); + return Ok(()); + } + let remapped_index = self.call_index_remap.get(&index).copied().unwrap_or(index); + self.assembler.call(remapped_index, argc); + Ok(()) +} +``` + +Keep all existing call-index remapping and operand-type recording behavior. + +**Step 2: Add exact same-local collection matching** + +Add a codegen helper: + +```rust +fn try_compile_same_local_collection_rebind( + &mut self, + target: LocalSlot, + expr: &Expr, +) -> Result +``` + +Return `false` unless all conditions hold: + +- `self.enable_local_move_semantics`; +- `expr` is `Expr::Call(index, _, args)`; +- builtin is `Set` with three args or `ArrayPush` with two args; +- `args[0]` is exactly `Expr::Var(target)`. + +Matching exact parser IR is intentional. Do not broaden to borrow wrappers or unrelated host calls. + +**Step 3: Emit delayed ownership transfer** + +For a match: + +```rust +for arg in args { + self.compile_scalar_expr(arg)?; +} +self.assembler.push_const(Value::Null); +self.emit_stloc(target)?; +self.emit_direct_call(*index, args)?; +Ok(true) +``` + +This order keeps `target` available throughout key/RHS evaluation. Do not call `emit_move_ldloc`, since it clears immediately after loading the first argument. + +**Step 4: Integrate before the generic RHS path** + +In `assign_expr_to_slot`, replace the unconditional expression compilation with: + +```rust +if !self.try_compile_same_local_collection_rebind(slot, expr)? { + self.compile_scalar_expr(expr)?; +} +self.emit_stloc(slot)?; +``` + +Keep all callable/type inference and final `type_state` updates unchanged. The existing final `emit_stloc(slot)` performs the result rebind. + +**Step 5: Run GREEN tests** + +```bash +cargo test --locked --test compiler_tests \ + same_local_collection_set_clears_target_immediately_before_call -- --nocapture +cargo test --locked --test compiler_tests rustscript_local_move -- --nocapture +``` + +Expected: delayed-clear shape passes; existing local-move tests remain green. + +**Step 6: Commit** + +```bash +git add src/compiler/codegen.rs tests/compiler/compiler_common_tests.rs +git commit -m "perf: lower collection rebinds as delayed moves" +``` + +--- + +### Task 3: Cover self-reference, aliases, array push, and compatibility mode + +**Objective:** Lock down language semantics and fallback boundaries. + +**Files:** +- Modify: `tests/compiler/compiler_common_tests.rs` +- Modify: `tests/compiler/compiler_rustscript_tests.rs` +- Modify only if a failing test exposes an implementation defect: `src/compiler/codegen.rs` + +**Step 1: Add self-reference behavior cases** + +Add runtime cases for: + +```rust +let mut a = [10, 20]; +a[0] = a[1]; +a[0]; +``` + +and a key/RHS expression that records evaluation order through ordinary RustScript functions/counters. Assert key evaluates before RHS and both can read `a` before the delayed move. + +**Step 2: Add map and append cases** + +Cover: + +- map field overwrite and null deletion; +- array `index == len` append through `Set`; +- explicit same-local `a = array_push(a, value)` if accepted by current frontend. + +Assert each same-local call has `Ldc Null; Stloc target` immediately before `Call`, followed by the normal result `Stloc target`. + +**Step 3: Preserve alias rejection** + +Retain and narrow-run the existing cases where `let b = a; a[0] = ...` or mutable-borrow aliases cause compile errors. The delayed move relies on the lifetime pass’s existing `require_collection_mutation_permitted` proof. + +```bash +cargo test --locked --test compiler_tests collection_alias -- --nocapture +``` + +**Step 4: Verify fallback boundaries** + +Add tests proving no delayed clear is emitted when: + +- move semantics are disabled by a compatibility frontend; +- the assignment destination differs from the collection source; +- the call is not built-in `Set` / `ArrayPush`; +- arity or IR shape differs. + +**Step 5: Run targeted suite and commit** + +```bash +cargo test --locked --test compiler_tests collection -- --nocapture +cargo test --locked --test compiler_tests indexed_assignment -- --nocapture +``` + +```bash +git add tests/compiler/compiler_common_tests.rs \ + tests/compiler/compiler_rustscript_tests.rs src/compiler/codegen.rs +git commit -m "test: cover delayed collection move semantics" +``` + +--- + +### Task 4: Verify engine compatibility and absence of deep COW lowering + +**Objective:** Confirm the compiler-only change works on every execution path with unchanged bytecode ABI. + +**Files:** +- Modify: `tests/vm/functional_parity_tests.rs` +- Modify: `examples/mini_bench.rs` +- Do not modify: VM opcode/wire/runtime files + +**Step 1: Add interpreter/JIT parity** + +Run repeated same-local array/map mutation with JIT disabled and enabled. Compare final stack and locals. + +```bash +cargo test --locked --test vm_tests \ + interpreter_and_jit_match_for_delayed_collection_move -- --nocapture +``` + +**Step 2: Add a focused release workload** + +Extend `examples/mini_bench.rs` with a collection-rebind loop using ordinary RustScript indexed assignment. Keep timing assertions out of CI. + +Compare `fe93300` and the implementation commit with identical release settings and at least seven trials. Report medians separately for interpreter and JIT. + +**Step 3: Verify unchanged ABI/runtime scope** + +```bash +git diff --exit-code fe93300 -- \ + src/bytecode.rs src/assembler.rs src/vmbc.rs \ + src/vm pd-vm-nostd/src/program.rs pd-vm-nostd/src/vm.rs +``` + +Expected: no production diff in opcode, VMBC, interpreter, JIT/native bridge, or no-std runtime files. + +**Step 4: Run full verification** + +```bash +cargo fmt --all -- --check +cargo test --locked --lib +cargo test --locked --test compiler_tests +cargo test --locked --test vm_tests +cargo test --locked --test jit_tests +cargo test --locked -p pd-vm-nostd +cargo test --locked --all-targets +cargo clippy --locked --all-targets --all-features -- -D warnings +``` + +**Step 5: Commit benchmark/parity tests** + +```bash +git add tests/vm/functional_parity_tests.rs examples/mini_bench.rs +git commit -m "bench: measure delayed collection moves" +``` + +--- + +## Acceptance criteria + +- Same-local `Set` / `ArrayPush` uses only existing opcodes. +- Target remains readable during key/RHS evaluation. +- `Ldc Null; Stloc target` occurs immediately before the builtin call. +- At `Arc::make_mut`, the target local no longer retains the stack collection handle; alias-free source therefore avoids deep COW detachment. +- The ordinary post-call `Stloc target` restores the updated collection. +- Existing alias rejection remains the ownership proof. +- Compatibility frontends with move semantics disabled retain their current lowering. +- Parser, lifetime model, runtime dispatch, opcode tables, VMBC, AOT/native lowering, and no-std runtime remain unchanged. diff --git a/plans/2026-07-20_jit-simplification-review.md b/plans/2026-07-20_jit-simplification-review.md new file mode 100644 index 00000000..24831ff4 --- /dev/null +++ b/plans/2026-07-20_jit-simplification-review.md @@ -0,0 +1,419 @@ +# RustScript Trace JIT 简化实施计划（修订版） + +> **For Hermes:** 本计划按独立阶段实施；每阶段单独验证、评审和提交，不跨阶段混合改动。 + +> **性质：** 架构简化计划。本次更新仅修改本计划文件，不修改源码。 + +**目标：** 降低 `pd-vm` trace JIT 的维护复杂度，使新增 builtin 特化通常只需修改 1–2 个权威入口，同时保持 interpreter、JIT、AOT、wasm/no-std 语义一致。 + +**架构原则：** 先移除孤立源码，再用声明式元数据生成机械接线；保留 typed IR、所有权验证和专用 lowering。deopt 与机器码链接属于后续架构决策，必须先建立状态模型和实测依据。 + +**技术栈：** Rust 2024、Cranelift 0.129、trace SSA、AOT SSA、Cargo feature matrix。 + +**仓库基线：** `/home/wow/rustscript/rustscript`，`master @ fd3ddc9f747eae821578cbc74447d174b9fdbbbb`。 + +--- + +## 一、复核后的代码规模 + +### 1.1 递归目录总量 + +| 范围 | 行数 | 备注 | +|---|---:|---| +| `src/vm/jit/**/*.rs` | 19,509 | 包含 `jit/native/mod.rs` 与 `jit/mod.rs` | +| `src/vm/native/*.rs` | 7,659 | 包含孤立的 `native/inline.rs` | +| **递归合计** | **27,168** | 原总数正确 | +| `src/vm/native/inline.rs` | 3,613 | 当前未进入模块树 | +| **当前活跃 JIT/native 源码** | **23,555** | 递归合计减去孤立文件 | +| `src/vm/aot/**/*.rs` | 8,835 | 独立 AOT 后端 | +| JIT 测试 | 9,677 | 含测试入口文件 | + +原表漏列了以下 610 行，但总计已包含： + +| 文件 | 行数 | +|---|---:| +| `src/vm/jit/native/mod.rs` | 446 | +| `src/vm/jit/mod.rs` | 16 | +| `src/vm/native/mod.rs` | 76 | +| `src/vm/native/offsets.rs` | 72 | + +### 1.2 结构计数 + +| 指标 | 当前值 | +|---|---:| +| `SsaInstKind` | 77 个变体 | +| `SpecializedBuiltinKind` | 36 个变体 | +| `NativeInlineStep` | 46 个变体 | +| `bridge.rs` `extern "C"` helper | 50 个 | +| `lower.rs` 中 `SsaInstKind::` 引用 | 209 处 | +| `lower.rs` 函数 | 112 个 | + +### 1.3 LuaJIT 对照的使用边界 + +LuaJIT 单后端 JIT 核心约 15k 行，可作为复杂度参照，但不作为 RustScript 的硬目标： + +- LuaJIT 也维护 interpreter、JIT、多 CPU 后端、C function 与 int/num IR； +- RustScript 额外维护 AOT、wasm/no-std、host pending/yield、fuel/epoch、所有权与多种 Value 表示； +- LuaJIT 可直接修补自己生成的机器码；Cranelift finalized code 没有同等公开链接接口； +- 行数比较只能说明 RustScript 有明显机械重复，不能单独证明某个抽象应删除。 + +**修订结论：** 以“新增 builtin 的权威修改点数量”和“重复语义数量”为主要指标；LOC 仅作趋势数据。 + +--- + +## 二、复核后的复杂度判断 + +### 2.1 builtin 特化存在结构性重复 + +一个 builtin 可能涉及： + +1. `src/vm/jit/ir.rs`：typed IR、inputs、render； +2. `src/vm/jit/recorder.rs`：选择、分析、发射； +3. `src/vm/native/bridge.rs`：fallible 或 owned helper； +4. `src/vm/native/codegen.rs`：Cranelift signature； +5. `src/vm/native/mod.rs`：地址导出； +6. `src/vm/jit/native/lower.rs`：临时槽、所有权、failure exit、helper 调用。 + +其中选择、arity、输入表示、输出表示、helper ABI 等信息适合声明式生成；所有权转移、别名处理、failure exit 和特殊控制流仍需 typed 逻辑。 + +### 2.2 活跃 native 机制应按“两套 + 一份孤立源码”描述 + +| 部分 | 状态 | +|---|---| +| trace SSA lowering | 活跃 | +| AOT SSA | 活跃且独立 | +| `src/vm/native/inline.rs` | 未被 `mod` 声明，全仓库没有外部消费者 | + +因此 `native/inline.rs` 当前不是第三套运行机制，可以优先删除；无需再猜测 wasm/no-std 是否复用。 + +### 2.3 deopt 的复杂度确实高，但各结构语义方向不同 + +- `SsaExit`：父 trace 离开点的恢复描述； +- `SideTraceImport`：子 trace 入口的值映射和表示适配； +- `VirtualFrameSnapshot`：尚未存在的 callee frame，要求完整 materialization； +- `dirty_locals`：已存在物理 frame 的稀疏覆盖集合； +- region dirty propagation：融合图中的跨边状态传播。 + +它们可以共享底层 slot schema，但不能直接合并为同一个列表后删除其余语义。 + +### 2.4 tail wrapper 可以共享构建逻辑，运行时数量未必能减到两个 + +当前 wrapper 涉及普通、inherited、owned、dispatcher 等不同 ABI 与清理契约。第一目标应是参数化生成与消除重复源码；是否能减少运行时 thunk 数量，需要 ABI 矩阵和性能数据证明。 + +### 2.5 诊断 API 已形成公共契约 + +`jit_native_*`、`jit_snapshot()`、`dump_jit_info()` 被测试、example 和外部调用使用。默认关闭 feature 会移除方法或改变输出，属于兼容性变更，不能归类为无风险内部整理。 + +--- + +## 三、修订后的执行顺序 + +## 阶段 0：删除孤立的 `native/inline.rs` + +**目标：** 先移除已确认没有消费者的 3,613 行源码。 + +**文件：** + +- Delete: `src/vm/native/inline.rs` +- Verify: `src/vm/native/mod.rs` +- Verify: `src/vm/jit/native/mod.rs` + +**步骤：** + +1. 在删除前再次搜索以下符号，确认只有文件内部定义和引用： + - `NativeInlineStep` + - `InlineEmitCtx` + - `emit_native_inline_step` +2. 确认 `src/vm/native/mod.rs` 没有 `mod inline;`。 +3. 删除文件。 +4. 运行格式、workspace tests、strict Clippy、文档和 AArch64 check/Clippy。 +5. 分别检查默认 feature、`--no-default-features`、wasm/no-std workspace 成员。 +6. 独立评审删除范围，确认无构建脚本或源码生成器按路径读取该文件。 + +**验收：** + +- 所有门禁通过； +- 活跃行为和生成产物不变； +- JIT/native 递归总量下降 3,613 行。 + +**风险：** 低。主要风险是非 Rust 模块消费者按文件路径读取源码，因此必须搜索 build script、脚本和文档引用。 + +--- + +## 阶段 1：声明式 builtin 元数据试点 + +**目标：** 先证明一个元数据模型能覆盖三类差异明显的操作，再推广到全部 builtin。 + +**试点操作：** + +| 操作 | 代表语义 | +|---|---| +| `StringLen` | 纯读取、标量结果 | +| `RegexMatch` | fallible helper、failure exit、bridge error | +| `ArraySet` | owned mutation、别名、源值保留、failure exit | + +### 1.1 设计权威元数据 + +**候选文件：** + +- Create: `src/vm/jit/builtin_spec.rs` +- Modify: `src/vm/jit/mod.rs` +- Modify: `src/vm/jit/recorder.rs` +- Modify: `src/vm/jit/ir.rs` +- Modify: `src/vm/jit/native/lower.rs` +- Modify: `src/vm/native/codegen.rs` +- Modify: `src/vm/native/mod.rs` + +元数据至少表达： + +- builtin identity 与 arity； +- 输入数量、runtime representation 和静态类型要求； +- 输出 representation、known type、是否拥有临时槽； +- effect：pure / fallible / owned mutation / iterator state； +- failure exit 要求； +- bridge ABI class； +- mutation 输入中哪些值可能与容器别名； +- 成功和失败时的所有权契约。 + +### 1.2 采用生成 typed glue 的宏，不采用运行时字符串分派 + +建议形态： + +- 宏或 const spec 生成 builtin 选择和基础分析； +- 生成 signature/address registry； +- 继续保留 typed `SsaInstKind`； +- 特殊 lowering 仍由明确函数实现； +- helper 地址使用函数项或强类型 ID，不使用 `&'static str`； +- 不引入运行时统一 opcode dispatch。 + +### 1.3 试点验证 + +每个试点必须验证： + +- interpreter/JIT/AOT 结果一致； +- SSA dump 的语义结构一致； +- fuel/epoch 计数不因生成层增加 synthetic op； +- helper 失败恢复 virtual frame 并只重放一次； +- mutation helper 在源/目标/参数别名下 clone-before-transfer； +- bridge TLS 在 replay 前清理； +- native trace 数量、代码字节、编译时间和 benchmark 无显著退化。 + +### 1.4 试点决策门 + +只有满足以下条件才推广： + +- 新增同类 builtin 通常只改 spec + 专用语义实现； +- 所有权和 failure exit 信息不再散落到弱类型字符串； +- generated code 可读、错误位置可定位； +- compile time 和运行性能无显著退化； +- 三类试点均无需为自身增加例外旁路。 + +**初始收益目标：** 减少修改点和重复 match；不预设净删 4–6k 行。 + +--- + +## 阶段 2：推广 builtin 元数据并建立复杂度报告 + +**目标：** 将机械部分逐族迁移，避免一次重写 36 类特化。 + +**建议迁移顺序：** + +1. len/type/predicate 类； +2. string/bytes 纯转换； +3. regex 和其他 fallible helper； +4. array/map 查询； +5. array/map mutation； +6. iterator state。 + +每一族独立提交，并运行该族定向测试与完整 JIT tests。 + +**复杂度报告指标：** + +- 新 builtin 的权威修改点数量； +- 重复 selection/analyze/emit 分支数量； +- helper signature/address 重复数量； +- typed IR 变体数量； +- JIT/native LOC 趋势； +- native compile latency、代码字节和执行 benchmark。 + +LOC 只报告，不设置阻断阈值。 + +--- + +## 阶段 3：诊断实现整理，保持公共 API + +**目标：** 整理 `runtime.rs` 的诊断实现，不直接移除公开方法。 + +**候选方案：** + +- 将 snapshot/dump/metrics 组装移动到 `src/vm/jit/diagnostics.rs`； +- `Vm` 上的公开方法保留为薄转发； +- 只对高成本采集项目设置内部开关； +- 如果需要默认关闭公开诊断能力，先写兼容性提案并确定版本策略； +- feature 关闭时的返回语义必须明确，不能静默伪造零值。 + +**验收：** + +- 现有调用方无需改动； +- dump 和 snapshot 内容保持兼容； +- 默认热路径没有新增分支或分配； +- 不承诺固定删减行数。 + +--- + +## 阶段 4：tail wrapper ABI 分类与参数化生成 + +**目标：** 减少重复 Cranelift 构建代码，先不承诺运行时只剩两个 wrapper。 + +**步骤：** + +1. 建立 ABI/语义矩阵： + - 参数； + - 返回 status； + - inherited state； + - owned slot 清理； + - direct-link slot； + - dispatcher trace ID； + - W^X 与 keepalive 生命周期。 +2. 识别可共享的 block builder、status tail 和 owned cleanup 片段。 +3. 参数化源码生成，同时保持不同 ABI 的独立入口。 +4. 用反汇编、代码字节和 tail-link benchmark 比较前后结果。 +5. 只有 ABI 完全一致且没有额外间接跳转时，才合并运行时 wrapper。 + +**验收：** direct link、inherited handoff、owned clear、fuel/epoch、失效重发布测试全部通过。 + +--- + +## 阶段 5：统一 slot-state 模型的设计提案 + +**目标：** 先建立形式化状态模型，暂不改 deopt 实现。 + +提案必须分别定义： + +- 物理 frame 的稀疏恢复； +- virtual frame 的完整构造； +- parent exit 到 side entry 的表示转换； +- dirty-local 传播； +- borrowed source 的并行赋值语义； +- region fusion 中 SSA ID remap； +- clean/dirty local、operand stack 和 capture cell 的权威来源。 + +可以共享的底层结构是 `SlotId + Materialization + OwnershipMode`；是否合并 `SsaExit`、`SideTraceImport`、`VirtualFrameSnapshot`，由模型证明和原型结果决定。 + +**必测矩阵：** + +- dirty source 与 dirty destination； +- 两种索引顺序； +- swap/cycle； +- 重复 borrowed heap value； +- virtual frame clean local； +- nested virtual frame； +- region remap ID 碰撞； +- observable single-owner drop； +- helper/instruction failure replay。 + +**决策门：** 若模型只减少类型数量，却增加恢复分支或弱化所有权证明，则停止该阶段。 + +--- + +## 阶段 6：dynasm-rs 链接 spike（可选） + +**事实边界：** dynasm-rs 可以修补其自己管理的 executable buffer，不能直接修改 Cranelift 已 finalized 的代码。 + +spike 只评估三种明确方案： + +1. 保留现有 atomic link slot； +2. 使用 dynasm-rs 生成可修补 entry/exit stub，Cranelift trace 跳到 stub； +3. 用 dynasm-rs 或自有后端生成完整 trace。 + +方案 2 仍可能保留间接层；方案 3 接近新增完整后端。不得预设引入 dynasm-rs 后即可删除 region fusion 和全部 wrapper。 + +**评估指标：** + +- hot side-link 执行延迟； +- trace 编译延迟； +- 机器码字节； +- x86_64/AArch64 覆盖； +- W^X、并发 patch、指令缓存同步； +- unwind/debug/反汇编支持； +- 维护代码量和测试矩阵。 + +只有端到端收益显著且维护成本可接受，才进入正式设计。 + +--- + +## 四、阶段顺序与风险 + +| 顺序 | 阶段 | 预期收益 | 风险 | 是否立即实施 | +|---:|---|---|---|---| +| 0 | 删除孤立 `native/inline.rs` | 明确减少 3,613 行 | 低 | 是 | +| 1 | 三类 builtin 元数据试点 | 验证架构 | 中 | 设计评审后 | +| 2 | 按族推广 + 复杂度报告 | 降低新增特化成本 | 中 | 试点通过后 | +| 3 | 诊断实现整理 | 文件职责更清晰 | 低至中 | 保持 API 前提下 | +| 4 | wrapper 参数化 | 减少重复生成逻辑 | 中 | builtin 整理后 | +| 5 | slot-state/deopt 提案 | 降低推理成本 | 高 | 暂缓实现 | +| 6 | dynasm-rs spike | 评估直接 patch 路径 | 高 | 可选 | + +每阶段独立 PR，不合并提交。阶段 0–2 完成后重新统计活跃代码和修改点，再决定阶段 4–6 是否值得继续。 + +--- + +## 五、验收标准 + +### 核心目标 + +- 新增普通 builtin 特化通常只需修改 1–2 个权威入口； +- typed IR、所有权和 failure exit 契约保持显式； +- interpreter/JIT/AOT/wasm/no-std 语义一致； +- native 性能无显著退化； +- 复杂度减少来自删除重复语义，不来自压缩格式或弱化检查。 + +### 通用验证 + +- `cargo fmt --all -- --check` +- `cargo test --workspace --all-features` +- `cargo clippy --workspace --all-targets --all-features -- -D warnings` +- `cargo doc --workspace --all-features --no-deps` +- AArch64 workspace check 与 Clippy +- 默认 feature 与 `--no-default-features` 构建 +- JIT 定向 ownership/deopt/identity/schema/region/direct-link tests +- 现有性能基准 A/B + +### 评审要求 + +每阶段至少覆盖： + +- 语义一致性； +- identity 与失效； +- ownership/drop； +- virtual-frame deopt； +- helper failure replay； +- ABI 与链接生命周期； +- feature/API 兼容性。 + +--- + +## 六、已解决和开放问题 + +### 已解决 + +1. `native/inline.rs` 的消费者：当前没有消费者，文件未进入模块树。 +2. 递归总量 27,168 行：总数正确，原表漏列 610 行。 +3. 诊断 feature：默认关闭会影响公共 API，不能作为无风险阶段。 + +### 开放问题 + +1. 是否接受用宏/生成 typed glue 作为 builtin spec 的主要实现，而非运行时通用表？ +2. `SsaInstKind` 是否继续保持每个语义操作独立变体，还是为同 ABI 的 helper-backed 操作引入受约束的通用变体？ +3. 公开诊断 API 的长期兼容策略是什么？ +4. wrapper 参数化后，是否仍有足够收益继续减少运行时 thunk 数量？ +5. 阶段 1–2 完成后，deopt 复杂度是否仍是最高维护成本？ +6. dynasm-rs 的目标是 patchable stub，还是完整 trace 后端？ + +--- + +## 七、当前建议 + +先执行阶段 0，并单独评审；随后只做阶段 1 的三类试点。试点通过后再确定完整迁移范围。 + +暂不承诺将活跃源码降到 15–18k 行。当前最重要的成功指标是：**新增 builtin 特化的权威修改点降到 1–2 个，同时保留完整语义和所有权证明。** diff --git a/plans/2026-07-29_jit-slot-state-model-proposal.md b/plans/2026-07-29_jit-slot-state-model-proposal.md new file mode 100644 index 00000000..89d58f4b --- /dev/null +++ b/plans/2026-07-29_jit-slot-state-model-proposal.md @@ -0,0 +1,105 @@ +# Trace JIT slot-state 模型提案 + +> 范围：阶段 5 的设计产物。本文不修改 deopt、region fusion 或 side-link 的运行时实现。 + +## 目标与非目标 + +目标是建立一个能检验恢复路径正确性的状态模型，并明确哪些数据是权威来源。它只复用底层概念，暂不合并 `SsaExit`、`SideTraceImport` 与 `VirtualFrameSnapshot` 的类型。 + +非目标： + +- 不以减少类型数为目标； +- 不将 owned 与 borrowed 值归并为一个无约束字段； +- 不改变当前 direct-link、W^X、keepalive 或 atomic slot 发布协议； +- 不以新增恢复分支交换较少的数据结构。 + +## 基本模型 + +```text +SlotId = FrameId × SlotKind × SlotIndex +SlotKind = Operand | Local | Capture + +StateCell = { + slot: SlotId, + materialization: Materialization, + ownership: OwnershipMode, + dirty: bool, +} + +OwnershipMode = Borrowed | Owned | Scalar +Materialization = existing SsaMaterialization +``` + +`SlotId` 是恢复计划的地址，不是值本身。`Materialization` 保持现有 SSA 语义；任何未来的通用模型都必须保留其具体变体。`OwnershipMode` 用于校验转移是否合法，不能从 `ValueType` 推断。 + +## 权威状态来源 + +| 场景 | operand stack / locals 的权威来源 | capture cell 的权威来源 | 允许写入者 | +|---|---|---|---| +| 物理 frame 正在解释执行 | `Vm` 物理 stack 与 locals | `CallableEnvironment` | interpreter / 已完成的 native 恢复 | +| native trace 内 | SSA value + dirty 集合 | 已获准借用的 environment | native lowering 产生的显式 store | +| native exit | `SsaExit` materializations 与 dirty 集合 | 现有 capture cell | sparse restore helper | +| virtual frame | `VirtualFrameSnapshot` 的完整 materializations | 关联 callable environment | virtual-frame restore helper | +| parent exit → side entry | `SideTraceImport` 的经验证 mapping | parent/child 共享关系 | side-entry materialization | +| fused region | remap 后 SSA ID 与 link metadata | 原 callable environment | region lowering | + +物理 frame 的 clean slot 必须继续从 `Vm` 读取；dirty slot 必须由当前 SSA materialization 覆盖。virtual frame 没有可回退的物理 locals，故必须完整构造。 + +## 五类状态转换 + +### 1. 物理 frame 的稀疏恢复 + +输入为 `SsaExit`、物理 frame 标识与 dirty cells。对每个 dirty cell：先计算 source，后根据 ownership 选择 replace、move 或 scalar store。clean locals 与 clean operand stack 保持 `Vm` 中的当前内容。 + +前置条件：exit entry shape 与物理 frame 的 stack/local 边界一致。后置条件：每个 dirty `SlotId` 与 interpreter 在相同 bytecode IP 的可观察值相同。 + +### 2. virtual frame 的完整构造 + +输入为 `VirtualFrameSnapshot`。它必须为每个 operand 与 local 提供 materialization；不得把缺失条目解释为“沿用物理 frame”。构造顺序按 frame 链从 caller 到 callee；每个 callable environment 在恢复前完成身份校验。 + +### 3. parent exit 到 side entry + +`SideTraceImport` 是带验证的表示转换，不是 copy 的快捷路径。它只能导入 parent exit 的显式 inputs，按 child entry 参数顺序建立 mapping；frame、stack 深度、local 数、SSA repr 与 ownership 任一不匹配时拒绝 admission。 + +### 4. dirty-local 传播 + +dirty 集合是写入事实的权威来源。记录器在 store/可观察 mutation 后添加 cell；控制流 merge 取并集；region fusion 在 ID remap 后保持同一 slot 语义。exit lowering 只能恢复 dirty cells，不能依赖“值看似相同”省略所有权转移。 + +### 5. borrowed 的并行赋值 + +同一恢复批次先计算全部 source，再写 destination。borrowed heap source 在任何 destination overwrite 前完成保留或 clone；owned source 在成功转移后只清理一次。swap、cycle、重复 borrowed source 都不得依赖目标迭代顺序。 + +## 不变量 + +1. 一个可观察的 owned heap value 在任意时刻只有一个销毁责任方。 +2. 同一 borrowed source 可写入多个 destination，且每个 destination 的 retain/clone 语义独立。 +3. 任何失败 helper 在可观察写入之前返回；已完成的 writes 必须有明确的重放或回滚契约。 +4. `SlotId` 的 frame 域在 region fusion 后仍唯一；SSA ID 重映射不能改变 frame/slot 身份。 +5. side entry 不得绕过 parent exit 的 repr、ownership、frame 与 generation 校验。 +6. virtual frame 的每个 slot 都有 materialization；物理 frame 只允许 clean-slot 回退。 + +## 必测矩阵 + +| 情形 | 断言 | +|---|---| +| dirty source 与 dirty destination | source 与 destination 均按其 materialization 恢复 | +| 正向与反向索引 | 结果不依赖 destination 遍历顺序 | +| swap / cycle | 不丢失任一 heap 值，drop 次数正确 | +| 重复 borrowed heap value | 每个目标仍可独立存活 | +| virtual frame clean local | 完整 materialization，不读取 caller 的物理 local | +| nested virtual frame | frame 链、return IP 与 callable 身份一致 | +| region remap ID 碰撞 | 不混淆不同 frame 的 `SlotId` | +| observable single-owner drop | 销毁一次且仅一次 | +| helper/instruction failure replay | 恢复后与 interpreter 的重试行为一致 | +| side import repr/ownership mismatch | admission 拒绝且不发布 native link | + +## 原型与决策门 + +后续原型只可先引入内部 `SlotStatePlan` 视图，并由现有 `SsaExit`、`SideTraceImport`、`VirtualFrameSnapshot` 生成；三种现有类型保留为外层契约。原型需证明： + +- 上述测试矩阵全部通过； +- 恢复路径分支数不增加； +- helper failure、drop 与 side-entry admission 的诊断信息未减弱； +- JIT 热路径没有显著性能退化。 + +若模型仅减少类型名称或字段数，却增加恢复分支、模糊所有权或使任何不变量无法局部验证，则停止，不进入实现阶段。 diff --git a/src/compiler/codegen.rs b/src/compiler/codegen.rs index a5ab2240..5b9b43f3 100644 --- a/src/compiler/codegen.rs +++ b/src/compiler/codegen.rs @@ -677,14 +677,15 @@ impl Compiler { Expr::ClosureCall(closure, args) => { let prototype_id = self.emit_closure_callable(closure)?; self.record_closure_param_hints(prototype_id, args); - self.compile_callvalue_args(args)?; + self.compile_callvalue_args(args, ValueType::Unknown)?; } Expr::LocalCall(index, _, args) => { if let Some(prototype_id) = self.callable_prototype_bindings.get(index).copied() { self.record_closure_param_hints(prototype_id, args); } + let return_type = self.callable_local_return_type(*index); self.emit_copy_ldloc(*index)?; - self.compile_callvalue_args(args)?; + self.compile_callvalue_args(args, return_type)?; } Expr::Add(lhs, rhs) => { let lhs_ty = self.value_type_of_expr(lhs); @@ -1474,17 +1475,29 @@ impl Compiler { .function_slots .get(&index) .ok_or(CompileError::CallableUsedAsValue)?; + let return_type = self + .function_decls + .get(&index) + .map(|decl| decl.return_type) + .unwrap_or(ValueType::Unknown); self.emit_copy_ldloc(slot)?; - return self.compile_callvalue_args(args); + return self.compile_callvalue_args(args, return_type); } self.compile_direct_call(index, args) } - fn compile_callvalue_args(&mut self, args: &[Expr]) -> Result<(), CompileError> { + fn compile_callvalue_args( + &mut self, + args: &[Expr], + return_type: ValueType, + ) -> Result<(), CompileError> { for arg in args { self.compile_scalar_expr(arg)?; } let argc = u8::try_from(args.len()).map_err(|_| CompileError::CallArityOverflow)?; + if return_type != ValueType::Unknown { + self.record_operand_types(ValueType::Callable, return_type); + } self.assembler.call_value(argc); Ok(()) } @@ -1763,6 +1776,18 @@ impl Compiler { ValueType::from(self.infer_bound_type(expr)) } + fn callable_local_return_type(&self, slot: LocalSlot) -> ValueType { + match self + .type_map + .local_schemas + .get(slot as usize) + .and_then(|schema| schema.as_ref()) + { + Some(TypeSchema::Callable { result, .. }) => result.coarse_value_type(), + _ => ValueType::Unknown, + } + } + fn record_operand_types(&mut self, lhs: ValueType, rhs: ValueType) { if lhs == ValueType::Unknown || rhs == ValueType::Unknown { return; diff --git a/src/vm/aot/artifact.rs b/src/vm/aot/artifact.rs index b7f387f7..fbed9d48 100644 --- a/src/vm/aot/artifact.rs +++ b/src/vm/aot/artifact.rs @@ -13,7 +13,8 @@ use super::compile::CompiledProgram; const MAGIC: [u8; 4] = *b"PAT\0"; const VERSION: u16 = 6; const ABI_VERSION: u16 = 5; -const FLAGS: u16 = 0; +const FLAG_INTERPRETER_BOUNDARY_ONLY: u16 = 1; +const SUPPORTED_FLAGS: u16 = FLAG_INTERPRETER_BOUNDARY_ONLY; #[derive(Debug)] pub enum AotArtifactError { @@ -129,10 +130,12 @@ impl Vm { pub fn load_aot_artifact(&mut self, bytes: &[u8]) -> Result<(), AotArtifactError> { let expected_program_hash = self.ensure_program_cache_key(); let decoded = decode_artifact(bytes, Some(expected_program_hash))?; - self.aot_program = Some(CompiledProgram::from_code( - decoded.code, - decoded.resume_ips, - )?); + let compiled = if decoded.interpreter_boundary_only { + CompiledProgram::from_interpreter_boundary_code(decoded.code, decoded.resume_ips)? + } else { + CompiledProgram::from_code(decoded.code, decoded.resume_ips)? + }; + self.aot_program = Some(compiled); self.aot_exec_count = 0; Ok(()) } @@ -151,10 +154,12 @@ impl Vm { ) -> Result { let decoded = decode_artifact(bytes, None)?; let mut vm = Vm::new_with_jit_config(decoded.program, jit_config); - vm.aot_program = Some(CompiledProgram::from_code( - decoded.code, - decoded.resume_ips, - )?); + let compiled = if decoded.interpreter_boundary_only { + CompiledProgram::from_interpreter_boundary_code(decoded.code, decoded.resume_ips)? + } else { + CompiledProgram::from_code(decoded.code, decoded.resume_ips)? + }; + vm.aot_program = Some(compiled); vm.aot_exec_count = 0; Ok(vm) } @@ -172,6 +177,7 @@ struct DecodedArtifact { program: Program, code: Vec, resume_ips: Vec, + interpreter_boundary_only: bool, } fn encode_artifact( @@ -183,7 +189,12 @@ fn encode_artifact( out.extend_from_slice(&MAGIC); out.extend_from_slice(&VERSION.to_le_bytes()); out.extend_from_slice(&ABI_VERSION.to_le_bytes()); - out.extend_from_slice(&FLAGS.to_le_bytes()); + let flags = if aot_program.interpreter_boundary_only { + FLAG_INTERPRETER_BOUNDARY_ONLY + } else { + 0 + }; + out.extend_from_slice(&flags.to_le_bytes()); out.push(u8::try_from(std::mem::size_of::()).expect("pointer width fits u8")); write_string("arch", std::env::consts::ARCH, &mut out)?; @@ -244,9 +255,10 @@ fn decode_artifact( } let flags = cursor.read_u16()?; - if flags != FLAGS { + if flags & !SUPPORTED_FLAGS != 0 { return Err(AotArtifactError::UnsupportedFlags(flags)); } + let interpreter_boundary_only = flags & FLAG_INTERPRETER_BOUNDARY_ONLY != 0; let pointer_width = cursor.read_u8()?; validate_runtime_field( @@ -324,6 +336,7 @@ fn decode_artifact( program, code, resume_ips, + interpreter_boundary_only, }) } @@ -427,6 +440,41 @@ mod tests { use super::*; use crate::{BytecodeBuilder, Program, Value, ValueType, VmStatus}; + #[test] + fn aot_artifact_preserves_interpreter_boundary_mode() { + let mut bc = BytecodeBuilder::new(); + bc.ret(); + let mut vm = Vm::new(Program::new(Vec::new(), bc.finish())); + vm.compile_aot().expect("aot compile should succeed"); + vm.aot_program + .as_mut() + .expect("compiled program") + .interpreter_boundary_only = true; + + let encoded = vm + .encode_aot_artifact() + .expect("artifact encode should succeed"); + assert_eq!( + u16::from_le_bytes([encoded[8], encoded[9]]), + FLAG_INTERPRETER_BOUNDARY_ONLY + ); + let standalone = Vm::new_from_aot_artifact_with_jit_config( + &encoded, + JitConfig { + enabled: false, + ..JitConfig::default() + }, + ) + .expect("boundary artifact should load"); + assert!( + standalone + .aot_program + .as_ref() + .expect("loaded aot program") + .interpreter_boundary_only + ); + } + #[test] fn aot_artifact_decode_rejects_invalid_magic_and_trailing_bytes() { let mut bc = BytecodeBuilder::new(); diff --git a/src/vm/aot/compile.rs b/src/vm/aot/compile.rs index 34bf2a50..c8896093 100644 --- a/src/vm/aot/compile.rs +++ b/src/vm/aot/compile.rs @@ -15,16 +15,16 @@ use crate::vm::native::ExecutableBuffer; #[cfg(feature = "cranelift-jit")] use crate::vm::native::{ HeapIntrinsicAddrs, HeapIntrinsicRefs, NativeFrameState, OP_BUILTIN_CALL, OP_CALL, - STATUS_CONTINUE, STATUS_ERROR, alloc_buffer_signature, alloc_byte_buffer_entry_address, - alloc_value_buffer_entry_address, aot_call_boundary_interrupt_entry_address, - array_push_entry_address, box_heap_value_signature, clear_value_slot_entry_address, - clone_value_signature, clone_value_to_slot_entry_address, collection_get_signature, - collection_mutation_signature, collection_set_entry_address, copy_bytes_entry_address, - copy_bytes_signature, detect_native_stack_layout, enter_call_value_entry_address, - enter_call_value_signature, entry_signature, frame_state_entry_address, frame_state_signature, - free_buffer_signature, helper_entry_offset, helper_signature, - init_null_value_slot_entry_address, jump_with_status, leave_frame_entry_address, - leave_frame_signature, pack_shared_signature, resolve_offsets, + STATUS_CONTINUE, STATUS_ERROR, STATUS_TRACE_EXIT, alloc_buffer_signature, + alloc_byte_buffer_entry_address, alloc_value_buffer_entry_address, + aot_call_boundary_interrupt_entry_address, array_push_entry_address, box_heap_value_signature, + clear_value_slot_entry_address, clone_value_signature, clone_value_to_slot_entry_address, + collection_get_signature, collection_mutation_signature, collection_set_entry_address, + copy_bytes_entry_address, copy_bytes_signature, detect_native_stack_layout, + enter_call_value_entry_address, enter_call_value_signature, entry_signature, + frame_state_entry_address, frame_state_signature, free_buffer_signature, helper_entry_offset, + helper_signature, init_null_value_slot_entry_address, jump_with_status, + leave_frame_entry_address, leave_frame_signature, pack_shared_signature, resolve_offsets, restore_active_exit_state_entry_address, restore_exit_signature, restore_exit_state_entry_address, shared_array_from_buffer_entry_address, shared_bytes_from_buffer_entry_address, shared_string_from_buffer_entry_address, @@ -82,6 +82,7 @@ pub(crate) struct CompiledProgram { pub(crate) entry: NativeProgramEntry, pub(crate) code: Arc<[u8]>, pub(crate) resume_ips: Arc<[usize]>, + pub(crate) interpreter_boundary_only: bool, } struct ProgramKeepAlive { @@ -102,6 +103,21 @@ impl ProgramKeepAlive { impl CompiledProgram { pub(crate) fn from_code(code: Vec, resume_ips: Vec) -> VmResult { + Self::from_code_with_boundary_mode(code, resume_ips, false) + } + + pub(crate) fn from_interpreter_boundary_code( + code: Vec, + resume_ips: Vec, + ) -> VmResult { + Self::from_code_with_boundary_mode(code, resume_ips, true) + } + + fn from_code_with_boundary_mode( + code: Vec, + resume_ips: Vec, + interpreter_boundary_only: bool, + ) -> VmResult { let keepalive = ProgramKeepAlive::from_code(&code)?; let entry = unsafe { std::mem::transmute::<*const u8, NativeProgramEntry>(keepalive.entry()) }; @@ -110,6 +126,7 @@ impl CompiledProgram { entry, code: Arc::<[u8]>::from(code.into_boxed_slice()), resume_ips: Arc::<[usize]>::from(resume_ips.into_boxed_slice()), + interpreter_boundary_only, }) } @@ -151,6 +168,15 @@ impl From for AotCompileError { static CRANELIFT_AOT_ID: AtomicU64 = AtomicU64::new(1); #[cfg(feature = "cranelift-jit")] static CRANELIFT_AOT_ISA: OnceLock> = OnceLock::new(); +#[cfg(feature = "cranelift-jit")] +// regalloc2 stores virtual-register indices in 21 bits. Keep enough room for +// instruction results and checkpoint loaders after allocating block params. +const MAX_MONOLITHIC_AOT_BLOCK_PARAMS: usize = 1_000_000; + +#[cfg(feature = "cranelift-jit")] +fn exceeds_monolithic_aot_block_param_budget(total_block_params: usize) -> bool { + total_block_params > MAX_MONOLITHIC_AOT_BLOCK_PARAMS +} pub(crate) fn compile_program(program: &Program) -> VmResult { compile_program_inner(program).map_err(|err| VmError::JitNative(err.to_string())) @@ -162,6 +188,11 @@ fn compile_program_inner(program: &Program) -> Result(); if trace_enabled { let total_insts = ssa .blocks @@ -169,11 +200,6 @@ fn compile_program_inner(program: &Program) -> Result(); let external_checkpoints = ssa.checkpoints.iter().filter(|cp| cp.external).count(); - let total_block_params = ssa - .blocks - .iter() - .map(|block| block.params.len()) - .sum::(); let max_block_params = ssa .blocks .iter() @@ -206,7 +232,88 @@ fn compile_program_inner(program: &Program) -> Result Ok(compiled), + Err(AotCompileError::Codegen(message)) + if message.contains("Code for function is too large") => + { + compile_interpreter_boundary_program() + } + Err(error) => Err(error), + } +} + +#[cfg(all(test, feature = "cranelift-jit"))] +mod boundary_budget_tests { + use super::{MAX_MONOLITHIC_AOT_BLOCK_PARAMS, exceeds_monolithic_aot_block_param_budget}; + + #[test] + fn oversized_aot_ssa_crosses_the_monolithic_block_param_budget() { + assert!(!exceeds_monolithic_aot_block_param_budget( + MAX_MONOLITHIC_AOT_BLOCK_PARAMS + )); + assert!(exceeds_monolithic_aot_block_param_budget( + MAX_MONOLITHIC_AOT_BLOCK_PARAMS + 1 + )); + } +} + +#[cfg(feature = "cranelift-jit")] +fn compile_interpreter_boundary_program() -> Result { + let isa = native_isa()?; + let jit_builder = JITBuilder::with_isa(isa, cranelift_module::default_libcall_names()); + let mut module = JITModule::new(jit_builder); + let pointer_type = module.target_config().pointer_type(); + let call_conv = module.target_config().default_call_conv; + let mut ctx = module.make_context(); + ctx.func.signature = entry_signature(pointer_type, call_conv); + let id = CRANELIFT_AOT_ID.fetch_add(1, Ordering::Relaxed); + let name = format!("pd_vm_aot_interpreter_boundary_{id}"); + let func_id = module + .declare_function(&name, Linkage::Local, &ctx.func.signature) + .map_err(|err| { + AotCompileError::Codegen(format!("declare aot interpreter boundary failed: {err}")) + })?; + { + let mut fb_ctx = FunctionBuilderContext::new(); + let mut b = FunctionBuilder::new(&mut ctx.func, &mut fb_ctx); + let entry = b.create_block(); + b.append_block_params_for_function_params(entry); + b.switch_to_block(entry); + let status = b.ins().iconst(types::I32, STATUS_TRACE_EXIT as i64); + b.ins().return_(&[status]); + b.seal_all_blocks(); + b.finalize(); + } + module.define_function(func_id, &mut ctx).map_err(|err| { + AotCompileError::Codegen(format!("define aot interpreter boundary failed: {err}")) + })?; + let code_len = ctx + .compiled_code() + .ok_or_else(|| { + AotCompileError::Codegen( + "aot interpreter boundary produced no machine code".to_string(), + ) + })? + .code_buffer() + .len(); + module.clear_context(&mut ctx); + module.finalize_definitions().map_err(|err| { + AotCompileError::Codegen(format!("finalize aot interpreter boundary failed: {err}")) + })?; + let entry = module.get_finalized_function(func_id); + let code = unsafe { std::slice::from_raw_parts(entry, code_len).to_vec() }; + CompiledProgram::from_interpreter_boundary_code(code, vec![0]) + .map_err(|err| AotCompileError::Codegen(err.to_string())) } #[cfg(not(feature = "cranelift-jit"))] diff --git a/src/vm/aot/runtime.rs b/src/vm/aot/runtime.rs index 8320e5ef..13029006 100644 --- a/src/vm/aot/runtime.rs +++ b/src/vm/aot/runtime.rs @@ -45,7 +45,14 @@ impl Vm { )); out.push_str(&format!(" aot executions: {}\n", self.aot_exec_count)); out.push_str(&format!(" code_bytes={}\n", program.code.len())); - out.push_str(" lowering=ssa\n"); + out.push_str(&format!( + " lowering={}\n", + if program.interpreter_boundary_only { + "interpreter-boundary" + } else { + "ssa" + } + )); out.push_str(&format!(" resume ips: {}\n", format_resume_ips(program))); out } diff --git a/src/vm/aot/ssa.rs b/src/vm/aot/ssa.rs index f81b921f..b7c0e9e9 100644 --- a/src/vm/aot/ssa.rs +++ b/src/vm/aot/ssa.rs @@ -1169,8 +1169,10 @@ impl<'a> Builder<'a> { let mut queue = VecDeque::from([self.lowered.entry_ip]); for region in &self.lowered.regions { if region.prototype_id.is_some() { - self.incoming_shapes - .insert(region.start_ip, entry_shape.clone()); + self.incoming_shapes.insert( + region.start_ip, + function_region_entry_shape(self.program, region.prototype_id, &entry_shape), + ); queue.push_back(region.start_ip); } } @@ -1319,6 +1321,30 @@ fn merge_value_repr(lhs: AotSsaValueRepr, rhs: AotSsaValueRepr) -> AotSsaValueRe } } +fn function_region_entry_shape( + program: &Program, + prototype_id: Option, + root_shape: &FrameShape, +) -> FrameShape { + let mut shape = root_shape.clone(); + let Some(prototype_id) = prototype_id else { + return shape; + }; + let Some(prototype) = program.callable_prototypes.get(prototype_id as usize) else { + return shape; + }; + let Some(crate::compiler::TypeSchema::Callable { params, .. }) = prototype.schema.as_ref() + else { + return shape; + }; + for (slot, schema) in prototype.parameter_slots.iter().zip(params.iter()) { + if let Some(repr) = shape.locals.get_mut(*slot as usize) { + *repr = value_type_repr(schema.coarse_value_type()); + } + } + shape +} + pub(crate) fn build_aot_ssa(program: &Program) -> Result { let ambiguous_calls = collect_ambiguous_direct_host_calls(program)?; if ambiguous_calls.is_empty() { @@ -1469,8 +1495,9 @@ impl<'a> Builder<'a> { for _ in 0..=usize::from(*argc) { resume_frame.pop(*call_ip, "callvalue")?; } + let return_repr = value_type_repr(operand_types_at(self.program, *call_ip).1); resume_frame.stack.push(FrameValue { - value: AotSsaValue::new(AotSsaValueId::new(0), AotSsaValueRepr::Tagged), + value: AotSsaValue::new(AotSsaValueId::new(0), return_repr), }); Ok(ProcessResult::CallValue { argc: *argc, @@ -2004,6 +2031,36 @@ fn apply_direct_instruction( }; Some(tagged_kind) } + (AotSsaValueRepr::Tagged, AotSsaValueRepr::I64) + if operand_types_at(program, ip) == (ValueType::Int, ValueType::Int) => + { + let lhs_int = emitter.emit( + ip, + AotSsaInstKind::TaggedToInt { + input: lhs.value.id, + }, + AotSsaValueRepr::I64, + ); + Some(AotSsaInstKind::IntCmpEq { + lhs: lhs_int.id, + rhs: rhs.value.id, + }) + } + (AotSsaValueRepr::I64, AotSsaValueRepr::Tagged) + if operand_types_at(program, ip) == (ValueType::Int, ValueType::Int) => + { + let rhs_int = emitter.emit( + ip, + AotSsaInstKind::TaggedToInt { + input: rhs.value.id, + }, + AotSsaValueRepr::I64, + ); + Some(AotSsaInstKind::IntCmpEq { + lhs: lhs.value.id, + rhs: rhs_int.id, + }) + } _ => None, }; if let Some(kind) = kind { diff --git a/src/vm/jit/builtin_spec.rs b/src/vm/jit/builtin_spec.rs new file mode 100644 index 00000000..8f23622c --- /dev/null +++ b/src/vm/jit/builtin_spec.rs @@ -0,0 +1,603 @@ +//! Declarative metadata for specialized builtin recording. +//! +//! Each `BuiltinSpec` captures the mechanical, per-builtin facts that the +//! recorder needs to select, analyze, and emit a specialized SSA +//! instruction. The goal is to reduce the six-layer touch-point tax +//! (selection → analysis → emit → bridge → codegen → lowering) to a +//! single authoritative spec plus dedicated semantic implementations. +//! +//! Scope: pilot (StringLen, RegexMatch, ArraySet) + family 1 +//! (len/type/predicate). Non-covered builtins continue to use their +//! existing hand-written paths. + +use super::ir::SsaValueRepr; +use crate::ValueType; + +/// How a builtin interacts with the VM heap and failure domain. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum BuiltinEffect { + /// Pure read-only operation; no heap mutation, no failure exit. + Pure, + /// Calls a fallible bridge helper; failure triggers a deopt exit. + FallibleHelper, + /// Owned mutation with clone-before-transfer semantics and failure exit. + OwnedMutation, +} + +/// Runtime representation requirement for one input operand. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum InputRepr { + /// Must be `SsaValueRepr::I64` (int). + Int, + /// Must be a heap pointer of the given container type. + HeapPtr(HeapInputKind), + /// Any tagged value (used for owned mutation values). + Tagged, + /// Any representation; used as-is. + Any, +} + +/// Heap container kinds relevant to builtin specialization. +#[allow(dead_code)] // Variants used by future builtin families. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum HeapInputKind { + String, + Bytes, + Array, + Map, +} + +impl HeapInputKind { + pub(crate) fn value_type(self) -> ValueType { + match self { + Self::String => ValueType::String, + Self::Bytes => ValueType::Bytes, + Self::Array => ValueType::Array, + Self::Map => ValueType::Map, + } + } +} + +/// Output type produced by a specialized builtin. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum OutputKind { + Int, + Bool, + Tagged(ValueType), + /// Tagged string with `ValueInfo::type_name()` (used by `TypeOf`). + TypeName, + /// Tagged value whose type is not statically known. + #[allow(dead_code)] // Used by future builtin families. + TaggedUnknown, +} + +impl OutputKind { + pub(crate) fn repr(self) -> SsaValueRepr { + match self { + Self::Int => SsaValueRepr::I64, + Self::Bool => SsaValueRepr::Bool, + Self::Tagged(_) | Self::TypeName | Self::TaggedUnknown => SsaValueRepr::Tagged, + } + } +} + +/// Declarative specification for one specialized builtin. +/// +/// The recorder reads this to drive generic analyze/emit paths. +/// Dedicated lowering implementations in `lower.rs` remain typed and +/// are *not* replaced by this table. +pub(crate) struct BuiltinSpec { + /// Human-readable name for diagnostics. + pub(crate) name: &'static str, + /// Number of arguments popped from the analysis frame (in reverse order). + pub(crate) arity: usize, + /// Input requirements, in pop order (last argument first). + pub(crate) inputs: &'static [InputRepr], + /// Output type. + pub(crate) output: OutputKind, + /// Effect classification. + pub(crate) effect: BuiltinEffect, + /// Whether the builtin requires a failure exit on helper error. + pub(crate) needs_failure_exit: bool, +} + +impl BuiltinSpec { + /// Whether this builtin is a pure read-only operation with no side effects. + pub(crate) fn is_pure(&self) -> bool { + matches!(self.effect, BuiltinEffect::Pure) + } +} + +/// `string.len()` — pure read, scalar result. +pub(crate) const STRING_LEN_SPEC: BuiltinSpec = BuiltinSpec { + name: "string_len", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::String)], + output: OutputKind::Int, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `re_match(pattern, text)` — fallible helper with bridge error. +pub(crate) const REGEX_MATCH_SPEC: BuiltinSpec = BuiltinSpec { + name: "regex_match", + arity: 2, + inputs: &[ + InputRepr::HeapPtr(HeapInputKind::String), // text (popped second) + InputRepr::HeapPtr(HeapInputKind::String), // pattern (popped first) + ], + output: OutputKind::Bool, + effect: BuiltinEffect::FallibleHelper, + needs_failure_exit: true, +}; + +/// `array.set(index, value)` — owned mutation, aliasing, failure exit. +/// +/// Note: the recorder additionally detects the append-pattern +/// (`index == array.len()`) and rewrites to `ArrayPush`. That +/// optimization is semantic, not mechanical, and stays in the +/// recorder's typed emit path. +pub(crate) const ARRAY_SET_SPEC: BuiltinSpec = BuiltinSpec { + name: "array_set", + arity: 3, + inputs: &[ + InputRepr::Any, // value (popped third) + InputRepr::Int, // index (popped second) + InputRepr::Tagged, // array (popped first, must be owned Tagged) + ], + output: OutputKind::Tagged(ValueType::Array), + effect: BuiltinEffect::OwnedMutation, + needs_failure_exit: true, +}; + +// ── Family 1: len / type / predicate ──────────────────────────────── + +/// `len(value)` — pure read, scalar result. +pub(crate) const VALUE_LEN_SPEC: BuiltinSpec = BuiltinSpec { + name: "value_len", + arity: 1, + inputs: &[InputRepr::Any], + output: OutputKind::Int, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `bytes.len()` — pure read, scalar result. +pub(crate) const BYTES_LEN_SPEC: BuiltinSpec = BuiltinSpec { + name: "bytes_len", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::Bytes)], + output: OutputKind::Int, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `array.len()` — pure read, scalar result. +pub(crate) const ARRAY_LEN_SPEC: BuiltinSpec = BuiltinSpec { + name: "array_len", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::Array)], + output: OutputKind::Int, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `map.len()` — pure read, scalar result. +pub(crate) const MAP_LEN_SPEC: BuiltinSpec = BuiltinSpec { + name: "map_len", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::Map)], + output: OutputKind::Int, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `type(value)` — pure read, tagged string result. +pub(crate) const TYPE_OF_SPEC: BuiltinSpec = BuiltinSpec { + name: "type_of", + arity: 1, + inputs: &[InputRepr::Any], + output: OutputKind::TypeName, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `string.contains(needle)` — pure read, bool result. +pub(crate) const STRING_CONTAINS_SPEC: BuiltinSpec = BuiltinSpec { + name: "string_contains", + arity: 2, + inputs: &[ + InputRepr::HeapPtr(HeapInputKind::String), // needle (popped second) + InputRepr::HeapPtr(HeapInputKind::String), // text (popped first) + ], + output: OutputKind::Bool, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `array.has(index)` — pure read, bool result. +pub(crate) const ARRAY_HAS_SPEC: BuiltinSpec = BuiltinSpec { + name: "array_has", + arity: 2, + inputs: &[ + InputRepr::Int, // index (popped second) + InputRepr::HeapPtr(HeapInputKind::Array), // array (popped first) + ], + output: OutputKind::Bool, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +// ── Family 2: string/bytes pure transformations ───────────────────── + +/// `string.slice(start, length)` — pure transformation. +pub(crate) const STRING_SLICE_SPEC: BuiltinSpec = BuiltinSpec { + name: "string_slice", + arity: 3, + inputs: &[ + InputRepr::Int, // length (popped third) + InputRepr::Int, // start (popped second) + InputRepr::HeapPtr(HeapInputKind::String), // text (popped first) + ], + output: OutputKind::Tagged(ValueType::String), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `bytes.slice(start, length)` — pure transformation. +pub(crate) const BYTES_SLICE_SPEC: BuiltinSpec = BuiltinSpec { + name: "bytes_slice", + arity: 3, + inputs: &[ + InputRepr::Int, // length (popped third) + InputRepr::Int, // start (popped second) + InputRepr::HeapPtr(HeapInputKind::Bytes), // bytes (popped first) + ], + output: OutputKind::Tagged(ValueType::Bytes), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `string.get(index)` — pure read, string result. +pub(crate) const STRING_GET_SPEC: BuiltinSpec = BuiltinSpec { + name: "string_get", + arity: 2, + inputs: &[ + InputRepr::Int, // index (popped second) + InputRepr::HeapPtr(HeapInputKind::String), // text (popped first) + ], + output: OutputKind::Tagged(ValueType::String), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `bytes.get(index)` — pure read, int result. +pub(crate) const BYTES_GET_SPEC: BuiltinSpec = BuiltinSpec { + name: "bytes_get", + arity: 2, + inputs: &[ + InputRepr::Int, // index (popped second) + InputRepr::HeapPtr(HeapInputKind::Bytes), // bytes (popped first) + ], + output: OutputKind::Int, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `bytes.has(index)` — pure read, bool result. +pub(crate) const BYTES_HAS_SPEC: BuiltinSpec = BuiltinSpec { + name: "bytes_has", + arity: 2, + inputs: &[ + InputRepr::Int, // index (popped second) + InputRepr::HeapPtr(HeapInputKind::Bytes), // bytes (popped first) + ], + output: OutputKind::Bool, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `string.replace_literal(needle, replacement)` — pure transformation. +pub(crate) const STRING_REPLACE_LITERAL_SPEC: BuiltinSpec = BuiltinSpec { + name: "string_replace_literal", + arity: 3, + inputs: &[ + InputRepr::HeapPtr(HeapInputKind::String), // replacement (popped third) + InputRepr::HeapPtr(HeapInputKind::String), // needle (popped second) + InputRepr::HeapPtr(HeapInputKind::String), // text (popped first) + ], + output: OutputKind::Tagged(ValueType::String), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `string.lower_ascii()` — pure transformation. +pub(crate) const STRING_LOWER_ASCII_SPEC: BuiltinSpec = BuiltinSpec { + name: "string_lower_ascii", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::String)], + output: OutputKind::Tagged(ValueType::String), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `string.split_literal(delimiter)` — pure transformation, array result. +pub(crate) const STRING_SPLIT_LITERAL_SPEC: BuiltinSpec = BuiltinSpec { + name: "string_split_literal", + arity: 2, + inputs: &[ + InputRepr::HeapPtr(HeapInputKind::String), // delimiter (popped second) + InputRepr::HeapPtr(HeapInputKind::String), // text (popped first) + ], + output: OutputKind::Tagged(ValueType::Array), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `bytes.from_array_u8(array)` — pure transformation. +pub(crate) const BYTES_FROM_ARRAY_U8_SPEC: BuiltinSpec = BuiltinSpec { + name: "bytes_from_array_u8", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::Array)], + output: OutputKind::Tagged(ValueType::Bytes), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `bytes.to_utf8_ascii()` — pure transformation. +pub(crate) const BYTES_TO_UTF8_ASCII_SPEC: BuiltinSpec = BuiltinSpec { + name: "bytes_to_utf8_ascii", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::Bytes)], + output: OutputKind::Tagged(ValueType::String), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `bytes.to_array_u8()` — pure transformation. +pub(crate) const BYTES_TO_ARRAY_U8_SPEC: BuiltinSpec = BuiltinSpec { + name: "bytes_to_array_u8", + arity: 1, + inputs: &[InputRepr::HeapPtr(HeapInputKind::Bytes)], + output: OutputKind::Tagged(ValueType::Array), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `to_string(value)` — pure transformation. +pub(crate) const TO_STRING_SPEC: BuiltinSpec = BuiltinSpec { + name: "to_string", + arity: 1, + inputs: &[InputRepr::Any], + output: OutputKind::Tagged(ValueType::String), + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +// ── Family 3: regex fallible ──────────────────────────────────────── + +/// `re_replace(pattern, text, replacement)` — fallible helper. +pub(crate) const REGEX_REPLACE_SPEC: BuiltinSpec = BuiltinSpec { + name: "regex_replace", + arity: 3, + inputs: &[ + InputRepr::HeapPtr(HeapInputKind::String), // replacement (popped third) + InputRepr::HeapPtr(HeapInputKind::String), // text (popped second) + InputRepr::HeapPtr(HeapInputKind::String), // pattern (popped first) + ], + output: OutputKind::Tagged(ValueType::String), + effect: BuiltinEffect::FallibleHelper, + needs_failure_exit: true, +}; + +// ── Family 4: array/map queries ───────────────────────────────────── + +/// `array.get(index)` — pure read, unknown tagged result. +pub(crate) const ARRAY_GET_SPEC: BuiltinSpec = BuiltinSpec { + name: "array_get", + arity: 2, + inputs: &[ + InputRepr::Int, // index (popped second) + InputRepr::HeapPtr(HeapInputKind::Array), // array (popped first) + ], + output: OutputKind::TaggedUnknown, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `map.get(key)` — pure read, unknown tagged result. +pub(crate) const MAP_GET_SPEC: BuiltinSpec = BuiltinSpec { + name: "map_get", + arity: 2, + inputs: &[ + InputRepr::Any, // key (popped second) + InputRepr::HeapPtr(HeapInputKind::Map), // map (popped first) + ], + output: OutputKind::TaggedUnknown, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `map.has(key)` — pure read, bool result. +pub(crate) const MAP_HAS_SPEC: BuiltinSpec = BuiltinSpec { + name: "map_has", + arity: 2, + inputs: &[ + InputRepr::Any, // key (popped second) + InputRepr::HeapPtr(HeapInputKind::Map), // map (popped first) + ], + output: OutputKind::Bool, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +// ── Family 5: array/map mutations ─────────────────────────────────── + +/// `array.push(value)` — owned mutation, returns the mutated array. +pub(crate) const ARRAY_PUSH_SPEC: BuiltinSpec = BuiltinSpec { + name: "array_push", + arity: 2, + inputs: &[ + InputRepr::Any, // value (popped second) + InputRepr::Tagged, // array (popped first, must be owned Tagged) + ], + output: OutputKind::Tagged(ValueType::Array), + effect: BuiltinEffect::OwnedMutation, + needs_failure_exit: true, +}; + +/// `map.set(key, value)` — owned mutation, returns the mutated map. +pub(crate) const MAP_SET_SPEC: BuiltinSpec = BuiltinSpec { + name: "map_set", + arity: 3, + inputs: &[ + InputRepr::Any, // value (popped third) + InputRepr::Any, // key (popped second) + InputRepr::Tagged, // map (popped first, must be owned Tagged) + ], + output: OutputKind::Tagged(ValueType::Map), + effect: BuiltinEffect::OwnedMutation, + needs_failure_exit: true, +}; + +// ── Family 6: map iterators ───────────────────────────────────────── + +/// `map_iter_next(slot)` — advance iterator, bool result. +pub(crate) const MAP_ITER_NEXT_SPEC: BuiltinSpec = BuiltinSpec { + name: "map_iter_next", + arity: 1, + inputs: &[InputRepr::Int], // slot + output: OutputKind::Bool, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `map_iter_take_key(slot)` — take current key, tagged result. +pub(crate) const MAP_ITER_TAKE_KEY_SPEC: BuiltinSpec = BuiltinSpec { + name: "map_iter_take_key", + arity: 1, + inputs: &[InputRepr::Int], // slot + output: OutputKind::TaggedUnknown, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// `map_iter_take_value(slot)` — take current value, tagged result. +pub(crate) const MAP_ITER_TAKE_VALUE_SPEC: BuiltinSpec = BuiltinSpec { + name: "map_iter_take_value", + arity: 1, + inputs: &[InputRepr::Int], // slot + output: OutputKind::TaggedUnknown, + effect: BuiltinEffect::Pure, + needs_failure_exit: false, +}; + +/// Look up the spec for a specialized builtin kind, if one exists. +/// +/// Returns `None` for builtins not yet covered by the spec-driven +/// path; their hand-written recorder paths remain authoritative. +pub(crate) fn spec_for( + kind: super::recorder::SpecializedBuiltinKind, +) -> Option<&'static BuiltinSpec> { + match kind { + super::recorder::SpecializedBuiltinKind::StringLen => Some(&STRING_LEN_SPEC), + super::recorder::SpecializedBuiltinKind::RegexMatch => Some(®EX_MATCH_SPEC), + super::recorder::SpecializedBuiltinKind::ArraySet => Some(&ARRAY_SET_SPEC), + super::recorder::SpecializedBuiltinKind::ValueLen => Some(&VALUE_LEN_SPEC), + super::recorder::SpecializedBuiltinKind::BytesLen => Some(&BYTES_LEN_SPEC), + super::recorder::SpecializedBuiltinKind::ArrayLen => Some(&ARRAY_LEN_SPEC), + super::recorder::SpecializedBuiltinKind::MapLen => Some(&MAP_LEN_SPEC), + super::recorder::SpecializedBuiltinKind::TypeOf => Some(&TYPE_OF_SPEC), + super::recorder::SpecializedBuiltinKind::StringContains => Some(&STRING_CONTAINS_SPEC), + super::recorder::SpecializedBuiltinKind::ArrayHas => Some(&ARRAY_HAS_SPEC), + super::recorder::SpecializedBuiltinKind::StringSlice => Some(&STRING_SLICE_SPEC), + super::recorder::SpecializedBuiltinKind::BytesSlice => Some(&BYTES_SLICE_SPEC), + super::recorder::SpecializedBuiltinKind::StringGet => Some(&STRING_GET_SPEC), + super::recorder::SpecializedBuiltinKind::BytesGet => Some(&BYTES_GET_SPEC), + super::recorder::SpecializedBuiltinKind::BytesHas => Some(&BYTES_HAS_SPEC), + super::recorder::SpecializedBuiltinKind::StringReplaceLiteral => { + Some(&STRING_REPLACE_LITERAL_SPEC) + } + super::recorder::SpecializedBuiltinKind::StringLowerAscii => Some(&STRING_LOWER_ASCII_SPEC), + super::recorder::SpecializedBuiltinKind::StringSplitLiteral => { + Some(&STRING_SPLIT_LITERAL_SPEC) + } + super::recorder::SpecializedBuiltinKind::BytesFromArrayU8 => { + Some(&BYTES_FROM_ARRAY_U8_SPEC) + } + super::recorder::SpecializedBuiltinKind::BytesToUtf8Ascii => { + Some(&BYTES_TO_UTF8_ASCII_SPEC) + } + super::recorder::SpecializedBuiltinKind::BytesToArrayU8 => Some(&BYTES_TO_ARRAY_U8_SPEC), + super::recorder::SpecializedBuiltinKind::ToString => Some(&TO_STRING_SPEC), + super::recorder::SpecializedBuiltinKind::RegexReplace => Some(®EX_REPLACE_SPEC), + super::recorder::SpecializedBuiltinKind::ArrayGet => Some(&ARRAY_GET_SPEC), + super::recorder::SpecializedBuiltinKind::MapGet => Some(&MAP_GET_SPEC), + super::recorder::SpecializedBuiltinKind::MapHas => Some(&MAP_HAS_SPEC), + super::recorder::SpecializedBuiltinKind::ArrayPush => Some(&ARRAY_PUSH_SPEC), + super::recorder::SpecializedBuiltinKind::MapSet => Some(&MAP_SET_SPEC), + super::recorder::SpecializedBuiltinKind::MapIterNext => Some(&MAP_ITER_NEXT_SPEC), + super::recorder::SpecializedBuiltinKind::MapIterTakeKey => Some(&MAP_ITER_TAKE_KEY_SPEC), + super::recorder::SpecializedBuiltinKind::MapIterTakeValue => { + Some(&MAP_ITER_TAKE_VALUE_SPEC) + } + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const ALL_SPECS: &[&BuiltinSpec] = &[ + &STRING_LEN_SPEC, + ®EX_MATCH_SPEC, + &ARRAY_SET_SPEC, + &VALUE_LEN_SPEC, + &BYTES_LEN_SPEC, + &ARRAY_LEN_SPEC, + &MAP_LEN_SPEC, + &TYPE_OF_SPEC, + &STRING_CONTAINS_SPEC, + &ARRAY_HAS_SPEC, + ]; + + #[test] + fn all_specs_have_consistent_arity_and_inputs() { + for spec in ALL_SPECS { + assert_eq!( + spec.arity, + spec.inputs.len(), + "{}: arity must match inputs", + spec.name + ); + } + } + + #[test] + fn effect_classification_is_explicit() { + const { + assert!(matches!(STRING_LEN_SPEC.effect, BuiltinEffect::Pure)); + assert!(!STRING_LEN_SPEC.needs_failure_exit); + assert!(matches!( + REGEX_MATCH_SPEC.effect, + BuiltinEffect::FallibleHelper + )); + assert!(REGEX_MATCH_SPEC.needs_failure_exit); + assert!(matches!( + ARRAY_SET_SPEC.effect, + BuiltinEffect::OwnedMutation + )); + assert!(ARRAY_SET_SPEC.needs_failure_exit); + } + for spec in ALL_SPECS { + if spec.needs_failure_exit { + assert!( + !matches!(spec.effect, BuiltinEffect::Pure), + "{}: pure builtins must not need failure exit", + spec.name + ); + } + } + } +} diff --git a/src/vm/jit/diagnostics.rs b/src/vm/jit/diagnostics.rs new file mode 100644 index 00000000..9fe0dafe --- /dev/null +++ b/src/vm/jit/diagnostics.rs @@ -0,0 +1,124 @@ +use super::super::Vm; +use super::{JitMetrics, JitSnapshot, native}; + +impl Vm { + pub(super) fn jit_diagnostics_snapshot(&self) -> JitSnapshot { + self.jit.snapshot(self.jit_diagnostics_metrics()) + } + + pub(super) fn jit_diagnostics_dump(&self, include_machine_code: bool) -> String { + let mut out = self + .jit + .dump_text(self.program.debug.as_ref(), self.jit_diagnostics_metrics()); + out.push_str(&format!( + " native codegen backend: {}\n", + native::selected_codegen_backend() + )); + out.push_str(&format!( + " native trace executions: {}\n", + self.native_trace_exec_count + )); + out.push_str(&format!( + " native trace handoffs: {}\n", + self.jit_native_link_handoff_count + )); + out.push_str(&format!( + " native region entries: {}\n", + self.jit_native_region_entry_count + )); + out.push_str(&format!( + " native internal region edges: {}\n", + self.jit_native_region_edge_count + )); + out.push_str(&format!( + " native direct side links: {}\n", + self.jit_native_direct_link_count + )); + out.push_str(&format!( + " native compile time: {} ns (regions={} ns)\n", + self.jit_native_compile_time_ns, self.jit_native_region_compile_time_ns + )); + out.push_str(&format!( + " native code bytes: {} (regions={})\n", + self.jit_native_code_bytes(), + self.jit_native_region_code_bytes() + )); + if self.jit_native_bridge_stats_enabled { + let mut bridge_entries: Vec<(&'static str, u64)> = self + .jit_native_bridge_counts + .iter() + .map(|(name, count)| (*name, *count)) + .collect(); + bridge_entries.sort_unstable_by_key(|(name, _)| *name); + let total_bridge_hits = bridge_entries + .iter() + .fold(0u64, |acc, (_, count)| acc.saturating_add(*count)); + out.push_str(&format!( + " native bridge hits: {} (helpers={})\n", + total_bridge_hits, + bridge_entries.len() + )); + for (name, count) in bridge_entries { + out.push_str(&format!(" bridge {}: {}\n", name, count)); + } + } + let native_trace_count = self.native_traces.iter().flatten().count(); + if native_trace_count == 0 { + out.push_str(" native traces: 0\n"); + return out; + } + + out.push_str(&format!(" native traces: {}\n", native_trace_count)); + for (id, native) in self.native_traces.iter().enumerate() { + if let Some(native) = native { + out.push_str(&format!( + " native trace#{} entry=0x{:X} code_bytes={} lowering={}\n", + id, + native.entry as usize, + native.code.len(), + native.lowering_kind.as_str() + )); + if include_machine_code { + out.push_str(" code:"); + for byte in native.code.iter() { + out.push_str(&format!(" {:02X}", byte)); + } + out.push('\n'); + } + if let Some(region) = &native.region { + out.push_str(&format!( + " region entry=0x{:X} code_bytes={} lowering={}\n", + region.entry as usize, + region.code.len(), + region.lowering_kind.as_str() + )); + if include_machine_code { + out.push_str(" code:"); + for byte in region.code.iter() { + out.push_str(&format!(" {:02X}", byte)); + } + out.push('\n'); + } + } + } + } + out + } + + fn jit_diagnostics_metrics(&self) -> JitMetrics { + JitMetrics { + boxed_load_site_count: 0, + boxed_store_site_count: 0, + trace_exit_count: self.jit_trace_exit_count, + native_loop_back_count: self.jit_native_loop_back_count, + helper_fallback_count: self.jit_helper_fallback_count, + native_trace_exec_count: self.native_trace_exec_count, + script_call_observations: 0, + monomorphic_call_sites: 0, + polymorphic_call_sites: 0, + inline_attempts: 0, + inline_successes: 0, + inline_rejections: 0, + } + } +} diff --git a/src/vm/jit/mod.rs b/src/vm/jit/mod.rs index 0afe9be2..88a70c09 100644 --- a/src/vm/jit/mod.rs +++ b/src/vm/jit/mod.rs @@ -1,4 +1,6 @@ +pub(crate) mod builtin_spec; pub(crate) mod deopt; +pub(crate) mod diagnostics; pub(crate) mod inline; pub(crate) mod ir; pub(crate) mod liveness; diff --git a/src/vm/jit/native/lower.rs b/src/vm/jit/native/lower.rs index ca4c9794..5eb8a5af 100644 --- a/src/vm/jit/native/lower.rs +++ b/src/vm/jit/native/lower.rs @@ -189,13 +189,16 @@ pub(crate) fn compile_tail_status_body(status: i32) -> VmResult VmResult { + debug_assert!(matches!(abi, TailCallAbi::Plain | TailCallAbi::Owned)); compile_standalone_native_function( - "pd_vm_tail_side_link", - |pointer_type, _| tail_entry_signature(pointer_type), + prefix, + move |pointer_type, _| abi.tail_signature(pointer_type), move |builder, pointer_type, _| { let entry = builder.create_block(); let deopt = builder.create_block(); @@ -203,6 +206,13 @@ pub(crate) fn compile_tail_side_link_body( builder.append_block_params_for_function_params(entry); builder.switch_to_block(entry); let vm_ptr = builder.block_params(entry)[0]; + let args = match abi { + TailCallAbi::Plain => vec![vm_ptr], + TailCallAbi::Owned => vec![vm_ptr, builder.block_params(entry)[1]], + TailCallAbi::Inherited => { + unreachable!("inherited packets use the trace dispatcher") + } + }; let slot_address = iconst_ptr_from_addr(builder, pointer_type, slot_address)?; let target = builder .ins() @@ -215,30 +225,112 @@ pub(crate) fn compile_tail_side_link_body( builder.ins().return_(&[status]); builder.switch_to_block(linked); - let signature = builder.import_signature(tail_entry_signature(pointer_type)); - builder - .ins() - .return_call_indirect(signature, target, &[vm_ptr]); + let signature = builder.import_signature(abi.tail_signature(pointer_type)); + builder.ins().return_call_indirect(signature, target, &args); Ok(()) }, ) } -pub(crate) fn compile_system_tail_wrapper(root_entry: *const u8) -> VmResult { +pub(crate) fn compile_tail_side_link_body( + slot_address: usize, + deopt_status: i32, +) -> VmResult { + compile_tail_side_link_body_with_abi( + "pd_vm_tail_side_link", + slot_address, + deopt_status, + TailCallAbi::Plain, + ) +} + +#[derive(Clone, Copy)] +enum TailCallAbi { + Plain, + Inherited, + Owned, +} + +impl TailCallAbi { + fn system_signature( + self, + pointer_type: cranelift_codegen::ir::Type, + call_conv: CallConv, + ) -> Signature { + match self { + Self::Plain | Self::Inherited => entry_signature(pointer_type, call_conv), + Self::Owned => system_owned_entry_signature(pointer_type, call_conv), + } + } + + fn tail_signature(self, pointer_type: cranelift_codegen::ir::Type) -> Signature { + match self { + Self::Plain => tail_entry_signature(pointer_type), + Self::Inherited => inherited_tail_entry_signature(pointer_type), + Self::Owned => tail_owned_entry_signature(pointer_type), + } + } +} + +fn create_inherited_state_packet( + builder: &mut FunctionBuilder<'_>, + pointer_type: cranelift_codegen::ir::Type, +) -> VmResult { + let pointer_bytes = pointer_type.bits() / 8; + let packet_bytes = pointer_bytes + .checked_mul((MAX_INHERITED_ENTRY_VALUES + 7) as u32) + .ok_or_else(|| { + VmError::JitNative("native inherited-state packet is too large".to_string()) + })?; + let packet = builder.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + packet_bytes, + pointer_bytes.trailing_zeros() as u8, + )); + let packet_ptr = builder.ins().stack_addr(pointer_type, packet, 0); + let inactive = builder.ins().iconst(pointer_type, 0); + builder.ins().store( + MemFlags::new(), + inactive, + packet_ptr, + INHERITED_STATE_ACTIVE_OFFSET, + ); + builder.ins().store( + MemFlags::new(), + inactive, + packet_ptr, + INHERITED_STATE_DYNAMIC_TARGET_OFFSET, + ); + Ok(packet_ptr) +} + +fn compile_system_tail_wrapper_with_abi( + prefix: &str, + root_entry: *const u8, + abi: TailCallAbi, +) -> VmResult { let root_entry = root_entry as usize; compile_standalone_native_function( - "pd_vm_tail_wrapper", - entry_signature, + prefix, + move |pointer_type, call_conv| abi.system_signature(pointer_type, call_conv), move |builder, pointer_type, _| { let entry = builder.create_block(); builder.append_block_params_for_function_params(entry); builder.switch_to_block(entry); let vm_ptr = builder.block_params(entry)[0]; + let args = match abi { + TailCallAbi::Plain => vec![vm_ptr], + TailCallAbi::Inherited => { + vec![ + vm_ptr, + create_inherited_state_packet(builder, pointer_type)?, + ] + } + TailCallAbi::Owned => vec![vm_ptr, builder.block_params(entry)[1]], + }; let root_entry = iconst_ptr_from_addr(builder, pointer_type, root_entry)?; - let signature = builder.import_signature(tail_entry_signature(pointer_type)); - let call = builder - .ins() - .call_indirect(signature, root_entry, &[vm_ptr]); + let signature = builder.import_signature(abi.tail_signature(pointer_type)); + let call = builder.ins().call_indirect(signature, root_entry, &args); let status = builder.inst_results(call)[0]; builder.ins().return_(&[status]); Ok(()) @@ -246,52 +338,17 @@ pub(crate) fn compile_system_tail_wrapper(root_entry: *const u8) -> VmResult VmResult { + compile_system_tail_wrapper_with_abi("pd_vm_tail_wrapper", root_entry, TailCallAbi::Plain) +} + pub(crate) fn compile_system_inherited_tail_wrapper( root_entry: *const u8, ) -> VmResult { - let root_entry = root_entry as usize; - compile_standalone_native_function( + compile_system_tail_wrapper_with_abi( "pd_vm_inherited_tail_wrapper", - entry_signature, - move |builder, pointer_type, _| { - let entry = builder.create_block(); - builder.append_block_params_for_function_params(entry); - builder.switch_to_block(entry); - let vm_ptr = builder.block_params(entry)[0]; - let pointer_bytes = pointer_type.bits() / 8; - let packet_bytes = pointer_bytes - .checked_mul((MAX_INHERITED_ENTRY_VALUES + 7) as u32) - .ok_or_else(|| { - VmError::JitNative("native inherited-state packet is too large".to_string()) - })?; - let packet = builder.create_sized_stack_slot(StackSlotData::new( - StackSlotKind::ExplicitSlot, - packet_bytes, - pointer_bytes.trailing_zeros() as u8, - )); - let packet_ptr = builder.ins().stack_addr(pointer_type, packet, 0); - let inactive = builder.ins().iconst(pointer_type, 0); - builder.ins().store( - MemFlags::new(), - inactive, - packet_ptr, - INHERITED_STATE_ACTIVE_OFFSET, - ); - builder.ins().store( - MemFlags::new(), - inactive, - packet_ptr, - INHERITED_STATE_DYNAMIC_TARGET_OFFSET, - ); - let root_entry = iconst_ptr_from_addr(builder, pointer_type, root_entry)?; - let signature = builder.import_signature(inherited_tail_entry_signature(pointer_type)); - let call = builder - .ins() - .call_indirect(signature, root_entry, &[vm_ptr, packet_ptr]); - let status = builder.inst_results(call)[0]; - builder.ins().return_(&[status]); - Ok(()) - }, + root_entry, + TailCallAbi::Inherited, ) } @@ -455,35 +512,11 @@ pub(crate) fn compile_tail_owned_side_link_body( slot_address: usize, deopt_status: i32, ) -> VmResult { - compile_standalone_native_function( + compile_tail_side_link_body_with_abi( "pd_vm_tail_owned_side_link", - |pointer_type, _| tail_owned_entry_signature(pointer_type), - move |builder, pointer_type, _| { - let entry = builder.create_block(); - let deopt = builder.create_block(); - let linked = builder.create_block(); - builder.append_block_params_for_function_params(entry); - builder.switch_to_block(entry); - let vm_ptr = builder.block_params(entry)[0]; - let owned_slot = builder.block_params(entry)[1]; - let slot_address = iconst_ptr_from_addr(builder, pointer_type, slot_address)?; - let target = builder - .ins() - .atomic_load(pointer_type, MemFlags::new(), slot_address); - let is_null = builder.ins().icmp_imm(IntCC::Equal, target, 0); - builder.ins().brif(is_null, deopt, &[], linked, &[]); - - builder.switch_to_block(deopt); - let status = builder.ins().iconst(types::I32, i64::from(deopt_status)); - builder.ins().return_(&[status]); - - builder.switch_to_block(linked); - let signature = builder.import_signature(tail_owned_entry_signature(pointer_type)); - builder - .ins() - .return_call_indirect(signature, target, &[vm_ptr, owned_slot]); - Ok(()) - }, + slot_address, + deopt_status, + TailCallAbi::Owned, ) } @@ -530,26 +563,7 @@ pub(crate) fn compile_tail_owned_clear_body(success_status: i32) -> VmResult VmResult { - let root_entry = root_entry as usize; - compile_standalone_native_function( - "pd_vm_tail_owned_wrapper", - system_owned_entry_signature, - move |builder, pointer_type, _| { - let entry = builder.create_block(); - builder.append_block_params_for_function_params(entry); - builder.switch_to_block(entry); - let vm_ptr = builder.block_params(entry)[0]; - let owned_slot = builder.block_params(entry)[1]; - let root_entry = iconst_ptr_from_addr(builder, pointer_type, root_entry)?; - let signature = builder.import_signature(tail_owned_entry_signature(pointer_type)); - let call = builder - .ins() - .call_indirect(signature, root_entry, &[vm_ptr, owned_slot]); - let status = builder.inst_results(call)[0]; - builder.ins().return_(&[status]); - Ok(()) - }, - ) + compile_system_tail_wrapper_with_abi("pd_vm_tail_owned_wrapper", root_entry, TailCallAbi::Owned) } fn try_compile_ssa_trace( @@ -2803,27 +2817,60 @@ fn lower_ssa_inst( out } SsaInstKind::TypeOf { value } => { - let value = values[value]; + let input = values[value]; + let input_repr = *value_reprs.get(value).ok_or_else(|| { + VmError::JitNative("SSA type_of input representation missing".to_string()) + })?; let out = owned_value_temp_slot_addr( b, pointer_type, owned_value_temps, SsaTempValueSlotKey::Output(output.id), )?; - let out_raw = ssa_call_type_of(b, pointer_type, string_refs, string_addrs, value)?; + let input_addr = ssa_ensure_boxed_value_addr( + b, + SsaBoxCtx { + exit_block, + pointer_type, + value_layout: layout.value, + helper_refs, + helper_addrs, + }, + Some(out), + input_repr, + input, + )?; + let out_raw = ssa_call_type_of(b, pointer_type, string_refs, string_addrs, input_addr)?; clear_owned_value_temp_slot(b, pointer_type, helper_refs, helper_addrs, out)?; ssa_store_heap_ptr_in_value(b, layout.value, out, layout.value.string_tag, out_raw); out } SsaInstKind::ToString { value } => { - let value = values[value]; + let input = values[value]; + let input_repr = *value_reprs.get(value).ok_or_else(|| { + VmError::JitNative("SSA to_string input representation missing".to_string()) + })?; let out = owned_value_temp_slot_addr( b, pointer_type, owned_value_temps, SsaTempValueSlotKey::Output(output.id), )?; - let out_raw = ssa_call_to_string(b, pointer_type, string_refs, string_addrs, value)?; + let input_addr = ssa_ensure_boxed_value_addr( + b, + SsaBoxCtx { + exit_block, + pointer_type, + value_layout: layout.value, + helper_refs, + helper_addrs, + }, + Some(out), + input_repr, + input, + )?; + let out_raw = + ssa_call_to_string(b, pointer_type, string_refs, string_addrs, input_addr)?; clear_owned_value_temp_slot(b, pointer_type, helper_refs, helper_addrs, out)?; ssa_store_heap_ptr_in_value(b, layout.value, out, layout.value.string_tag, out_raw); out diff --git a/src/vm/jit/recorder.rs b/src/vm/jit/recorder.rs index 642823d0..eaaa6524 100644 --- a/src/vm/jit/recorder.rs +++ b/src/vm/jit/recorder.rs @@ -5,6 +5,7 @@ use crate::compiler::TypeSchema; use crate::vm::{OpCode, Program, Value, ValueType, checked_int_div}; use super::JitTraceTerminal; +use super::builtin_spec::{self, InputRepr, OutputKind}; use super::deopt::materialize_ssa_values; use super::inline::{InlineCandidate, InlineRejectReason, classify_static_inline_candidate}; use super::ir::{ @@ -12,6 +13,10 @@ use super::ir::{ SsaValue, SsaValueId, SsaValueRepr, VirtualFrameSnapshot, }; +pub(super) const MAX_PROFITABLE_FRAME_LOCALS: usize = 64; +pub(super) const WIDE_FRAME_REASON: &str = + "trace frame has too many locals for profitable native execution"; + #[derive(Clone, Debug, PartialEq)] pub(crate) struct RecordedTrace { pub(crate) has_call: bool, @@ -680,7 +685,7 @@ enum HeapContainerKind { } #[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum SpecializedBuiltinKind { +pub(crate) enum SpecializedBuiltinKind { ValueLen, StringLen, BytesLen, @@ -934,6 +939,11 @@ pub(crate) fn record_trace_with_local_count( max_trace_len: usize, non_yielding_host_imports: &[bool], ) -> Result { + if local_count > MAX_PROFITABLE_FRAME_LOCALS { + return Err(TraceRecordError::UnsupportedTrace( + WIDE_FRAME_REASON.to_string(), + )); + } let loop_header_plan = infer_loop_header_plan( program, caller_frame_key, @@ -3220,6 +3230,7 @@ fn emit_float_compare( } impl HeapContainerKind { + #[allow(dead_code)] // Used by ensure_owned_heap_ptr and future families. fn value_type(self) -> ValueType { match self { Self::String => ValueType::String, @@ -3373,91 +3384,61 @@ fn analyze_specialized_builtin_call( frame: &mut AnalysisFrame, kind: SpecializedBuiltinKind, ) -> Result<&'static str, TraceRecordError> { - match kind { - SpecializedBuiltinKind::ValueLen => { - let _ = frame.pop()?; - frame.push(ValueInfo::int(None)); - Ok("value_len") - } - SpecializedBuiltinKind::StringLen => { - let _ = frame.pop()?; - frame.push(ValueInfo::int(None)); - Ok("string_len") - } - SpecializedBuiltinKind::BytesLen => { - let _ = frame.pop()?; - frame.push(ValueInfo::int(None)); - Ok("bytes_len") - } - SpecializedBuiltinKind::StringSlice => { - let _ = frame.pop()?; - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::String)); - Ok("string_slice") - } - SpecializedBuiltinKind::BytesSlice => { - let _ = frame.pop()?; - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::Bytes)); - Ok("bytes_slice") - } - SpecializedBuiltinKind::StringGet => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::String)); - Ok("string_get") - } - SpecializedBuiltinKind::BytesGet => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::int(None)); - Ok("bytes_get") - } - SpecializedBuiltinKind::BytesHas => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::bool(None)); - Ok("bytes_has") - } - SpecializedBuiltinKind::StringContains => { - let _ = frame.pop()?; + // Spec-driven fast path for pilot builtins. + if let Some(spec) = builtin_spec::spec_for(kind) { + for _ in 0..spec.arity { let _ = frame.pop()?; - frame.push(ValueInfo::bool(None)); - Ok("string_contains") } - SpecializedBuiltinKind::RegexMatch => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::bool(None)); - Ok("regex_match") - } - SpecializedBuiltinKind::RegexReplace => { - let _ = frame.pop()?; - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::String)); - Ok("regex_replace") - } - SpecializedBuiltinKind::StringReplaceLiteral => { - let _ = frame.pop()?; - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::String)); - Ok("string_replace_literal") - } - SpecializedBuiltinKind::StringLowerAscii => { - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::String)); - Ok("string_lower_ascii") - } - SpecializedBuiltinKind::TypeOf | SpecializedBuiltinKind::TypeOfKnown(_) => { + let info = match spec.output { + OutputKind::Int => ValueInfo::int(None), + OutputKind::Bool => ValueInfo::bool(None), + OutputKind::Tagged(vt) => ValueInfo::tagged_typed(vt), + OutputKind::TypeName => ValueInfo::type_name(), + OutputKind::TaggedUnknown => ValueInfo::tagged(), + }; + frame.push(info); + return Ok(spec.name); + } + match kind { + SpecializedBuiltinKind::ValueLen + | SpecializedBuiltinKind::StringLen + | SpecializedBuiltinKind::BytesLen + | SpecializedBuiltinKind::RegexMatch + | SpecializedBuiltinKind::ArraySet + | SpecializedBuiltinKind::ArrayLen + | SpecializedBuiltinKind::MapLen + | SpecializedBuiltinKind::TypeOf + | SpecializedBuiltinKind::StringContains + | SpecializedBuiltinKind::ArrayHas + | SpecializedBuiltinKind::StringSlice + | SpecializedBuiltinKind::BytesSlice + | SpecializedBuiltinKind::StringGet + | SpecializedBuiltinKind::BytesGet + | SpecializedBuiltinKind::BytesHas + | SpecializedBuiltinKind::StringReplaceLiteral + | SpecializedBuiltinKind::StringLowerAscii + | SpecializedBuiltinKind::StringSplitLiteral + | SpecializedBuiltinKind::BytesFromArrayU8 + | SpecializedBuiltinKind::BytesToUtf8Ascii + | SpecializedBuiltinKind::BytesToArrayU8 + | SpecializedBuiltinKind::ToString + | SpecializedBuiltinKind::RegexReplace + | SpecializedBuiltinKind::ArrayGet + | SpecializedBuiltinKind::MapGet + | SpecializedBuiltinKind::MapHas + | SpecializedBuiltinKind::ArrayPush + | SpecializedBuiltinKind::MapSet + | SpecializedBuiltinKind::MapIterNext + | SpecializedBuiltinKind::MapIterTakeKey + | SpecializedBuiltinKind::MapIterTakeValue => { + unreachable!("spec-covered builtins are handled by the spec-driven path") + } + SpecializedBuiltinKind::TypeOfKnown(_) => { let _ = frame.pop()?; frame.push(ValueInfo::type_name()); Ok("type_of") } - SpecializedBuiltinKind::ToString | SpecializedBuiltinKind::ToStringIdentity => { + SpecializedBuiltinKind::ToStringIdentity => { let _ = frame.pop()?; frame.push(ValueInfo::tagged_typed(ValueType::String)); Ok( @@ -3468,12 +3449,6 @@ fn analyze_specialized_builtin_call( }, ) } - SpecializedBuiltinKind::StringSplitLiteral => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::Array)); - Ok("string_split_literal") - } SpecializedBuiltinKind::StringConcat => { let _ = frame.pop()?; let _ = frame.pop()?; @@ -3486,94 +3461,10 @@ fn analyze_specialized_builtin_call( frame.push(ValueInfo::tagged_typed(ValueType::Bytes)); Ok("bytes_concat") } - SpecializedBuiltinKind::BytesFromArrayU8 => { - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::Bytes)); - Ok("bytes_from_array_u8") - } - SpecializedBuiltinKind::BytesToUtf8Ascii => { - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::String)); - Ok("bytes_to_utf8_ascii") - } - SpecializedBuiltinKind::BytesToArrayU8 => { - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::Array)); - Ok("bytes_to_array_u8") - } SpecializedBuiltinKind::ArrayNew => { frame.push(ValueInfo::tagged_typed(ValueType::Array)); Ok("array_new") } - SpecializedBuiltinKind::ArrayLen => { - let _ = frame.pop()?; - frame.push(ValueInfo::int(None)); - Ok("array_len") - } - SpecializedBuiltinKind::ArrayGet => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged()); - Ok("array_get") - } - SpecializedBuiltinKind::ArrayHas => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::bool(None)); - Ok("array_has") - } - SpecializedBuiltinKind::ArraySet => { - let _ = frame.pop()?; - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::Array)); - Ok("array_set") - } - SpecializedBuiltinKind::ArrayPush => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::Array)); - Ok("array_push") - } - SpecializedBuiltinKind::MapLen => { - let _ = frame.pop()?; - frame.push(ValueInfo::int(None)); - Ok("map_len") - } - SpecializedBuiltinKind::MapGet => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged()); - Ok("map_get") - } - SpecializedBuiltinKind::MapHas => { - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::bool(None)); - Ok("map_has") - } - SpecializedBuiltinKind::MapSet => { - let _ = frame.pop()?; - let _ = frame.pop()?; - let _ = frame.pop()?; - frame.push(ValueInfo::tagged_typed(ValueType::Map)); - Ok("map_set") - } - SpecializedBuiltinKind::MapIterNext => { - let _ = frame.pop()?; - frame.push(ValueInfo::bool(None)); - Ok("map_iter_next") - } - SpecializedBuiltinKind::MapIterTakeKey => { - let _ = frame.pop()?; - frame.push(ValueInfo::tagged()); - Ok("map_iter_take_key") - } - SpecializedBuiltinKind::MapIterTakeValue => { - let _ = frame.pop()?; - frame.push(ValueInfo::tagged()); - Ok("map_iter_take_value") - } } } @@ -3584,657 +3475,371 @@ fn emit_specialized_builtin_call( frame: &mut SymbolicFrame, kind: SpecializedBuiltinKind, ) -> Result<(&'static str, SymbolicValue), TraceRecordError> { + // Spec-driven fast path for pilot builtins. + if let Some(spec) = builtin_spec::spec_for(kind) { + return emit_spec_driven_builtin(builder, block, ip, frame, kind, spec); + } match kind { - SpecializedBuiltinKind::ValueLen => { - let value = frame.pop()?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::I64, - SsaInstKind::ValueLen { - value: value.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::int(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("value_len", out)) - } - SpecializedBuiltinKind::StringLen => { - let text = frame.pop()?; - let text = ensure_heap_ptr( - builder, - block, - ip, - text, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::I64, - SsaInstKind::StringLen { - text: text.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::int(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("string_len", out)) + SpecializedBuiltinKind::ValueLen + | SpecializedBuiltinKind::StringLen + | SpecializedBuiltinKind::BytesLen + | SpecializedBuiltinKind::RegexMatch + | SpecializedBuiltinKind::ArraySet + | SpecializedBuiltinKind::ArrayLen + | SpecializedBuiltinKind::MapLen + | SpecializedBuiltinKind::TypeOf + | SpecializedBuiltinKind::StringContains + | SpecializedBuiltinKind::ArrayHas + | SpecializedBuiltinKind::StringSlice + | SpecializedBuiltinKind::BytesSlice + | SpecializedBuiltinKind::StringGet + | SpecializedBuiltinKind::BytesGet + | SpecializedBuiltinKind::BytesHas + | SpecializedBuiltinKind::StringReplaceLiteral + | SpecializedBuiltinKind::StringLowerAscii + | SpecializedBuiltinKind::StringSplitLiteral + | SpecializedBuiltinKind::BytesFromArrayU8 + | SpecializedBuiltinKind::BytesToUtf8Ascii + | SpecializedBuiltinKind::BytesToArrayU8 + | SpecializedBuiltinKind::ToString + | SpecializedBuiltinKind::RegexReplace + | SpecializedBuiltinKind::ArrayGet + | SpecializedBuiltinKind::MapGet + | SpecializedBuiltinKind::MapHas + | SpecializedBuiltinKind::ArrayPush + | SpecializedBuiltinKind::MapSet + | SpecializedBuiltinKind::MapIterNext + | SpecializedBuiltinKind::MapIterTakeKey + | SpecializedBuiltinKind::MapIterTakeValue => { + unreachable!("spec-covered builtins are handled by the spec-driven path") } - SpecializedBuiltinKind::BytesLen => { - let bytes = frame.pop()?; - let bytes = ensure_heap_ptr( - builder, - block, - ip, - bytes, - HeapContainerKind::Bytes.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::I64, - SsaInstKind::BytesLen { - bytes: bytes.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::int(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("bytes_len", out)) - } - SpecializedBuiltinKind::StringSlice => { - let length = ensure_int(builder, block, ip, frame.pop()?)?; - let start = ensure_int(builder, block, ip, frame.pop()?)?; - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::StringSlice { - text: text.value.id, - start: start.value.id, - length: length.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::String), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("string_slice", out)) - } - SpecializedBuiltinKind::BytesSlice => { - let length = ensure_int(builder, block, ip, frame.pop()?)?; - let start = ensure_int(builder, block, ip, frame.pop()?)?; - let bytes = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::Bytes.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::BytesSlice { - bytes: bytes.value.id, - start: start.value.id, - length: length.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Bytes), - }) + SpecializedBuiltinKind::TypeOfKnown(value_type) => { + let _ = frame.pop()?; + let type_name = match value_type { + ValueType::Null => "null", + ValueType::Int => "int", + ValueType::Float => "float", + ValueType::Bool => "bool", + ValueType::String => "string", + ValueType::Bytes => "bytes", + ValueType::Array => "array", + ValueType::Map => "map", + ValueType::Callable => "callable", + ValueType::Unknown => { + return Err(TraceRecordError::UnsupportedTrace( + "type_of known specialization received unknown type".to_string(), + )); + } + }; + let constant = Value::string(type_name); + let info = ValueInfo::type_name(); + let value = builder + .append_value_inst(block, ip, info.repr, SsaInstKind::Constant(constant)) .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("bytes_slice", out)) + Ok(("type_of", SymbolicValue { value, info })) } - SpecializedBuiltinKind::StringGet => { - let index = ensure_int(builder, block, ip, frame.pop()?)?; - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::StringGet { - text: text.value.id, - index: index.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::String), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("string_get", out)) + SpecializedBuiltinKind::ToStringIdentity => { + let value = frame.pop()?; + Ok(("to_string_identity", value)) } - SpecializedBuiltinKind::BytesGet => { - let index = ensure_int(builder, block, ip, frame.pop()?)?; - let bytes = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::Bytes.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::I64, - SsaInstKind::BytesGet { - bytes: bytes.value.id, - index: index.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::int(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("bytes_get", out)) + SpecializedBuiltinKind::StringConcat => { + let rhs = frame.pop()?; + let lhs = frame.pop()?; + emit_concat_binop(builder, block, ip, ConcatBinOpKind::String, lhs, rhs) } - SpecializedBuiltinKind::BytesHas => { - let index = ensure_int(builder, block, ip, frame.pop()?)?; - let bytes = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::Bytes.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Bool, - SsaInstKind::BytesHas { - bytes: bytes.value.id, - index: index.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::bool(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("bytes_has", out)) + SpecializedBuiltinKind::BytesConcat => { + let rhs = frame.pop()?; + let lhs = frame.pop()?; + emit_concat_binop(builder, block, ip, ConcatBinOpKind::Bytes, lhs, rhs) } - SpecializedBuiltinKind::StringContains => { - let needle = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; + SpecializedBuiltinKind::ArrayNew => { let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Bool, - SsaInstKind::StringContains { - text: text.value.id, - needle: needle.value.id, - }, - ) + .append_value_inst(block, ip, SsaValueRepr::Tagged, SsaInstKind::ArrayNew) .map(|value| SymbolicValue { value, - info: ValueInfo::bool(None), + info: ValueInfo::tagged_typed(ValueType::Array), }) .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("string_contains", out)) + Ok(("array_new", out)) } - SpecializedBuiltinKind::RegexMatch => { - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let pattern = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Bool, - SsaInstKind::RegexMatch { - pattern: pattern.value.id, - text: text.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::bool(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("regex_match", out)) - } - SpecializedBuiltinKind::RegexReplace => { - let replacement = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let pattern = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::RegexReplace { - pattern: pattern.value.id, - text: text.value.id, - replacement: replacement.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::String), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("regex_replace", out)) - } - SpecializedBuiltinKind::StringReplaceLiteral => { - let replacement = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let needle = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::StringReplaceLiteral { - text: text.value.id, - needle: needle.value.id, - replacement: replacement.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::String), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("string_replace_literal", out)) - } - SpecializedBuiltinKind::StringLowerAscii => { - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::StringLowerAscii { - text: text.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::String), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("string_lower_ascii", out)) - } - SpecializedBuiltinKind::TypeOfKnown(value_type) => { - let _ = frame.pop()?; - let type_name = match value_type { - ValueType::Null => "null", - ValueType::Int => "int", - ValueType::Float => "float", - ValueType::Bool => "bool", - ValueType::String => "string", - ValueType::Bytes => "bytes", - ValueType::Array => "array", - ValueType::Map => "map", - ValueType::Callable => "callable", - ValueType::Unknown => { - return Err(TraceRecordError::UnsupportedTrace( - "type_of known specialization received unknown type".to_string(), - )); + } +} + +/// Spec-driven emit path for pilot builtins. +/// +/// Reads the declarative `BuiltinSpec` to pop inputs in order, apply +/// representation constraints, and emit the typed `SsaInstKind`. +/// Semantic special cases (e.g. ArraySet→ArrayPush append detection) +/// remain as explicit typed logic within this function. +fn emit_spec_driven_builtin( + builder: &mut SsaTraceBuilder, + block: super::ir::SsaBlockId, + ip: usize, + frame: &mut SymbolicFrame, + kind: SpecializedBuiltinKind, + spec: &builtin_spec::BuiltinSpec, +) -> Result<(&'static str, SymbolicValue), TraceRecordError> { + // Pop and constrain inputs in spec order (reverse of push order). + let mut popped: Vec = Vec::with_capacity(spec.arity); + for input_repr in spec.inputs { + let raw = frame.pop()?; + let constrained = match input_repr { + InputRepr::Int => ensure_int(builder, block, ip, raw)?, + InputRepr::HeapPtr(heap_kind) => { + ensure_heap_ptr(builder, block, ip, raw, heap_kind.value_type())? + } + InputRepr::Tagged => { + if raw.info.repr != SsaValueRepr::Tagged { + return Err(TraceRecordError::TypeMismatch { + expected: "owned tagged", + actual: raw.info.repr, + }); } - }; - let constant = Value::string(type_name); - let info = ValueInfo::type_name(); - let value = builder - .append_value_inst(block, ip, info.repr, SsaInstKind::Constant(constant)) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("type_of", SymbolicValue { value, info })) - } - SpecializedBuiltinKind::ToStringIdentity => { - let value = frame.pop()?; - Ok(("to_string_identity", value)) - } - SpecializedBuiltinKind::ToString => { - let value = frame.pop()?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::ToString { - value: value.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::String), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("to_string", out)) - } - SpecializedBuiltinKind::TypeOf => { - let value = frame.pop()?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::TypeOf { - value: value.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::type_name(), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("type_of", out)) - } - SpecializedBuiltinKind::StringSplitLiteral => { - let delimiter = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let text = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::String.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::StringSplitLiteral { - text: text.value.id, - delimiter: delimiter.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Array), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("string_split_literal", out)) - } - SpecializedBuiltinKind::StringConcat => { - let rhs = frame.pop()?; - let lhs = frame.pop()?; - emit_concat_binop(builder, block, ip, ConcatBinOpKind::String, lhs, rhs) - } - SpecializedBuiltinKind::BytesConcat => { - let rhs = frame.pop()?; - let lhs = frame.pop()?; - emit_concat_binop(builder, block, ip, ConcatBinOpKind::Bytes, lhs, rhs) - } - SpecializedBuiltinKind::BytesFromArrayU8 => { - let array = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::Array.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::BytesFromArrayU8 { - array: array.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Bytes), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("bytes_from_array_u8", out)) - } - SpecializedBuiltinKind::BytesToUtf8Ascii => { - let bytes = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::Bytes.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::BytesToUtf8Ascii { - bytes: bytes.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::String), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("bytes_to_utf8_ascii", out)) - } - SpecializedBuiltinKind::BytesToArrayU8 => { - let bytes = ensure_heap_ptr( - builder, - block, - ip, - frame.pop()?, - HeapContainerKind::Bytes.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::BytesToArrayU8 { - bytes: bytes.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Array), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("bytes_to_array_u8", out)) - } - SpecializedBuiltinKind::ArrayNew => { - let out = builder - .append_value_inst(block, ip, SsaValueRepr::Tagged, SsaInstKind::ArrayNew) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Array), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("array_new", out)) - } - SpecializedBuiltinKind::ArrayLen => { - let array = frame.pop()?; - let array = ensure_heap_ptr( - builder, - block, - ip, - array, - HeapContainerKind::Array.value_type(), - )?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::I64, - SsaInstKind::ArrayLen { - array: array.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::int(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("array_len", out)) - } - SpecializedBuiltinKind::ArrayGet => { - let index = frame.pop()?; - let array = frame.pop()?; - let array = ensure_heap_ptr( - builder, - block, - ip, - array, - HeapContainerKind::Array.value_type(), - )?; - let index = ensure_int(builder, block, ip, index)?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::ArrayGet { - array: array.value.id, - index: index.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged(), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("array_get", out)) - } - SpecializedBuiltinKind::ArrayHas => { - let index = frame.pop()?; - let array = frame.pop()?; - let array = ensure_heap_ptr( - builder, - block, - ip, - array, - HeapContainerKind::Array.value_type(), - )?; - let index = ensure_int(builder, block, ip, index)?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Bool, - SsaInstKind::ArrayHas { - array: array.value.id, - index: index.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::bool(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("array_has", out)) - } - SpecializedBuiltinKind::ArraySet => { - let value = frame.pop()?; - let index = ensure_int(builder, block, ip, frame.pop()?)?; - let array = frame.pop()?; - if array.info.repr != SsaValueRepr::Tagged { - return Err(TraceRecordError::TypeMismatch { - expected: "owned tagged array", - actual: array.info.repr, - }); + raw } + InputRepr::Any => raw, + }; + popped.push(constrained); + } + + // Build the typed SsaInstKind from the popped values. + let (inst_kind, out_info, emitted_name) = match kind { + SpecializedBuiltinKind::StringLen => ( + SsaInstKind::StringLen { + text: popped[0].value.id, + }, + ValueInfo::int(None), + spec.name, + ), + SpecializedBuiltinKind::RegexMatch => ( + SsaInstKind::RegexMatch { + pattern: popped[1].value.id, + text: popped[0].value.id, + }, + ValueInfo::bool(None), + spec.name, + ), + SpecializedBuiltinKind::ValueLen => ( + SsaInstKind::ValueLen { + value: popped[0].value.id, + }, + ValueInfo::int(None), + spec.name, + ), + SpecializedBuiltinKind::BytesLen => ( + SsaInstKind::BytesLen { + bytes: popped[0].value.id, + }, + ValueInfo::int(None), + spec.name, + ), + SpecializedBuiltinKind::ArrayLen => ( + SsaInstKind::ArrayLen { + array: popped[0].value.id, + }, + ValueInfo::int(None), + spec.name, + ), + SpecializedBuiltinKind::MapLen => ( + SsaInstKind::MapLen { + map: popped[0].value.id, + }, + ValueInfo::int(None), + spec.name, + ), + SpecializedBuiltinKind::TypeOf => ( + SsaInstKind::TypeOf { + value: popped[0].value.id, + }, + ValueInfo::type_name(), + spec.name, + ), + SpecializedBuiltinKind::StringContains => ( + SsaInstKind::StringContains { + text: popped[1].value.id, + needle: popped[0].value.id, + }, + ValueInfo::bool(None), + spec.name, + ), + SpecializedBuiltinKind::ArrayHas => ( + SsaInstKind::ArrayHas { + array: popped[1].value.id, + index: popped[0].value.id, + }, + ValueInfo::bool(None), + spec.name, + ), + SpecializedBuiltinKind::StringSlice => ( + SsaInstKind::StringSlice { + text: popped[2].value.id, + start: popped[1].value.id, + length: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::String), + spec.name, + ), + SpecializedBuiltinKind::BytesSlice => ( + SsaInstKind::BytesSlice { + bytes: popped[2].value.id, + start: popped[1].value.id, + length: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::Bytes), + spec.name, + ), + SpecializedBuiltinKind::StringGet => ( + SsaInstKind::StringGet { + text: popped[1].value.id, + index: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::String), + spec.name, + ), + SpecializedBuiltinKind::BytesGet => ( + SsaInstKind::BytesGet { + bytes: popped[1].value.id, + index: popped[0].value.id, + }, + ValueInfo::int(None), + spec.name, + ), + SpecializedBuiltinKind::BytesHas => ( + SsaInstKind::BytesHas { + bytes: popped[1].value.id, + index: popped[0].value.id, + }, + ValueInfo::bool(None), + spec.name, + ), + SpecializedBuiltinKind::StringReplaceLiteral => ( + SsaInstKind::StringReplaceLiteral { + text: popped[2].value.id, + needle: popped[1].value.id, + replacement: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::String), + spec.name, + ), + SpecializedBuiltinKind::StringLowerAscii => ( + SsaInstKind::StringLowerAscii { + text: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::String), + spec.name, + ), + SpecializedBuiltinKind::StringSplitLiteral => ( + SsaInstKind::StringSplitLiteral { + text: popped[1].value.id, + delimiter: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::Array), + spec.name, + ), + SpecializedBuiltinKind::BytesFromArrayU8 => ( + SsaInstKind::BytesFromArrayU8 { + array: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::Bytes), + spec.name, + ), + SpecializedBuiltinKind::BytesToUtf8Ascii => ( + SsaInstKind::BytesToUtf8Ascii { + bytes: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::String), + spec.name, + ), + SpecializedBuiltinKind::BytesToArrayU8 => ( + SsaInstKind::BytesToArrayU8 { + bytes: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::Array), + spec.name, + ), + SpecializedBuiltinKind::ToString => ( + SsaInstKind::ToString { + value: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::String), + spec.name, + ), + SpecializedBuiltinKind::RegexReplace => ( + SsaInstKind::RegexReplace { + pattern: popped[2].value.id, + text: popped[1].value.id, + replacement: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::String), + spec.name, + ), + SpecializedBuiltinKind::ArrayGet => ( + SsaInstKind::ArrayGet { + array: popped[1].value.id, + index: popped[0].value.id, + }, + ValueInfo::tagged(), + spec.name, + ), + SpecializedBuiltinKind::MapGet => ( + SsaInstKind::MapGet { + map: popped[1].value.id, + key: popped[0].value.id, + }, + ValueInfo::tagged(), + spec.name, + ), + SpecializedBuiltinKind::MapHas => ( + SsaInstKind::MapHas { + map: popped[1].value.id, + key: popped[0].value.id, + }, + ValueInfo::bool(None), + spec.name, + ), + SpecializedBuiltinKind::ArrayPush => ( + SsaInstKind::ArrayPush { + array: popped[1].value.id, + value: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::Array), + spec.name, + ), + SpecializedBuiltinKind::MapSet => ( + SsaInstKind::MapSet { + map: popped[2].value.id, + key: popped[1].value.id, + value: popped[0].value.id, + }, + ValueInfo::tagged_typed(ValueType::Map), + spec.name, + ), + SpecializedBuiltinKind::MapIterNext => ( + SsaInstKind::MapIterNext { + slot: popped[0].value.id, + }, + ValueInfo::bool(None), + spec.name, + ), + SpecializedBuiltinKind::MapIterTakeKey => ( + SsaInstKind::MapIterTakeKey { + slot: popped[0].value.id, + }, + ValueInfo::tagged(), + spec.name, + ), + SpecializedBuiltinKind::MapIterTakeValue => ( + SsaInstKind::MapIterTakeValue { + slot: popped[0].value.id, + }, + ValueInfo::tagged(), + spec.name, + ), + SpecializedBuiltinKind::ArraySet => { + let value = &popped[0]; + let index = &popped[1]; + let array = &popped[2]; + // Append-pattern detection: index == array.len() → ArrayPush. let is_append = matches!( builder.defining_inst(index.value.id).map(|inst| &inst.kind), Some(SsaInstKind::ArrayLen { array: array_ptr }) @@ -4246,7 +3851,7 @@ fn emit_specialized_builtin_call( }) if *input == array.value.id ) ); - let kind = if is_append { + let ssa_kind = if is_append { SsaInstKind::ArrayPush { array: array.value.id, value: value.value.id, @@ -4258,186 +3863,32 @@ fn emit_specialized_builtin_call( value: value.value.id, } }; - let out = builder - .append_value_inst(block, ip, SsaValueRepr::Tagged, kind) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Array), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok((if is_append { "array_push" } else { "array_set" }, out)) - } - SpecializedBuiltinKind::ArrayPush => { - let value = frame.pop()?; - let array = frame.pop()?; - if array.info.repr != SsaValueRepr::Tagged { - return Err(TraceRecordError::TypeMismatch { - expected: "owned tagged array", - actual: array.info.repr, - }); - } - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::ArrayPush { - array: array.value.id, - value: value.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Array), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("array_push", out)) - } - SpecializedBuiltinKind::MapLen => { - let map = frame.pop()?; - let map = - ensure_heap_ptr(builder, block, ip, map, HeapContainerKind::Map.value_type())?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::I64, - SsaInstKind::MapLen { map: map.value.id }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::int(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("map_len", out)) - } - SpecializedBuiltinKind::MapGet => { - let key = frame.pop()?; - let map = frame.pop()?; - let map = - ensure_heap_ptr(builder, block, ip, map, HeapContainerKind::Map.value_type())?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::MapGet { - map: map.value.id, - key: key.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged(), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("map_get", out)) - } - SpecializedBuiltinKind::MapHas => { - let key = frame.pop()?; - let map = frame.pop()?; - let map = - ensure_heap_ptr(builder, block, ip, map, HeapContainerKind::Map.value_type())?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Bool, - SsaInstKind::MapHas { - map: map.value.id, - key: key.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::bool(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("map_has", out)) - } - SpecializedBuiltinKind::MapSet => { - let value = frame.pop()?; - let key = frame.pop()?; - let map = frame.pop()?; - if map.info.repr != SsaValueRepr::Tagged { - return Err(TraceRecordError::TypeMismatch { - expected: "owned tagged map", - actual: map.info.repr, - }); - } - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::MapSet { - map: map.value.id, - key: key.value.id, - value: value.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged_typed(ValueType::Map), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("map_set", out)) - } - SpecializedBuiltinKind::MapIterNext => { - let slot = ensure_int(builder, block, ip, frame.pop()?)?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Bool, - SsaInstKind::MapIterNext { - slot: slot.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::bool(None), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("map_iter_next", out)) - } - SpecializedBuiltinKind::MapIterTakeKey => { - let slot = ensure_int(builder, block, ip, frame.pop()?)?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::MapIterTakeKey { - slot: slot.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged(), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("map_iter_take_key", out)) - } - SpecializedBuiltinKind::MapIterTakeValue => { - let slot = ensure_int(builder, block, ip, frame.pop()?)?; - let out = builder - .append_value_inst( - block, - ip, - SsaValueRepr::Tagged, - SsaInstKind::MapIterTakeValue { - slot: slot.value.id, - }, - ) - .map(|value| SymbolicValue { - value, - info: ValueInfo::tagged(), - }) - .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; - Ok(("map_iter_take_value", out)) + ( + ssa_kind, + ValueInfo::tagged_typed(ValueType::Array), + if is_append { "array_push" } else { spec.name }, + ) } - } + _ => unreachable!("non-spec builtin in spec-driven emit"), + }; + + // Debug assertion: pure builtins should not require failure exits. + // This consumes spec.effect and spec.needs_failure_exit in production code. + debug_assert!( + !spec.is_pure() || !spec.needs_failure_exit, + "pure builtin {} must not need failure exit", + spec.name + ); + + let out = builder + .append_value_inst(block, ip, spec.output.repr(), inst_kind) + .map(|value| SymbolicValue { + value, + info: out_info, + }) + .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; + + Ok((emitted_name, out)) } fn ensure_entry_repr( @@ -5300,6 +4751,25 @@ mod tests { .expect("unboxed integer binop validator should override stale hint"); } + #[test] + fn rejects_wide_frames_before_building_native_trace_state() { + let mut bc = BytecodeBuilder::new(); + bc.ldloc(0); + bc.ldc(0); + bc.add(); + bc.stloc(0); + bc.ret(); + let program = Program::new(vec![Value::Int(1)], bc.finish()).with_local_count(80); + + let error = + record_trace(&program, 0, 0, 32, &[]).expect_err("wide frame should be rejected"); + assert!(matches!( + error, + TraceRecordError::UnsupportedTrace(detail) + if detail == "trace frame has too many locals for profitable native execution" + )); + } + #[test] fn records_linear_local_increment_trace_directly() { let mut bc = BytecodeBuilder::new(); diff --git a/src/vm/jit/runtime.rs b/src/vm/jit/runtime.rs index 30168b21..751ef231 100644 --- a/src/vm/jit/runtime.rs +++ b/src/vm/jit/runtime.rs @@ -10,7 +10,7 @@ use super::super::{ExecOutcome, Vm, VmError, VmResult}; use super::JitTrace; use super::ir::SsaExitId; use super::trace::TraceExitKey; -use super::{JitMetrics, JitTraceTerminal, native}; +use super::{JitTraceTerminal, native}; use crate::vm::native::ROOT_FRAME_KEY; use std::collections::HashMap; use std::sync::{Arc, Mutex}; @@ -69,7 +69,7 @@ fn elapsed_ns(started: std::time::Instant) -> u64 { pub(crate) struct NativeTrace { _keepalive: Arc>, _direct_keepalives: Vec>>, - entry: NativeTraceEntry, + pub(super) entry: NativeTraceEntry, tail_entry: NativeTraceEntry, direct_slots: Arc>>, pub(super) code: Arc<[u8]>, @@ -77,21 +77,21 @@ pub(crate) struct NativeTrace { terminal: JitTraceTerminal, has_call: bool, has_yielding_call: bool, - lowering_kind: native::TraceLoweringKind, + pub(super) lowering_kind: native::TraceLoweringKind, interrupt_settings: Option, compile_profile: native::NativeCompileProfile, drop_contract_events_enabled: bool, - region: Option, + pub(super) region: Option, } -struct NativeRegion { +pub(super) struct NativeRegion { _keepalive: Arc>, - entry: NativeTraceEntry, - code: Arc<[u8]>, + pub(super) entry: NativeTraceEntry, + pub(super) code: Arc<[u8]>, terminal: JitTraceTerminal, has_call: bool, has_yielding_call: bool, - lowering_kind: native::TraceLoweringKind, + pub(super) lowering_kind: native::TraceLoweringKind, generation: u64, key: TraceExitKey, child_trace_id: usize, @@ -803,7 +803,7 @@ impl Vm { } pub fn jit_snapshot(&self) -> super::JitSnapshot { - self.jit.snapshot(self.jit_runtime_metrics()) + self.jit_diagnostics_snapshot() } pub fn jit_exit_profiles(&self) -> Vec { @@ -844,102 +844,7 @@ impl Vm { } pub fn dump_jit_info_with_machine_code(&self, include_machine_code: bool) -> String { - let mut out = self - .jit - .dump_text(self.program.debug.as_ref(), self.jit_runtime_metrics()); - out.push_str(&format!( - " native codegen backend: {}\n", - native::selected_codegen_backend() - )); - out.push_str(&format!( - " native trace executions: {}\n", - self.native_trace_exec_count - )); - out.push_str(&format!( - " native trace handoffs: {}\n", - self.jit_native_link_handoff_count - )); - out.push_str(&format!( - " native region entries: {}\n", - self.jit_native_region_entry_count - )); - out.push_str(&format!( - " native internal region edges: {}\n", - self.jit_native_region_edge_count - )); - out.push_str(&format!( - " native direct side links: {}\n", - self.jit_native_direct_link_count - )); - out.push_str(&format!( - " native compile time: {} ns (regions={} ns)\n", - self.jit_native_compile_time_ns, self.jit_native_region_compile_time_ns - )); - out.push_str(&format!( - " native code bytes: {} (regions={})\n", - self.jit_native_code_bytes(), - self.jit_native_region_code_bytes() - )); - if self.jit_native_bridge_stats_enabled { - let mut bridge_entries: Vec<(&'static str, u64)> = self - .jit_native_bridge_counts - .iter() - .map(|(name, count)| (*name, *count)) - .collect(); - bridge_entries.sort_unstable_by_key(|(name, _)| *name); - let total_bridge_hits = bridge_entries - .iter() - .fold(0u64, |acc, (_, count)| acc.saturating_add(*count)); - out.push_str(&format!( - " native bridge hits: {} (helpers={})\n", - total_bridge_hits, - bridge_entries.len() - )); - for (name, count) in bridge_entries { - out.push_str(&format!(" bridge {}: {}\n", name, count)); - } - } - let native_trace_count = self.native_traces.iter().flatten().count(); - if native_trace_count == 0 { - out.push_str(" native traces: 0\n"); - return out; - } - - out.push_str(&format!(" native traces: {}\n", native_trace_count)); - for (id, native) in self.native_traces.iter().enumerate() { - if let Some(native) = native { - out.push_str(&format!( - " native trace#{} entry=0x{:X} code_bytes={} lowering={}\n", - id, - native.entry as usize, - native.code.len(), - native.lowering_kind.as_str() - )); - if include_machine_code { - out.push_str(" code:"); - for byte in native.code.iter() { - out.push_str(&format!(" {:02X}", byte)); - } - out.push('\n'); - } - if let Some(region) = &native.region { - out.push_str(&format!( - " region entry=0x{:X} code_bytes={} lowering={}\n", - region.entry as usize, - region.code.len(), - region.lowering_kind.as_str() - )); - if include_machine_code { - out.push_str(" code:"); - for byte in region.code.iter() { - out.push_str(&format!(" {:02X}", byte)); - } - out.push('\n'); - } - } - } - } - out + self.jit_diagnostics_dump(include_machine_code) } pub(in crate::vm) fn execute_jit_entry(&mut self, trace_id: usize) -> VmResult { @@ -1658,23 +1563,6 @@ impl Vm { self.jit_native_link_handoff_count } - fn jit_runtime_metrics(&self) -> JitMetrics { - JitMetrics { - boxed_load_site_count: 0, - boxed_store_site_count: 0, - trace_exit_count: self.jit_trace_exit_count, - native_loop_back_count: self.jit_native_loop_back_count, - helper_fallback_count: self.jit_helper_fallback_count, - native_trace_exec_count: self.native_trace_exec_count, - script_call_observations: 0, - monomorphic_call_sites: 0, - polymorphic_call_sites: 0, - inline_attempts: 0, - inline_successes: 0, - inline_rejections: 0, - } - } - fn record_jit_helper_fallback(&mut self) { self.jit_helper_fallback_count = self.jit_helper_fallback_count.saturating_add(1); } diff --git a/src/vm/jit/trace.rs b/src/vm/jit/trace.rs index 43afd16f..d3a92072 100644 --- a/src/vm/jit/trace.rs +++ b/src/vm/jit/trace.rs @@ -11,7 +11,9 @@ use super::inline::{ }; use super::ir::{SsaExitId, SsaMaterialization, SsaTrace, SsaValueId, SsaValueRepr}; use super::liveness::{boxed_load_site_count, boxed_store_site_count}; -use super::recorder::{RecordedTrace, TraceRecordError, record_trace_with_local_count}; +use super::recorder::{ + RecordedTrace, TraceRecordError, WIDE_FRAME_REASON, record_trace_with_local_count, +}; #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] pub(crate) struct TraceEntryKey { @@ -1031,6 +1033,12 @@ impl TraceJitEngine { Some(trace_id) } Err(reason) => { + if matches!( + &reason, + JitNyiReason::UnsupportedTrace(detail) if detail == WIDE_FRAME_REASON + ) { + self.config.enabled = false; + } self.attempts.push(JitAttempt { frame_key: key.frame_key, root_ip: key.root_ip, @@ -1858,6 +1866,32 @@ mod tests { assert!(engine.trace_exit_profiles.is_empty()); } + #[test] + fn wide_frame_rejection_disables_subsequent_trace_polling() { + let mut bc = BytecodeBuilder::new(); + bc.ldloc(0); + bc.ldc(0); + bc.add(); + bc.stloc(0); + bc.br(0); + let program = Program::new(vec![Value::Int(1)], bc.finish()).with_local_count(80); + let mut engine = TraceJitEngine::new(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 32, + }); + + assert_eq!( + engine.observe_hot_entry(ROOT_FRAME_KEY, 0, 0, &program), + None + ); + assert_eq!( + engine.observe_hot_entry(ROOT_FRAME_KEY, 0, 0, &program), + None + ); + assert!(!engine.config.enabled); + } + #[test] fn cached_inline_trace_requires_matching_entry_callable_prototype() { if !native_jit_supported() { diff --git a/src/vm/mod.rs b/src/vm/mod.rs index 150da557..ba148aec 100644 --- a/src/vm/mod.rs +++ b/src/vm/mod.rs @@ -944,7 +944,10 @@ impl Vm { self.owned_callables.clear(); self.draining_queued_callables = false; self.shutdown = false; - self.aot_interpreter_boundary_hit = false; + self.aot_interpreter_boundary_hit = self + .aot_program + .as_ref() + .is_some_and(|program| program.interpreter_boundary_only); self.waiting_host_op = None; self.io_state = crate::builtins::runtime::IoState::default(); self.map_iterators.clear(); @@ -2321,7 +2324,9 @@ impl Vm { } if debugger.is_none() && !self.interruption_enabled() - && (!allow_jit || (!self.jit_config().enabled && !self.has_aot_program())) + && (!allow_jit + || (!self.jit_config().enabled + && (!self.has_aot_program() || self.aot_interpreter_boundary_hit))) && let Some(status) = self.run_fast_interpreter(allow_jit)? { return Ok(status); @@ -2364,6 +2369,15 @@ impl Vm { continue; } + if self.aot_interpreter_boundary_hit + && debugger.is_none() + && !self.interruption_enabled() + && !self.jit_config().enabled + && let Some(status) = self.run_fast_interpreter(false)? + { + return Ok(status); + } + if allow_jit && self.jit_config().enabled && self.builtin_overrides.is_empty() diff --git a/src/vm/native/inline.rs b/src/vm/native/inline.rs deleted file mode 100644 index b7e9b275..00000000 --- a/src/vm/native/inline.rs +++ /dev/null @@ -1,3613 +0,0 @@ -#[cfg(feature = "cranelift-jit")] -use crate::builtins::BuiltinFunction; -#[cfg(feature = "cranelift-jit")] -use crate::vm::{VmError, VmResult}; -#[cfg(feature = "cranelift-jit")] -use cranelift_codegen::ir::condcodes::{FloatCC, IntCC}; -#[cfg(feature = "cranelift-jit")] -use cranelift_codegen::ir::{Block, BlockArg, InstBuilder, MemFlags, SigRef, types}; -#[cfg(feature = "cranelift-jit")] -use cranelift_frontend::FunctionBuilder; - -#[cfg(feature = "cranelift-jit")] -use super::{ - NativeStackLayout, OP_ADD, OP_AND, OP_BUILTIN_CALL, OP_CEQ, OP_CGT, OP_CLT, OP_DIV, OP_DUP, - OP_LDC, OP_LDLOC, OP_LSHR, OP_MOD, OP_MUL, OP_NEG, OP_NOT, OP_OR, OP_POP, OP_SHL, OP_SHR, - OP_STLOC, OP_SUB, STATUS_CONTINUE, ValueLayout, checked_add_i32, helper_entry_offset, -}; - -#[cfg(feature = "cranelift-jit")] -#[derive(Clone, Copy)] -pub(crate) struct ResolvedOffsets { - pub(crate) stack_ptr: i32, - pub(crate) stack_len: i32, - pub(crate) stack_cap: i32, - pub(crate) locals_ptr: i32, - pub(crate) locals_len: i32, - pub(crate) constants_ptr: i32, - pub(crate) constants_len: i32, - pub(crate) vm_ip: i32, - pub(crate) drop_contract_events_enabled: i32, - pub(crate) drop_contract_events: i32, -} - -#[cfg(feature = "cranelift-jit")] -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(crate) enum NativeInlineStep { - Ldc(u32), - Ldloc(u8), - Stloc(u8), - Pop, - Dup, - Add, - IAdd, - FAdd, - StringConcat, - BytesConcat, - StringLen, - BytesLen, - StringSlice, - BytesSlice, - StringGet, - BytesGet, - BytesHas, - BytesFromArrayU8, - BytesToArrayU8, - Sub, - ISub, - FSub, - Mul, - IMul, - FMul, - Div, - IDiv, - FDiv, - Mod, - IMod, - FMod, - Shl, - Shr, - Lshr, - And, - Or, - Not, - Neg, - INeg, - FNeg, - Ceq, - FCeq, - Clt, - FClt, - Cgt, - FCgt, -} - -#[cfg(feature = "cranelift-jit")] -#[derive(Clone, Copy)] -pub(crate) struct HeapIntrinsicRefs { - pub(crate) alloc_buffer_ref: SigRef, - pub(crate) free_buffer_ref: SigRef, - pub(crate) pack_shared_ref: SigRef, - pub(crate) drop_shared_ref: SigRef, - pub(crate) copy_bytes_ref: SigRef, -} - -#[cfg(feature = "cranelift-jit")] -#[derive(Clone, Copy)] -pub(crate) struct HeapIntrinsicAddrs { - pub(crate) alloc_byte_buffer: usize, - pub(crate) alloc_value_buffer: usize, - pub(crate) pack_string: usize, - pub(crate) pack_bytes: usize, - pub(crate) pack_array: usize, - pub(crate) copy_bytes: usize, - pub(crate) zero_bytes: usize, - pub(crate) drop_string: usize, - pub(crate) drop_bytes: usize, - pub(crate) drop_array: usize, -} - -#[cfg(feature = "cranelift-jit")] -#[derive(Clone, Copy)] -pub(crate) struct InlineEmitCtx { - pub(crate) vm_ptr: cranelift_codegen::ir::Value, - pub(crate) helper_ref: SigRef, - pub(crate) _vm_status_helper_ref: SigRef, - pub(crate) exit_block: Block, - pub(crate) pointer_type: cranelift_codegen::ir::Type, - pub(crate) layout: NativeStackLayout, - pub(crate) offsets: ResolvedOffsets, - pub(crate) heap_refs: HeapIntrinsicRefs, - pub(crate) heap_addrs: HeapIntrinsicAddrs, -} - -#[cfg(feature = "cranelift-jit")] -enum IntBinopKind { - Add, - Sub, - Mul, -} - -#[cfg(feature = "cranelift-jit")] -enum FloatBinopKind { - Add, - Sub, - Mul, - Div, - Mod, -} - -#[cfg(feature = "cranelift-jit")] -#[derive(Clone, Copy)] -enum ShiftKind { - Left, - ArithmeticRight, - LogicalRight, -} - -#[cfg(feature = "cranelift-jit")] -pub(crate) fn resolve_offsets(layout: NativeStackLayout) -> VmResult { - let stack_ptr = checked_add_i32( - layout.vm_stack_offset, - layout.stack_vec.ptr_offset, - "stack ptr offset overflow", - )?; - let stack_len = checked_add_i32( - layout.vm_stack_offset, - layout.stack_vec.len_offset, - "stack len offset overflow", - )?; - let stack_cap = checked_add_i32( - layout.vm_stack_offset, - layout.stack_vec.cap_offset, - "stack cap offset overflow", - )?; - let locals_ptr = checked_add_i32( - layout.vm_locals_offset, - layout.stack_vec.ptr_offset, - "locals ptr offset overflow", - )?; - let locals_len = checked_add_i32( - layout.vm_locals_offset, - layout.stack_vec.len_offset, - "locals len offset overflow", - )?; - - Ok(ResolvedOffsets { - stack_ptr, - stack_len, - stack_cap, - locals_ptr, - locals_len, - constants_ptr: layout.vm_program_constants_ptr_offset, - constants_len: layout.vm_program_constants_len_offset, - vm_ip: layout.vm_ip_offset, - drop_contract_events_enabled: layout.vm_drop_contract_events_enabled_offset, - drop_contract_events: layout.vm_drop_contract_events_offset, - }) -} - -#[cfg(feature = "cranelift-jit")] -pub(crate) fn emit_native_inline_step( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - step: NativeInlineStep, -) -> VmResult<()> { - match step { - NativeInlineStep::Ldc(index) => emit_inline_ldc(b, ctx, index, step_ip), - NativeInlineStep::Ldloc(index) => emit_inline_ldloc(b, ctx, index, step_ip), - NativeInlineStep::Stloc(index) => emit_inline_stloc(b, ctx, index, step_ip), - NativeInlineStep::Pop => emit_inline_pop(b, ctx, step_ip), - NativeInlineStep::Dup => emit_inline_dup(b, ctx, step_ip), - NativeInlineStep::Add | NativeInlineStep::IAdd => { - emit_inline_int_binop(b, ctx, step_ip, IntBinopKind::Add) - } - NativeInlineStep::FAdd => emit_inline_float_binop(b, ctx, step_ip, FloatBinopKind::Add), - NativeInlineStep::StringConcat => emit_inline_string_concat(b, ctx, step_ip), - NativeInlineStep::BytesConcat => emit_inline_bytes_concat(b, ctx, step_ip), - NativeInlineStep::StringLen => emit_inline_string_len(b, ctx, step_ip), - NativeInlineStep::BytesLen => emit_inline_bytes_len(b, ctx, step_ip), - NativeInlineStep::StringSlice => emit_inline_string_slice(b, ctx, step_ip), - NativeInlineStep::BytesSlice => emit_inline_bytes_slice(b, ctx, step_ip), - NativeInlineStep::StringGet => emit_inline_string_get(b, ctx, step_ip), - NativeInlineStep::BytesGet => emit_inline_bytes_get(b, ctx, step_ip), - NativeInlineStep::BytesHas => emit_inline_bytes_has(b, ctx, step_ip), - NativeInlineStep::BytesFromArrayU8 => emit_inline_bytes_from_array_u8(b, ctx, step_ip), - NativeInlineStep::BytesToArrayU8 => emit_inline_bytes_to_array_u8(b, ctx, step_ip), - NativeInlineStep::Sub | NativeInlineStep::ISub => { - emit_inline_int_binop(b, ctx, step_ip, IntBinopKind::Sub) - } - NativeInlineStep::FSub => emit_inline_float_binop(b, ctx, step_ip, FloatBinopKind::Sub), - NativeInlineStep::Mul | NativeInlineStep::IMul => { - emit_inline_int_binop(b, ctx, step_ip, IntBinopKind::Mul) - } - NativeInlineStep::FMul => emit_inline_float_binop(b, ctx, step_ip, FloatBinopKind::Mul), - NativeInlineStep::Div | NativeInlineStep::IDiv => { - emit_inline_int_divrem(b, ctx, step_ip, false) - } - NativeInlineStep::FDiv => emit_inline_float_binop(b, ctx, step_ip, FloatBinopKind::Div), - NativeInlineStep::Mod | NativeInlineStep::IMod => { - emit_inline_int_divrem(b, ctx, step_ip, true) - } - NativeInlineStep::FMod => emit_inline_float_binop(b, ctx, step_ip, FloatBinopKind::Mod), - NativeInlineStep::Shl => emit_inline_shift(b, ctx, step_ip, ShiftKind::Left), - NativeInlineStep::Shr => emit_inline_shift(b, ctx, step_ip, ShiftKind::ArithmeticRight), - NativeInlineStep::Lshr => emit_inline_shift(b, ctx, step_ip, ShiftKind::LogicalRight), - NativeInlineStep::And => emit_inline_bool_logic(b, ctx, step_ip, true), - NativeInlineStep::Or => emit_inline_bool_logic(b, ctx, step_ip, false), - NativeInlineStep::Not => emit_inline_not(b, ctx, step_ip), - NativeInlineStep::Neg | NativeInlineStep::INeg => emit_inline_neg(b, ctx, step_ip), - NativeInlineStep::FNeg => emit_inline_float_neg(b, ctx, step_ip), - NativeInlineStep::Ceq => emit_inline_int_eq(b, ctx, step_ip), - NativeInlineStep::FCeq => emit_inline_float_eq(b, ctx, step_ip), - NativeInlineStep::Clt => emit_inline_int_compare(b, ctx, step_ip, true), - NativeInlineStep::FClt => emit_inline_float_compare(b, ctx, step_ip, true), - NativeInlineStep::Cgt => emit_inline_int_compare(b, ctx, step_ip, false), - NativeInlineStep::FCgt => emit_inline_float_compare(b, ctx, step_ip, false), - } -} - -#[cfg(feature = "cranelift-jit")] -#[allow(clippy::too_many_arguments)] -fn emit_helper_step_from_call_tuple( - b: &mut FunctionBuilder, - vm_ptr: cranelift_codegen::ir::Value, - helper_ref: SigRef, - exit_block: Block, - next_block: Block, - offsets: ResolvedOffsets, - step_ip: usize, - tuple: (i64, i64, i64, i64), -) { - let (op, a, b_arg, c) = tuple; - let op_val = b.ins().iconst(types::I64, op); - let a_val = b.ins().iconst(types::I64, a); - let b_val = b.ins().iconst(types::I64, b_arg); - let c_val = b.ins().iconst(types::I64, c); - let pointer_type = b.func.signature.params[0].value_type; - let step_ip = i64::try_from(step_ip).expect("step ip must fit i64"); - let step_ip_val = b.ins().iconst(pointer_type, step_ip); - b.ins() - .store(MemFlags::new(), step_ip_val, vm_ptr, offsets.vm_ip); - let helper_ptr = b - .ins() - .load(pointer_type, MemFlags::new(), vm_ptr, helper_entry_offset()); - let call = b.ins().call_indirect( - helper_ref, - helper_ptr, - &[vm_ptr, op_val, a_val, b_val, c_val], - ); - let status = b.inst_results(call)[0]; - let is_continue = b - .ins() - .icmp_imm(IntCC::Equal, status, STATUS_CONTINUE as i64); - let else_args = [BlockArg::Value(status)]; - b.ins() - .brif(is_continue, next_block, &[], exit_block, &else_args); -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_ldc( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - index: u32, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let constants_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.constants_len, - ); - let idx = b.ins().iconst(ctx.pointer_type, i64::from(index)); - let in_bounds = b.ins().icmp(IntCC::UnsignedLessThan, idx, constants_len); - b.ins().brif(in_bounds, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let stack_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let stack_cap = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_cap, - ); - let has_capacity = b.ins().icmp(IntCC::UnsignedLessThan, stack_len, stack_cap); - let cap_ok = b.create_block(); - b.ins().brif(has_capacity, cap_ok, &[], slow, &[]); - - b.switch_to_block(cap_ok); - let constants_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.constants_ptr, - ); - let src_addr = value_addr( - b, - ctx.pointer_type, - constants_ptr, - idx, - ctx.layout.value.size, - ); - let src_tag = load_tag_i32(b, ctx.layout.value, src_addr); - let scalar = is_scalar_tag(b, ctx.layout.value, src_tag); - let scalar_ok = b.create_block(); - b.ins().brif(scalar, scalar_ok, &[], slow, &[]); - - b.switch_to_block(scalar_ok); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let dst_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - stack_len, - ctx.layout.value.size, - ); - copy_value_bytes(b, src_addr, dst_addr, ctx.layout.value.size); - let one = b.ins().iconst(ctx.pointer_type, 1); - let new_len = b.ins().iadd(stack_len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_LDC, i64::from(index), 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_ldloc( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - index: u8, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let locals_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.locals_len, - ); - let idx = b.ins().iconst(ctx.pointer_type, i64::from(index)); - let in_bounds = b.ins().icmp(IntCC::UnsignedLessThan, idx, locals_len); - b.ins().brif(in_bounds, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let stack_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let stack_cap = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_cap, - ); - let has_capacity = b.ins().icmp(IntCC::UnsignedLessThan, stack_len, stack_cap); - let cap_ok = b.create_block(); - b.ins().brif(has_capacity, cap_ok, &[], slow, &[]); - - b.switch_to_block(cap_ok); - let locals_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.locals_ptr, - ); - let src_addr = value_addr(b, ctx.pointer_type, locals_ptr, idx, ctx.layout.value.size); - let src_tag = load_tag_i32(b, ctx.layout.value, src_addr); - let scalar = is_scalar_tag(b, ctx.layout.value, src_tag); - let scalar_ok = b.create_block(); - b.ins().brif(scalar, scalar_ok, &[], slow, &[]); - - b.switch_to_block(scalar_ok); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let dst_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - stack_len, - ctx.layout.value.size, - ); - copy_value_bytes(b, src_addr, dst_addr, ctx.layout.value.size); - let one = b.ins().iconst(ctx.pointer_type, 1); - let new_len = b.ins().iadd(stack_len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_LDLOC, i64::from(index), 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_stloc( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - index: u8, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let stack_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let has_stack = b - .ins() - .icmp_imm(IntCC::UnsignedGreaterThanOrEqual, stack_len, 1); - b.ins().brif(has_stack, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let locals_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.locals_len, - ); - let idx = b.ins().iconst(ctx.pointer_type, i64::from(index)); - let in_bounds = b.ins().icmp(IntCC::UnsignedLessThan, idx, locals_len); - let bounds_ok = b.create_block(); - b.ins().brif(in_bounds, bounds_ok, &[], slow, &[]); - - b.switch_to_block(bounds_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let src_index = b.ins().isub(stack_len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let src_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - src_index, - ctx.layout.value.size, - ); - let locals_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.locals_ptr, - ); - let dst_addr = value_addr(b, ctx.pointer_type, locals_ptr, idx, ctx.layout.value.size); - let dst_tag = load_tag_i32(b, ctx.layout.value, dst_addr); - let dst_scalar = is_scalar_tag(b, ctx.layout.value, dst_tag); - let scalar_ok = b.create_block(); - b.ins().brif(dst_scalar, scalar_ok, &[], slow, &[]); - - b.switch_to_block(scalar_ok); - let dst_is_null = b - .ins() - .icmp_imm(IntCC::Equal, dst_tag, i64::from(ctx.layout.value.null_tag)); - let copy_block = b.create_block(); - let maybe_count_drop_block = b.create_block(); - b.ins() - .brif(dst_is_null, copy_block, &[], maybe_count_drop_block, &[]); - - b.switch_to_block(maybe_count_drop_block); - let drop_events_enabled = b.ins().load( - types::I8, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.drop_contract_events_enabled, - ); - let should_count_drop = b.ins().icmp_imm(IntCC::NotEqual, drop_events_enabled, 0); - let count_drop_block = b.create_block(); - b.ins() - .brif(should_count_drop, count_drop_block, &[], copy_block, &[]); - - b.switch_to_block(count_drop_block); - let drop_count = b.ins().load( - types::I64, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.drop_contract_events, - ); - let next_drop_count = b.ins().iadd_imm(drop_count, 1); - b.ins().store( - MemFlags::new(), - next_drop_count, - ctx.vm_ptr, - ctx.offsets.drop_contract_events, - ); - b.ins().jump(copy_block, &[]); - - b.switch_to_block(copy_block); - copy_value_bytes(b, src_addr, dst_addr, ctx.layout.value.size); - let new_len = b.ins().isub(stack_len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_STLOC, i64::from(index), 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_pop(b: &mut FunctionBuilder, ctx: InlineEmitCtx, step_ip: usize) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let has_stack = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(has_stack, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let one = b.ins().iconst(ctx.pointer_type, 1); - let top_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let top_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - top_index, - ctx.layout.value.size, - ); - let top_tag = load_tag_i32(b, ctx.layout.value, top_addr); - let scalar = is_scalar_tag(b, ctx.layout.value, top_tag); - let scalar_ok = b.create_block(); - b.ins().brif(scalar, scalar_ok, &[], slow, &[]); - - b.switch_to_block(scalar_ok); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_POP, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_dup(b: &mut FunctionBuilder, ctx: InlineEmitCtx, step_ip: usize) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let has_stack = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(has_stack, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let cap = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_cap, - ); - let has_capacity = b.ins().icmp(IntCC::UnsignedLessThan, len, cap); - let cap_ok = b.create_block(); - b.ins().brif(has_capacity, cap_ok, &[], slow, &[]); - - b.switch_to_block(cap_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let src_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let src_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - src_index, - ctx.layout.value.size, - ); - let src_tag = load_tag_i32(b, ctx.layout.value, src_addr); - let scalar = is_scalar_tag(b, ctx.layout.value, src_tag); - let scalar_ok = b.create_block(); - b.ins().brif(scalar, scalar_ok, &[], slow, &[]); - - b.switch_to_block(scalar_ok); - let dst_addr = value_addr(b, ctx.pointer_type, stack_ptr, len, ctx.layout.value.size); - copy_value_bytes(b, src_addr, dst_addr, ctx.layout.value.size); - let new_len = b.ins().iadd(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_DUP, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_int_binop( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - kind: IntBinopKind, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_int = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.int_tag)); - let rhs_int = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.int_tag)); - let both_int = b.ins().band(lhs_int, rhs_int); - b.ins().brif(both_int, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let lhs = b.ins().load( - types::I64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let rhs = b.ins().load( - types::I64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.int_payload_offset, - ); - let out = match kind { - IntBinopKind::Add => b.ins().iadd(lhs, rhs), - IntBinopKind::Sub => b.ins().isub(lhs, rhs), - IntBinopKind::Mul => b.ins().imul(lhs, rhs), - }; - b.ins().store( - MemFlags::new(), - out, - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - let op = match kind { - IntBinopKind::Add => OP_ADD, - IntBinopKind::Sub => OP_SUB, - IntBinopKind::Mul => OP_MUL, - }; - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (op, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_float_binop( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - kind: FloatBinopKind, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_float = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.float_tag)); - let rhs_float = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.float_tag)); - let both_float = b.ins().band(lhs_float, rhs_float); - b.ins().brif(both_float, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let lhs = b.ins().load( - types::F64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.float_payload_offset, - ); - let rhs = b.ins().load( - types::F64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.float_payload_offset, - ); - let out = match kind { - FloatBinopKind::Add => b.ins().fadd(lhs, rhs), - FloatBinopKind::Sub => b.ins().fsub(lhs, rhs), - FloatBinopKind::Mul => b.ins().fmul(lhs, rhs), - FloatBinopKind::Div => b.ins().fdiv(lhs, rhs), - FloatBinopKind::Mod => { - let quotient = b.ins().fdiv(lhs, rhs); - let truncated = b.ins().trunc(quotient); - let product = b.ins().fmul(truncated, rhs); - b.ins().fsub(lhs, product) - } - }; - b.ins().store( - MemFlags::new(), - out, - lhs_addr, - ctx.layout.value.float_payload_offset, - ); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - let op = match kind { - FloatBinopKind::Add => OP_ADD, - FloatBinopKind::Sub => OP_SUB, - FloatBinopKind::Mul => OP_MUL, - FloatBinopKind::Div => OP_DIV, - FloatBinopKind::Mod => OP_MOD, - }; - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (op, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -#[allow(clippy::too_many_arguments)] -fn emit_inline_concat( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - operand_tag: u32, - result_tag: u32, - pack_addr: usize, - drop_addr: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let lhs_index = b.ins().isub(len, two); - let rhs_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_ok = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(operand_tag)); - let rhs_ok = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(operand_tag)); - let both_ok = b.ins().band(lhs_ok, rhs_ok); - b.ins().brif(both_ok, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let lhs_raw = load_heap_ptr(b, ctx.layout.value, lhs_addr, ctx.pointer_type); - let rhs_raw = load_heap_ptr(b, ctx.layout.value, rhs_addr, ctx.pointer_type); - let lhs_data = load_heap_data_ptr(b, ctx.layout.value, lhs_raw); - let rhs_data = load_heap_data_ptr(b, ctx.layout.value, rhs_raw); - let lhs_bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - lhs_data, - ctx.layout.stack_vec.ptr_offset, - ); - let lhs_bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - lhs_data, - ctx.layout.stack_vec.len_offset, - ); - let rhs_bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - rhs_data, - ctx.layout.stack_vec.ptr_offset, - ); - let rhs_bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - rhs_data, - ctx.layout.stack_vec.len_offset, - ); - let max_usize = b.ins().iconst(ctx.pointer_type, -1); - let remaining = b.ins().isub(max_usize, lhs_bytes_len); - let overflow = b - .ins() - .icmp(IntCC::UnsignedGreaterThan, rhs_bytes_len, remaining); - let add_ok = b.create_block(); - b.ins().brif(overflow, slow, &[], add_ok, &[]); - - b.switch_to_block(add_ok); - let total_len = b.ins().iadd(lhs_bytes_len, rhs_bytes_len); - let exceeds_isize = b - .ins() - .icmp_imm(IntCC::UnsignedGreaterThan, total_len, i64::MAX); - let cap_ok = b.create_block(); - b.ins().brif(exceeds_isize, slow, &[], cap_ok, &[]); - - b.switch_to_block(cap_ok); - let out_ptr = call_alloc_buffer(b, ctx, ctx.heap_addrs.alloc_byte_buffer, total_len)?; - call_copy_bytes(b, ctx, out_ptr, lhs_bytes_ptr, lhs_bytes_len)?; - let rhs_dst = b.ins().iadd(out_ptr, lhs_bytes_len); - call_copy_bytes(b, ctx, rhs_dst, rhs_bytes_ptr, rhs_bytes_len)?; - let out_raw = call_pack_shared(b, ctx, pack_addr, out_ptr, total_len, total_len)?; - call_drop_shared(b, ctx, drop_addr, rhs_raw)?; - call_drop_shared(b, ctx, drop_addr, lhs_raw)?; - store_heap_ptr_in_value(b, ctx.layout.value, lhs_addr, result_tag, out_raw); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_ADD, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_string_concat( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - emit_inline_concat( - b, - ctx, - step_ip, - ctx.layout.value.string_tag, - ctx.layout.value.string_tag, - ctx.heap_addrs.pack_string, - ctx.heap_addrs.drop_string, - ) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bytes_concat( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - emit_inline_concat( - b, - ctx, - step_ip, - ctx.layout.value.bytes_tag, - ctx.layout.value.bytes_tag, - ctx.heap_addrs.pack_bytes, - ctx.heap_addrs.drop_bytes, - ) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_string_len( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let loop_block = b.create_block(); - let exit = b.create_block(); - let next = b.create_block(); - b.append_block_param(loop_block, ctx.pointer_type); - b.append_block_param(loop_block, ctx.pointer_type); - b.append_block_param(exit, ctx.pointer_type); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(enough, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let one = b.ins().iconst(ctx.pointer_type, 1); - let index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let slot_addr = value_addr(b, ctx.pointer_type, stack_ptr, index, ctx.layout.value.size); - let tag = load_tag_i32(b, ctx.layout.value, slot_addr); - let is_string = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(ctx.layout.value.string_tag)); - let type_ok = b.create_block(); - b.ins().brif(is_string, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let string_raw = load_heap_ptr(b, ctx.layout.value, slot_addr, ctx.pointer_type); - let string_data = load_heap_data_ptr(b, ctx.layout.value, string_raw); - let bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - string_data, - ctx.layout.stack_vec.ptr_offset, - ); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - string_data, - ctx.layout.stack_vec.len_offset, - ); - let zero = b.ins().iconst(ctx.pointer_type, 0); - b.ins() - .jump(loop_block, &[BlockArg::Value(zero), BlockArg::Value(zero)]); - - b.switch_to_block(loop_block); - let byte_index = b.block_params(loop_block)[0]; - let char_count = b.block_params(loop_block)[1]; - let done = b - .ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, byte_index, bytes_len); - let step_block = b.create_block(); - b.ins() - .brif(done, exit, &[BlockArg::Value(char_count)], step_block, &[]); - - b.switch_to_block(step_block); - let byte_ptr = b.ins().iadd(bytes_ptr, byte_index); - let byte = load_byte(b, byte_ptr); - let cont = is_utf8_continuation_byte(b, byte); - let advanced_count = b.ins().iadd_imm(char_count, 1); - let next_count = b.ins().select(cont, char_count, advanced_count); - let next_index = b.ins().iadd_imm(byte_index, 1); - b.ins().jump( - loop_block, - &[BlockArg::Value(next_index), BlockArg::Value(next_count)], - ); - - b.switch_to_block(exit); - let count = b.block_params(exit)[0]; - call_drop_shared(b, ctx, ctx.heap_addrs.drop_string, string_raw)?; - store_int_in_value(b, ctx.layout.value, slot_addr, count); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::Len, 1), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bytes_len( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(enough, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let one = b.ins().iconst(ctx.pointer_type, 1); - let index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let slot_addr = value_addr(b, ctx.pointer_type, stack_ptr, index, ctx.layout.value.size); - let tag = load_tag_i32(b, ctx.layout.value, slot_addr); - let is_bytes = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(ctx.layout.value.bytes_tag)); - let type_ok = b.create_block(); - b.ins().brif(is_bytes, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let bytes_raw = load_heap_ptr(b, ctx.layout.value, slot_addr, ctx.pointer_type); - let bytes_data = load_heap_data_ptr(b, ctx.layout.value, bytes_raw); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.len_offset, - ); - call_drop_shared(b, ctx, ctx.heap_addrs.drop_bytes, bytes_raw)?; - store_int_in_value(b, ctx.layout.value, slot_addr, bytes_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::Len, 1), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bytes_slice( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let copy_block = b.create_block(); - let next = b.create_block(); - b.append_block_param(copy_block, ctx.pointer_type); - b.append_block_param(copy_block, ctx.pointer_type); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 3); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let three = b.ins().iconst(ctx.pointer_type, 3); - let source_index = b.ins().isub(len, three); - let start_index = b.ins().isub(len, two); - let length_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let source_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - source_index, - ctx.layout.value.size, - ); - let start_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - start_index, - ctx.layout.value.size, - ); - let length_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - length_index, - ctx.layout.value.size, - ); - let source_tag = load_tag_i32(b, ctx.layout.value, source_addr); - let start_tag = load_tag_i32(b, ctx.layout.value, start_addr); - let length_tag = load_tag_i32(b, ctx.layout.value, length_addr); - let source_ok = b.ins().icmp_imm( - IntCC::Equal, - source_tag, - i64::from(ctx.layout.value.bytes_tag), - ); - let start_ok = b - .ins() - .icmp_imm(IntCC::Equal, start_tag, i64::from(ctx.layout.value.int_tag)); - let length_ok_tag = b.ins().icmp_imm( - IntCC::Equal, - length_tag, - i64::from(ctx.layout.value.int_tag), - ); - let both_ints = b.ins().band(start_ok, length_ok_tag); - let all_ok = b.ins().band(source_ok, both_ints); - b.ins().brif(all_ok, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let bytes_raw = load_heap_ptr(b, ctx.layout.value, source_addr, ctx.pointer_type); - let bytes_data = load_heap_data_ptr(b, ctx.layout.value, bytes_raw); - let bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.ptr_offset, - ); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.len_offset, - ); - let start = b.ins().load( - types::I64, - MemFlags::new(), - start_addr, - ctx.layout.value.int_payload_offset, - ); - let slice_len_value = b.ins().load( - types::I64, - MemFlags::new(), - length_addr, - ctx.layout.value.int_payload_offset, - ); - let start_negative = b.ins().icmp_imm(IntCC::SignedLessThan, start, 0); - let length_positive = b - .ins() - .icmp_imm(IntCC::SignedGreaterThan, slice_len_value, 0); - let start_non_negative = b.ins().bnot(start_negative); - let positive = b.ins().band(start_non_negative, length_positive); - let range_block = b.create_block(); - let empty_block = b.create_block(); - b.ins().brif(positive, range_block, &[], empty_block, &[]); - - b.switch_to_block(range_block); - let start_in_bounds = b.ins().icmp(IntCC::UnsignedLessThan, start, bytes_len); - let in_bounds_block = b.create_block(); - b.ins() - .brif(start_in_bounds, in_bounds_block, &[], empty_block, &[]); - - b.switch_to_block(in_bounds_block); - let available = b.ins().isub(bytes_len, start); - let take_full = b - .ins() - .icmp(IntCC::UnsignedGreaterThan, slice_len_value, available); - let actual_len = b.ins().select(take_full, available, slice_len_value); - let slice_ptr = b.ins().iadd(bytes_ptr, start); - b.ins().jump( - copy_block, - &[BlockArg::Value(slice_ptr), BlockArg::Value(actual_len)], - ); - - b.switch_to_block(empty_block); - let zero = b.ins().iconst(ctx.pointer_type, 0); - b.ins().jump( - copy_block, - &[BlockArg::Value(bytes_ptr), BlockArg::Value(zero)], - ); - - b.switch_to_block(copy_block); - let slice_ptr = b.block_params(copy_block)[0]; - let actual_len = b.block_params(copy_block)[1]; - let out_ptr = call_alloc_buffer(b, ctx, ctx.heap_addrs.alloc_byte_buffer, actual_len)?; - call_copy_bytes(b, ctx, out_ptr, slice_ptr, actual_len)?; - let out_raw = call_pack_shared( - b, - ctx, - ctx.heap_addrs.pack_bytes, - out_ptr, - actual_len, - actual_len, - )?; - call_drop_shared(b, ctx, ctx.heap_addrs.drop_bytes, bytes_raw)?; - store_heap_ptr_in_value( - b, - ctx.layout.value, - source_addr, - ctx.layout.value.bytes_tag, - out_raw, - ); - let new_len = b.ins().isub(len, two); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::Slice, 3), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bytes_get( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let source_index = b.ins().isub(len, two); - let index_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let source_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - source_index, - ctx.layout.value.size, - ); - let index_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - index_index, - ctx.layout.value.size, - ); - let source_tag = load_tag_i32(b, ctx.layout.value, source_addr); - let index_tag = load_tag_i32(b, ctx.layout.value, index_addr); - let source_ok = b.ins().icmp_imm( - IntCC::Equal, - source_tag, - i64::from(ctx.layout.value.bytes_tag), - ); - let index_ok = b - .ins() - .icmp_imm(IntCC::Equal, index_tag, i64::from(ctx.layout.value.int_tag)); - let all_ok = b.ins().band(source_ok, index_ok); - b.ins().brif(all_ok, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let bytes_raw = load_heap_ptr(b, ctx.layout.value, source_addr, ctx.pointer_type); - let bytes_data = load_heap_data_ptr(b, ctx.layout.value, bytes_raw); - let bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.ptr_offset, - ); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.len_offset, - ); - let index = b.ins().load( - types::I64, - MemFlags::new(), - index_addr, - ctx.layout.value.int_payload_offset, - ); - let negative = b.ins().icmp_imm(IntCC::SignedLessThan, index, 0); - let bounds_ok = b.create_block(); - b.ins().brif(negative, slow, &[], bounds_ok, &[]); - - b.switch_to_block(bounds_ok); - let in_bounds = b.ins().icmp(IntCC::UnsignedLessThan, index, bytes_len); - let load_ok = b.create_block(); - b.ins().brif(in_bounds, load_ok, &[], slow, &[]); - - b.switch_to_block(load_ok); - let byte_ptr = b.ins().iadd(bytes_ptr, index); - let byte = load_byte(b, byte_ptr); - let byte_i64 = b.ins().uextend(types::I64, byte); - call_drop_shared(b, ctx, ctx.heap_addrs.drop_bytes, bytes_raw)?; - store_int_in_value(b, ctx.layout.value, source_addr, byte_i64); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::Get, 2), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bytes_has( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let source_index = b.ins().isub(len, two); - let index_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let source_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - source_index, - ctx.layout.value.size, - ); - let index_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - index_index, - ctx.layout.value.size, - ); - let source_tag = load_tag_i32(b, ctx.layout.value, source_addr); - let index_tag = load_tag_i32(b, ctx.layout.value, index_addr); - let source_ok = b.ins().icmp_imm( - IntCC::Equal, - source_tag, - i64::from(ctx.layout.value.bytes_tag), - ); - let index_ok = b - .ins() - .icmp_imm(IntCC::Equal, index_tag, i64::from(ctx.layout.value.int_tag)); - let all_ok = b.ins().band(source_ok, index_ok); - b.ins().brif(all_ok, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let bytes_raw = load_heap_ptr(b, ctx.layout.value, source_addr, ctx.pointer_type); - let bytes_data = load_heap_data_ptr(b, ctx.layout.value, bytes_raw); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.len_offset, - ); - let index = b.ins().load( - types::I64, - MemFlags::new(), - index_addr, - ctx.layout.value.int_payload_offset, - ); - let negative = b.ins().icmp_imm(IntCC::SignedLessThan, index, 0); - let in_bounds = b.ins().icmp(IntCC::UnsignedLessThan, index, bytes_len); - let not_negative = b.ins().bnot(negative); - let present = b.ins().band(not_negative, in_bounds); - call_drop_shared(b, ctx, ctx.heap_addrs.drop_bytes, bytes_raw)?; - store_bool_in_value(b, ctx.layout.value, source_addr, present); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::Has, 2), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_string_get( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let loop_block = b.create_block(); - let next = b.create_block(); - b.append_block_param(loop_block, ctx.pointer_type); - b.append_block_param(loop_block, ctx.pointer_type); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let source_index = b.ins().isub(len, two); - let index_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let source_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - source_index, - ctx.layout.value.size, - ); - let index_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - index_index, - ctx.layout.value.size, - ); - let source_tag = load_tag_i32(b, ctx.layout.value, source_addr); - let index_tag = load_tag_i32(b, ctx.layout.value, index_addr); - let source_ok = b.ins().icmp_imm( - IntCC::Equal, - source_tag, - i64::from(ctx.layout.value.string_tag), - ); - let index_ok = b - .ins() - .icmp_imm(IntCC::Equal, index_tag, i64::from(ctx.layout.value.int_tag)); - let all_ok = b.ins().band(source_ok, index_ok); - b.ins().brif(all_ok, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let string_raw = load_heap_ptr(b, ctx.layout.value, source_addr, ctx.pointer_type); - let string_data = load_heap_data_ptr(b, ctx.layout.value, string_raw); - let bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - string_data, - ctx.layout.stack_vec.ptr_offset, - ); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - string_data, - ctx.layout.stack_vec.len_offset, - ); - let index = b.ins().load( - types::I64, - MemFlags::new(), - index_addr, - ctx.layout.value.int_payload_offset, - ); - let negative = b.ins().icmp_imm(IntCC::SignedLessThan, index, 0); - let loop_entry = b.create_block(); - b.ins().brif(negative, slow, &[], loop_entry, &[]); - - b.switch_to_block(loop_entry); - let zero = b.ins().iconst(ctx.pointer_type, 0); - b.ins() - .jump(loop_block, &[BlockArg::Value(zero), BlockArg::Value(zero)]); - - b.switch_to_block(loop_block); - let byte_index = b.block_params(loop_block)[0]; - let char_index = b.block_params(loop_block)[1]; - let past_end = b - .ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, byte_index, bytes_len); - let scan_block = b.create_block(); - b.ins().brif(past_end, slow, &[], scan_block, &[]); - - b.switch_to_block(scan_block); - let byte_ptr = b.ins().iadd(bytes_ptr, byte_index); - let byte = load_byte(b, byte_ptr); - let width = utf8_char_width(b, ctx.pointer_type, byte); - let at_target = b.ins().icmp(IntCC::Equal, char_index, index); - let copy_block = b.create_block(); - let advance_block = b.create_block(); - b.ins().brif(at_target, copy_block, &[], advance_block, &[]); - - b.switch_to_block(copy_block); - let out_ptr = call_alloc_buffer(b, ctx, ctx.heap_addrs.alloc_byte_buffer, width)?; - call_copy_bytes(b, ctx, out_ptr, byte_ptr, width)?; - let out_raw = call_pack_shared(b, ctx, ctx.heap_addrs.pack_string, out_ptr, width, width)?; - call_drop_shared(b, ctx, ctx.heap_addrs.drop_string, string_raw)?; - store_heap_ptr_in_value( - b, - ctx.layout.value, - source_addr, - ctx.layout.value.string_tag, - out_raw, - ); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(advance_block); - let next_byte = b.ins().iadd(byte_index, width); - let next_char = b.ins().iadd_imm(char_index, 1); - b.ins().jump( - loop_block, - &[BlockArg::Value(next_byte), BlockArg::Value(next_char)], - ); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::Get, 2), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_string_slice( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let seek_start = b.create_block(); - let seek_end = b.create_block(); - let copy_block = b.create_block(); - let empty_block = b.create_block(); - let next = b.create_block(); - b.append_block_param(seek_start, ctx.pointer_type); - b.append_block_param(seek_start, ctx.pointer_type); - b.append_block_param(seek_end, ctx.pointer_type); - b.append_block_param(seek_end, ctx.pointer_type); - b.append_block_param(seek_end, ctx.pointer_type); - b.append_block_param(copy_block, ctx.pointer_type); - b.append_block_param(copy_block, ctx.pointer_type); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 3); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let three = b.ins().iconst(ctx.pointer_type, 3); - let source_index = b.ins().isub(len, three); - let start_index = b.ins().isub(len, two); - let length_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let source_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - source_index, - ctx.layout.value.size, - ); - let start_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - start_index, - ctx.layout.value.size, - ); - let length_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - length_index, - ctx.layout.value.size, - ); - let source_tag = load_tag_i32(b, ctx.layout.value, source_addr); - let start_tag = load_tag_i32(b, ctx.layout.value, start_addr); - let length_tag = load_tag_i32(b, ctx.layout.value, length_addr); - let source_ok = b.ins().icmp_imm( - IntCC::Equal, - source_tag, - i64::from(ctx.layout.value.string_tag), - ); - let start_ok = b - .ins() - .icmp_imm(IntCC::Equal, start_tag, i64::from(ctx.layout.value.int_tag)); - let length_ok_tag = b.ins().icmp_imm( - IntCC::Equal, - length_tag, - i64::from(ctx.layout.value.int_tag), - ); - let both_ints = b.ins().band(start_ok, length_ok_tag); - let all_ok = b.ins().band(source_ok, both_ints); - b.ins().brif(all_ok, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let string_raw = load_heap_ptr(b, ctx.layout.value, source_addr, ctx.pointer_type); - let string_data = load_heap_data_ptr(b, ctx.layout.value, string_raw); - let bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - string_data, - ctx.layout.stack_vec.ptr_offset, - ); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - string_data, - ctx.layout.stack_vec.len_offset, - ); - let start = b.ins().load( - types::I64, - MemFlags::new(), - start_addr, - ctx.layout.value.int_payload_offset, - ); - let slice_chars = b.ins().load( - types::I64, - MemFlags::new(), - length_addr, - ctx.layout.value.int_payload_offset, - ); - let zero = b.ins().iconst(ctx.pointer_type, 0); - let start_negative = b.ins().icmp_imm(IntCC::SignedLessThan, start, 0); - let length_positive = b.ins().icmp_imm(IntCC::SignedGreaterThan, slice_chars, 0); - let start_non_negative = b.ins().bnot(start_negative); - let positive = b.ins().band(start_non_negative, length_positive); - let seek_entry = b.create_block(); - b.ins().brif(positive, seek_entry, &[], empty_block, &[]); - - b.switch_to_block(seek_entry); - b.ins() - .jump(seek_start, &[BlockArg::Value(zero), BlockArg::Value(zero)]); - - b.switch_to_block(seek_start); - let byte_index = b.block_params(seek_start)[0]; - let char_index = b.block_params(seek_start)[1]; - let reached_start = b.ins().icmp(IntCC::Equal, char_index, start); - let start_found = b.create_block(); - let scan_more = b.create_block(); - b.ins() - .brif(reached_start, start_found, &[], scan_more, &[]); - - b.switch_to_block(scan_more); - let at_end = b - .ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, byte_index, bytes_len); - b.ins().brif(at_end, empty_block, &[], start_found, &[]); - - b.switch_to_block(start_found); - let current_ptr = b.ins().iadd(bytes_ptr, byte_index); - let current_byte = load_byte(b, current_ptr); - let current_width = utf8_char_width(b, ctx.pointer_type, current_byte); - let start_now = b.ins().icmp(IntCC::Equal, char_index, start); - let advance_to_start = b.create_block(); - b.ins().brif( - start_now, - seek_end, - &[ - BlockArg::Value(byte_index), - BlockArg::Value(byte_index), - BlockArg::Value(slice_chars), - ], - advance_to_start, - &[], - ); - - b.switch_to_block(advance_to_start); - let next_byte = b.ins().iadd(byte_index, current_width); - let next_char = b.ins().iadd_imm(char_index, 1); - b.ins().jump( - seek_start, - &[BlockArg::Value(next_byte), BlockArg::Value(next_char)], - ); - - b.switch_to_block(seek_end); - let slice_start = b.block_params(seek_end)[0]; - let end_byte = b.block_params(seek_end)[1]; - let remaining_chars = b.block_params(seek_end)[2]; - let no_chars_left = b.ins().icmp_imm(IntCC::Equal, remaining_chars, 0); - let reached_end = b - .ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, end_byte, bytes_len); - let finish_now = b.ins().bor(no_chars_left, reached_end); - let finish_block = b.create_block(); - let advance_block = b.create_block(); - b.ins() - .brif(finish_now, finish_block, &[], advance_block, &[]); - - b.switch_to_block(advance_block); - let end_ptr = b.ins().iadd(bytes_ptr, end_byte); - let end_byte_value = load_byte(b, end_ptr); - let end_width = utf8_char_width(b, ctx.pointer_type, end_byte_value); - let next_end = b.ins().iadd(end_byte, end_width); - let remaining_next = b.ins().isub(remaining_chars, one); - b.ins().jump( - seek_end, - &[ - BlockArg::Value(slice_start), - BlockArg::Value(next_end), - BlockArg::Value(remaining_next), - ], - ); - - b.switch_to_block(finish_block); - let slice_len = b.ins().isub(end_byte, slice_start); - b.ins().jump( - copy_block, - &[BlockArg::Value(slice_start), BlockArg::Value(slice_len)], - ); - - b.switch_to_block(empty_block); - b.ins() - .jump(copy_block, &[BlockArg::Value(zero), BlockArg::Value(zero)]); - - b.switch_to_block(copy_block); - let slice_start = b.block_params(copy_block)[0]; - let slice_len = b.block_params(copy_block)[1]; - let source_ptr = b.ins().iadd(bytes_ptr, slice_start); - let out_ptr = call_alloc_buffer(b, ctx, ctx.heap_addrs.alloc_byte_buffer, slice_len)?; - call_copy_bytes(b, ctx, out_ptr, source_ptr, slice_len)?; - let out_raw = call_pack_shared( - b, - ctx, - ctx.heap_addrs.pack_string, - out_ptr, - slice_len, - slice_len, - )?; - call_drop_shared(b, ctx, ctx.heap_addrs.drop_string, string_raw)?; - store_heap_ptr_in_value( - b, - ctx.layout.value, - source_addr, - ctx.layout.value.string_tag, - out_raw, - ); - let new_len = b.ins().isub(len, two); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::Slice, 3), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bytes_from_array_u8( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let validate_loop = b.create_block(); - let copy_loop = b.create_block(); - let next = b.create_block(); - b.append_block_param(validate_loop, ctx.pointer_type); - b.append_block_param(copy_loop, ctx.pointer_type); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(enough, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let one = b.ins().iconst(ctx.pointer_type, 1); - let stack_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let source_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - stack_index, - ctx.layout.value.size, - ); - let source_tag = load_tag_i32(b, ctx.layout.value, source_addr); - let is_array = b.ins().icmp_imm( - IntCC::Equal, - source_tag, - i64::from(ctx.layout.value.array_tag), - ); - let type_ok = b.create_block(); - b.ins().brif(is_array, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let array_raw = load_heap_ptr(b, ctx.layout.value, source_addr, ctx.pointer_type); - let array_data = load_heap_data_ptr(b, ctx.layout.value, array_raw); - let values_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - array_data, - ctx.layout.stack_vec.ptr_offset, - ); - let values_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - array_data, - ctx.layout.stack_vec.len_offset, - ); - let zero = b.ins().iconst(ctx.pointer_type, 0); - b.ins().jump(validate_loop, &[BlockArg::Value(zero)]); - - b.switch_to_block(validate_loop); - let validate_index = b.block_params(validate_loop)[0]; - let done = b.ins().icmp( - IntCC::UnsignedGreaterThanOrEqual, - validate_index, - values_len, - ); - let validated = b.create_block(); - let validate_step = b.create_block(); - b.ins().brif(done, validated, &[], validate_step, &[]); - - b.switch_to_block(validate_step); - let element_addr = value_addr( - b, - ctx.pointer_type, - values_ptr, - validate_index, - ctx.layout.value.size, - ); - let element_tag = load_tag_i32(b, ctx.layout.value, element_addr); - let is_int = b.ins().icmp_imm( - IntCC::Equal, - element_tag, - i64::from(ctx.layout.value.int_tag), - ); - let int_ok = b.create_block(); - b.ins().brif(is_int, int_ok, &[], slow, &[]); - - b.switch_to_block(int_ok); - let value = b.ins().load( - types::I64, - MemFlags::new(), - element_addr, - ctx.layout.value.int_payload_offset, - ); - let non_negative = b.ins().icmp_imm(IntCC::SignedGreaterThanOrEqual, value, 0); - let le_255 = b.ins().icmp_imm(IntCC::SignedLessThanOrEqual, value, 255); - let valid_byte = b.ins().band(non_negative, le_255); - let validate_next = b.create_block(); - b.ins().brif(valid_byte, validate_next, &[], slow, &[]); - - b.switch_to_block(validate_next); - let next_index = b.ins().iadd_imm(validate_index, 1); - b.ins().jump(validate_loop, &[BlockArg::Value(next_index)]); - - b.switch_to_block(validated); - let out_ptr = call_alloc_buffer(b, ctx, ctx.heap_addrs.alloc_byte_buffer, values_len)?; - b.ins().jump(copy_loop, &[BlockArg::Value(zero)]); - - b.switch_to_block(copy_loop); - let copy_index = b.block_params(copy_loop)[0]; - let copy_done = b - .ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, copy_index, values_len); - let finish = b.create_block(); - let copy_step = b.create_block(); - b.ins().brif(copy_done, finish, &[], copy_step, &[]); - - b.switch_to_block(copy_step); - let element_addr = value_addr( - b, - ctx.pointer_type, - values_ptr, - copy_index, - ctx.layout.value.size, - ); - let value = b.ins().load( - types::I64, - MemFlags::new(), - element_addr, - ctx.layout.value.int_payload_offset, - ); - let byte = b.ins().ireduce(types::I8, value); - let dst = b.ins().iadd(out_ptr, copy_index); - b.ins().store(MemFlags::new(), byte, dst, 0); - let next_index = b.ins().iadd_imm(copy_index, 1); - b.ins().jump(copy_loop, &[BlockArg::Value(next_index)]); - - b.switch_to_block(finish); - let out_raw = call_pack_shared( - b, - ctx, - ctx.heap_addrs.pack_bytes, - out_ptr, - values_len, - values_len, - )?; - call_drop_shared(b, ctx, ctx.heap_addrs.drop_array, array_raw)?; - store_heap_ptr_in_value( - b, - ctx.layout.value, - source_addr, - ctx.layout.value.bytes_tag, - out_raw, - ); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::BytesFromArrayU8, 1), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bytes_to_array_u8( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let fill_loop = b.create_block(); - let next = b.create_block(); - b.append_block_param(fill_loop, ctx.pointer_type); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(enough, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let one = b.ins().iconst(ctx.pointer_type, 1); - let stack_index = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let source_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - stack_index, - ctx.layout.value.size, - ); - let source_tag = load_tag_i32(b, ctx.layout.value, source_addr); - let is_bytes = b.ins().icmp_imm( - IntCC::Equal, - source_tag, - i64::from(ctx.layout.value.bytes_tag), - ); - let type_ok = b.create_block(); - b.ins().brif(is_bytes, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let bytes_raw = load_heap_ptr(b, ctx.layout.value, source_addr, ctx.pointer_type); - let bytes_data = load_heap_data_ptr(b, ctx.layout.value, bytes_raw); - let bytes_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.ptr_offset, - ); - let bytes_len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - bytes_data, - ctx.layout.stack_vec.len_offset, - ); - let value_size = i64::from(ctx.layout.value.size); - let max_values = b.ins().iconst(ctx.pointer_type, i64::MAX / value_size); - let too_large = b - .ins() - .icmp(IntCC::UnsignedGreaterThan, bytes_len, max_values); - let cap_ok = b.create_block(); - b.ins().brif(too_large, slow, &[], cap_ok, &[]); - - b.switch_to_block(cap_ok); - let out_ptr = call_alloc_buffer(b, ctx, ctx.heap_addrs.alloc_value_buffer, bytes_len)?; - let total_bytes = b.ins().imul_imm(bytes_len, value_size); - call_zero_bytes(b, ctx, out_ptr, total_bytes)?; - let zero = b.ins().iconst(ctx.pointer_type, 0); - b.ins().jump(fill_loop, &[BlockArg::Value(zero)]); - - b.switch_to_block(fill_loop); - let fill_index = b.block_params(fill_loop)[0]; - let done = b - .ins() - .icmp(IntCC::UnsignedGreaterThanOrEqual, fill_index, bytes_len); - let finish = b.create_block(); - let fill_step = b.create_block(); - b.ins().brif(done, finish, &[], fill_step, &[]); - - b.switch_to_block(fill_step); - let src_ptr = b.ins().iadd(bytes_ptr, fill_index); - let byte = load_byte(b, src_ptr); - let byte_i64 = b.ins().uextend(types::I64, byte); - let element_addr = value_addr( - b, - ctx.pointer_type, - out_ptr, - fill_index, - ctx.layout.value.size, - ); - store_int_in_value(b, ctx.layout.value, element_addr, byte_i64); - let next_index = b.ins().iadd_imm(fill_index, 1); - b.ins().jump(fill_loop, &[BlockArg::Value(next_index)]); - - b.switch_to_block(finish); - let out_raw = call_pack_shared( - b, - ctx, - ctx.heap_addrs.pack_array, - out_ptr, - bytes_len, - bytes_len, - )?; - call_drop_shared(b, ctx, ctx.heap_addrs.drop_bytes, bytes_raw)?; - store_heap_ptr_in_value( - b, - ctx.layout.value, - source_addr, - ctx.layout.value.array_tag, - out_raw, - ); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - builtin_helper_tuple(BuiltinFunction::BytesToArrayU8, 1), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_int_divrem( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - is_mod: bool, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let non_zero = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_int = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.int_tag)); - let rhs_int = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.int_tag)); - let both_int = b.ins().band(lhs_int, rhs_int); - b.ins().brif(both_int, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let lhs = b.ins().load( - types::I64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let rhs = b.ins().load( - types::I64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.int_payload_offset, - ); - let rhs_not_zero = b.ins().icmp_imm(IntCC::NotEqual, rhs, 0); - b.ins().brif(rhs_not_zero, non_zero, &[], slow, &[]); - - b.switch_to_block(non_zero); - let min_i64 = b.ins().iconst(types::I64, i64::MIN); - let neg_one = b.ins().iconst(types::I64, -1); - let lhs_is_min = b.ins().icmp(IntCC::Equal, lhs, min_i64); - let rhs_is_neg_one = b.ins().icmp(IntCC::Equal, rhs, neg_one); - let overflow_case = b.ins().band(lhs_is_min, rhs_is_neg_one); - let normal_block = b.create_block(); - b.ins().brif(overflow_case, slow, &[], normal_block, &[]); - - b.switch_to_block(normal_block); - let out = if is_mod { - b.ins().srem(lhs, rhs) - } else { - b.ins().sdiv(lhs, rhs) - }; - b.ins().store( - MemFlags::new(), - out, - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (if is_mod { OP_MOD } else { OP_DIV }, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_shift( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - kind: ShiftKind, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let shift_ok = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_int = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.int_tag)); - let rhs_int = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.int_tag)); - let both_int = b.ins().band(lhs_int, rhs_int); - b.ins().brif(both_int, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let lhs = b.ins().load( - types::I64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let rhs = b.ins().load( - types::I64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.int_payload_offset, - ); - let shift_ge_zero = b.ins().icmp_imm(IntCC::SignedGreaterThanOrEqual, rhs, 0); - let shift_le_63 = b.ins().icmp_imm(IntCC::SignedLessThanOrEqual, rhs, 63); - let shift_in_range = b.ins().band(shift_ge_zero, shift_le_63); - b.ins().brif(shift_in_range, shift_ok, &[], slow, &[]); - - b.switch_to_block(shift_ok); - let out = match kind { - ShiftKind::Left => b.ins().ishl(lhs, rhs), - ShiftKind::ArithmeticRight => b.ins().sshr(lhs, rhs), - ShiftKind::LogicalRight => b.ins().ushr(lhs, rhs), - }; - b.ins().store( - MemFlags::new(), - out, - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - ( - match kind { - ShiftKind::Left => OP_SHL, - ShiftKind::ArithmeticRight => OP_SHR, - ShiftKind::LogicalRight => OP_LSHR, - }, - 0, - 0, - 0, - ), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_bool_logic( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - is_and: bool, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let type_ok = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_bool = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.bool_tag)); - let rhs_bool = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.bool_tag)); - let both_bool = b.ins().band(lhs_bool, rhs_bool); - b.ins().brif(both_bool, type_ok, &[], slow, &[]); - - b.switch_to_block(type_ok); - let lhs = b.ins().load( - types::I8, - MemFlags::new(), - lhs_addr, - ctx.layout.value.bool_payload_offset, - ); - let rhs = b.ins().load( - types::I8, - MemFlags::new(), - rhs_addr, - ctx.layout.value.bool_payload_offset, - ); - let lhs_non_zero = b.ins().icmp_imm(IntCC::NotEqual, lhs, 0); - let rhs_non_zero = b.ins().icmp_imm(IntCC::NotEqual, rhs, 0); - let out_bool = if is_and { - b.ins().band(lhs_non_zero, rhs_non_zero) - } else { - b.ins().bor(lhs_non_zero, rhs_non_zero) - }; - store_bool_in_value(b, ctx.layout.value, lhs_addr, out_bool); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (if is_and { OP_AND } else { OP_OR }, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_not(b: &mut FunctionBuilder, ctx: InlineEmitCtx, step_ip: usize) -> VmResult<()> { - let next = b.create_block(); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_NOT, 0, 0, 0), - ); - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_neg(b: &mut FunctionBuilder, ctx: InlineEmitCtx, step_ip: usize) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let has_stack = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(has_stack, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let one = b.ins().iconst(ctx.pointer_type, 1); - let idx = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let addr = value_addr(b, ctx.pointer_type, stack_ptr, idx, ctx.layout.value.size); - let tag = load_tag_i32(b, ctx.layout.value, addr); - let is_int = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(ctx.layout.value.int_tag)); - let int_ok = b.create_block(); - b.ins().brif(is_int, int_ok, &[], slow, &[]); - - b.switch_to_block(int_ok); - let value = b.ins().load( - types::I64, - MemFlags::new(), - addr, - ctx.layout.value.int_payload_offset, - ); - let neg = b.ins().irsub_imm(value, 0); - b.ins().store( - MemFlags::new(), - neg, - addr, - ctx.layout.value.int_payload_offset, - ); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_NEG, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_float_neg( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let has_stack = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 1); - b.ins().brif(has_stack, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let one = b.ins().iconst(ctx.pointer_type, 1); - let idx = b.ins().isub(len, one); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let addr = value_addr(b, ctx.pointer_type, stack_ptr, idx, ctx.layout.value.size); - let tag = load_tag_i32(b, ctx.layout.value, addr); - let is_float = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(ctx.layout.value.float_tag)); - let float_ok = b.create_block(); - b.ins().brif(is_float, float_ok, &[], slow, &[]); - - b.switch_to_block(float_ok); - let value = b.ins().load( - types::F64, - MemFlags::new(), - addr, - ctx.layout.value.float_payload_offset, - ); - let neg = b.ins().fneg(value); - b.ins().store( - MemFlags::new(), - neg, - addr, - ctx.layout.value.float_payload_offset, - ); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_NEG, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_int_compare( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - less_than: bool, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_int = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.int_tag)); - let rhs_int = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.int_tag)); - let both_int = b.ins().band(lhs_int, rhs_int); - b.ins().brif(both_int, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let lhs = b.ins().load( - types::I64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let rhs = b.ins().load( - types::I64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.int_payload_offset, - ); - let cmp = if less_than { - b.ins().icmp(IntCC::SignedLessThan, lhs, rhs) - } else { - b.ins().icmp(IntCC::SignedGreaterThan, lhs, rhs) - }; - store_bool_in_value(b, ctx.layout.value, lhs_addr, cmp); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (if less_than { OP_CLT } else { OP_CGT }, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_float_compare( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, - less_than: bool, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_float = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.float_tag)); - let rhs_float = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.float_tag)); - let both_float = b.ins().band(lhs_float, rhs_float); - b.ins().brif(both_float, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let lhs = b.ins().load( - types::F64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.float_payload_offset, - ); - let rhs = b.ins().load( - types::F64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.float_payload_offset, - ); - let cmp = if less_than { - b.ins().fcmp(FloatCC::LessThan, lhs, rhs) - } else { - b.ins().fcmp(FloatCC::GreaterThan, lhs, rhs) - }; - store_bool_in_value(b, ctx.layout.value, lhs_addr, cmp); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (if less_than { OP_CLT } else { OP_CGT }, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_int_eq(b: &mut FunctionBuilder, ctx: InlineEmitCtx, step_ip: usize) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let int_fast = b.create_block(); - let bool_check = b.create_block(); - let bool_fast = b.create_block(); - let null_check = b.create_block(); - let null_fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let tags_equal = b.ins().icmp(IntCC::Equal, lhs_tag, rhs_tag); - let tag_eq = b.create_block(); - b.ins().brif(tags_equal, tag_eq, &[], slow, &[]); - - b.switch_to_block(tag_eq); - let lhs_int = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.int_tag)); - b.ins().brif(lhs_int, int_fast, &[], bool_check, &[]); - - b.switch_to_block(bool_check); - let lhs_bool = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.bool_tag)); - b.ins().brif(lhs_bool, bool_fast, &[], null_check, &[]); - - b.switch_to_block(null_check); - let lhs_null = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.null_tag)); - b.ins().brif(lhs_null, null_fast, &[], slow, &[]); - - b.switch_to_block(int_fast); - let lhs = b.ins().load( - types::I64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.int_payload_offset, - ); - let rhs = b.ins().load( - types::I64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.int_payload_offset, - ); - let cmp = b.ins().icmp(IntCC::Equal, lhs, rhs); - store_bool_in_value(b, ctx.layout.value, lhs_addr, cmp); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(bool_fast); - let lhs_bool_value = b.ins().load( - types::I8, - MemFlags::new(), - lhs_addr, - ctx.layout.value.bool_payload_offset, - ); - let rhs_bool_value = b.ins().load( - types::I8, - MemFlags::new(), - rhs_addr, - ctx.layout.value.bool_payload_offset, - ); - let bool_eq = b.ins().icmp(IntCC::Equal, lhs_bool_value, rhs_bool_value); - store_bool_in_value(b, ctx.layout.value, lhs_addr, bool_eq); - let new_len_bool = b.ins().isub(len, one); - b.ins().store( - MemFlags::new(), - new_len_bool, - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - b.ins().jump(next, &[]); - - b.switch_to_block(null_fast); - let null_eq = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.null_tag)); - store_bool_in_value(b, ctx.layout.value, lhs_addr, null_eq); - let new_len_null = b.ins().isub(len, one); - b.ins().store( - MemFlags::new(), - new_len_null, - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_CEQ, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn emit_inline_float_eq( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - step_ip: usize, -) -> VmResult<()> { - let slow = b.create_block(); - let len_ok = b.create_block(); - let fast = b.create_block(); - let next = b.create_block(); - - let len = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_len, - ); - let enough = b.ins().icmp_imm(IntCC::UnsignedGreaterThanOrEqual, len, 2); - b.ins().brif(enough, len_ok, &[], slow, &[]); - - b.switch_to_block(len_ok); - let one = b.ins().iconst(ctx.pointer_type, 1); - let two = b.ins().iconst(ctx.pointer_type, 2); - let rhs_index = b.ins().isub(len, one); - let lhs_index = b.ins().isub(len, two); - let stack_ptr = b.ins().load( - ctx.pointer_type, - MemFlags::new(), - ctx.vm_ptr, - ctx.offsets.stack_ptr, - ); - let lhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - lhs_index, - ctx.layout.value.size, - ); - let rhs_addr = value_addr( - b, - ctx.pointer_type, - stack_ptr, - rhs_index, - ctx.layout.value.size, - ); - let lhs_tag = load_tag_i32(b, ctx.layout.value, lhs_addr); - let rhs_tag = load_tag_i32(b, ctx.layout.value, rhs_addr); - let lhs_float = b - .ins() - .icmp_imm(IntCC::Equal, lhs_tag, i64::from(ctx.layout.value.float_tag)); - let rhs_float = b - .ins() - .icmp_imm(IntCC::Equal, rhs_tag, i64::from(ctx.layout.value.float_tag)); - let both_float = b.ins().band(lhs_float, rhs_float); - b.ins().brif(both_float, fast, &[], slow, &[]); - - b.switch_to_block(fast); - let lhs = b.ins().load( - types::F64, - MemFlags::new(), - lhs_addr, - ctx.layout.value.float_payload_offset, - ); - let rhs = b.ins().load( - types::F64, - MemFlags::new(), - rhs_addr, - ctx.layout.value.float_payload_offset, - ); - let cmp = b.ins().fcmp(FloatCC::Equal, lhs, rhs); - store_bool_in_value(b, ctx.layout.value, lhs_addr, cmp); - let new_len = b.ins().isub(len, one); - b.ins() - .store(MemFlags::new(), new_len, ctx.vm_ptr, ctx.offsets.stack_len); - b.ins().jump(next, &[]); - - b.switch_to_block(slow); - emit_helper_step_from_call_tuple( - b, - ctx.vm_ptr, - ctx.helper_ref, - ctx.exit_block, - next, - ctx.offsets, - step_ip, - (OP_CEQ, 0, 0, 0), - ); - - b.switch_to_block(next); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn value_addr( - b: &mut FunctionBuilder, - pointer_type: cranelift_codegen::ir::Type, - base_ptr: cranelift_codegen::ir::Value, - index: cranelift_codegen::ir::Value, - value_size: i32, -) -> cranelift_codegen::ir::Value { - let stride = b.ins().iconst(pointer_type, i64::from(value_size)); - let offset = b.ins().imul(index, stride); - b.ins().iadd(base_ptr, offset) -} - -#[cfg(feature = "cranelift-jit")] -fn tag_type(layout: ValueLayout) -> cranelift_codegen::ir::Type { - match layout.tag_size { - 1 => types::I8, - 2 => types::I16, - 4 => types::I32, - _ => types::I32, - } -} - -#[cfg(feature = "cranelift-jit")] -fn load_tag_i32( - b: &mut FunctionBuilder, - layout: ValueLayout, - value_addr: cranelift_codegen::ir::Value, -) -> cranelift_codegen::ir::Value { - let raw = b.ins().load( - tag_type(layout), - MemFlags::new(), - value_addr, - layout.tag_offset, - ); - match layout.tag_size { - 1 | 2 => b.ins().uextend(types::I32, raw), - _ => raw, - } -} - -#[cfg(feature = "cranelift-jit")] -fn store_tag( - b: &mut FunctionBuilder, - layout: ValueLayout, - value_addr: cranelift_codegen::ir::Value, - tag: u32, -) { - let ty = tag_type(layout); - let raw = b.ins().iconst(ty, i64::from(tag)); - b.ins() - .store(MemFlags::new(), raw, value_addr, layout.tag_offset); -} - -#[cfg(feature = "cranelift-jit")] -fn is_scalar_tag( - b: &mut FunctionBuilder, - layout: ValueLayout, - tag: cranelift_codegen::ir::Value, -) -> cranelift_codegen::ir::Value { - let is_null = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(layout.null_tag)); - let is_int = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(layout.int_tag)); - let is_float = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(layout.float_tag)); - let is_bool = b - .ins() - .icmp_imm(IntCC::Equal, tag, i64::from(layout.bool_tag)); - let scalar_a = b.ins().bor(is_null, is_int); - let scalar_b = b.ins().bor(is_float, is_bool); - b.ins().bor(scalar_a, scalar_b) -} - -#[cfg(feature = "cranelift-jit")] -fn store_bool_in_value( - b: &mut FunctionBuilder, - layout: ValueLayout, - value_addr: cranelift_codegen::ir::Value, - bool_value: cranelift_codegen::ir::Value, -) { - store_tag(b, layout, value_addr, layout.bool_tag); - let one = b.ins().iconst(types::I8, 1); - let zero = b.ins().iconst(types::I8, 0); - let byte_value = b.ins().select(bool_value, one, zero); - b.ins().store( - MemFlags::new(), - byte_value, - value_addr, - layout.bool_payload_offset, - ); -} - -#[cfg(feature = "cranelift-jit")] -fn copy_value_bytes( - b: &mut FunctionBuilder, - src_addr: cranelift_codegen::ir::Value, - dst_addr: cranelift_codegen::ir::Value, - size: i32, -) { - let mut offset = 0i32; - while offset + 8 <= size { - let chunk = b.ins().load(types::I64, MemFlags::new(), src_addr, offset); - b.ins().store(MemFlags::new(), chunk, dst_addr, offset); - offset += 8; - } - if offset + 4 <= size { - let chunk = b.ins().load(types::I32, MemFlags::new(), src_addr, offset); - b.ins().store(MemFlags::new(), chunk, dst_addr, offset); - offset += 4; - } - while offset < size { - let chunk = b.ins().load(types::I8, MemFlags::new(), src_addr, offset); - b.ins().store(MemFlags::new(), chunk, dst_addr, offset); - offset += 1; - } -} - -#[cfg(feature = "cranelift-jit")] -fn store_int_in_value( - b: &mut FunctionBuilder, - layout: ValueLayout, - value_addr: cranelift_codegen::ir::Value, - int_value: cranelift_codegen::ir::Value, -) { - store_tag(b, layout, value_addr, layout.int_tag); - b.ins().store( - MemFlags::new(), - int_value, - value_addr, - layout.int_payload_offset, - ); -} - -#[cfg(feature = "cranelift-jit")] -fn store_heap_ptr_in_value( - b: &mut FunctionBuilder, - layout: ValueLayout, - value_addr: cranelift_codegen::ir::Value, - tag: u32, - heap_ptr: cranelift_codegen::ir::Value, -) { - store_tag(b, layout, value_addr, tag); - b.ins().store( - MemFlags::new(), - heap_ptr, - value_addr, - layout.heap_payload_offset, - ); -} - -#[cfg(feature = "cranelift-jit")] -fn load_heap_ptr( - b: &mut FunctionBuilder, - layout: ValueLayout, - value_addr: cranelift_codegen::ir::Value, - pointer_type: cranelift_codegen::ir::Type, -) -> cranelift_codegen::ir::Value { - b.ins().load( - pointer_type, - MemFlags::new(), - value_addr, - layout.heap_payload_offset, - ) -} - -#[cfg(feature = "cranelift-jit")] -fn load_heap_data_ptr( - b: &mut FunctionBuilder, - layout: ValueLayout, - heap_ptr: cranelift_codegen::ir::Value, -) -> cranelift_codegen::ir::Value { - if layout.arc_data_offset == 0 { - heap_ptr - } else { - b.ins() - .iadd_imm(heap_ptr, i64::from(layout.arc_data_offset)) - } -} - -#[cfg(feature = "cranelift-jit")] -fn iconst_ptr_from_addr( - b: &mut FunctionBuilder, - pointer_type: cranelift_codegen::ir::Type, - addr: usize, -) -> VmResult { - let addr = i64::try_from(addr) - .map_err(|_| VmError::JitNative("native helper address out of range".to_string()))?; - Ok(b.ins().iconst(pointer_type, addr)) -} - -#[cfg(feature = "cranelift-jit")] -fn call_alloc_buffer( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - addr: usize, - cap: cranelift_codegen::ir::Value, -) -> VmResult { - let helper_ptr = iconst_ptr_from_addr(b, ctx.pointer_type, addr)?; - let call = b - .ins() - .call_indirect(ctx.heap_refs.alloc_buffer_ref, helper_ptr, &[cap]); - Ok(b.inst_results(call)[0]) -} - -#[cfg(feature = "cranelift-jit")] -fn call_pack_shared( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - addr: usize, - ptr: cranelift_codegen::ir::Value, - len: cranelift_codegen::ir::Value, - cap: cranelift_codegen::ir::Value, -) -> VmResult { - let helper_ptr = iconst_ptr_from_addr(b, ctx.pointer_type, addr)?; - let call = b - .ins() - .call_indirect(ctx.heap_refs.pack_shared_ref, helper_ptr, &[ptr, len, cap]); - Ok(b.inst_results(call)[0]) -} - -#[cfg(feature = "cranelift-jit")] -fn call_drop_shared( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - addr: usize, - ptr: cranelift_codegen::ir::Value, -) -> VmResult<()> { - let helper_ptr = iconst_ptr_from_addr(b, ctx.pointer_type, addr)?; - b.ins() - .call_indirect(ctx.heap_refs.drop_shared_ref, helper_ptr, &[ptr]); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn call_copy_bytes( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - dst: cranelift_codegen::ir::Value, - src: cranelift_codegen::ir::Value, - len: cranelift_codegen::ir::Value, -) -> VmResult<()> { - let helper_ptr = iconst_ptr_from_addr(b, ctx.pointer_type, ctx.heap_addrs.copy_bytes)?; - b.ins() - .call_indirect(ctx.heap_refs.copy_bytes_ref, helper_ptr, &[dst, src, len]); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn call_zero_bytes( - b: &mut FunctionBuilder, - ctx: InlineEmitCtx, - dst: cranelift_codegen::ir::Value, - len: cranelift_codegen::ir::Value, -) -> VmResult<()> { - let helper_ptr = iconst_ptr_from_addr(b, ctx.pointer_type, ctx.heap_addrs.zero_bytes)?; - b.ins() - .call_indirect(ctx.heap_refs.free_buffer_ref, helper_ptr, &[dst, len]); - Ok(()) -} - -#[cfg(feature = "cranelift-jit")] -fn load_byte( - b: &mut FunctionBuilder, - ptr: cranelift_codegen::ir::Value, -) -> cranelift_codegen::ir::Value { - let byte = b.ins().load(types::I8, MemFlags::new(), ptr, 0); - b.ins().uextend(types::I32, byte) -} - -#[cfg(feature = "cranelift-jit")] -fn is_utf8_continuation_byte( - b: &mut FunctionBuilder, - byte: cranelift_codegen::ir::Value, -) -> cranelift_codegen::ir::Value { - let mask = b.ins().iconst(types::I32, 0xC0); - let masked = b.ins().band(byte, mask); - b.ins().icmp_imm(IntCC::Equal, masked, 0x80) -} - -#[cfg(feature = "cranelift-jit")] -fn utf8_char_width( - b: &mut FunctionBuilder, - pointer_type: cranelift_codegen::ir::Type, - byte: cranelift_codegen::ir::Value, -) -> cranelift_codegen::ir::Value { - let one = b.ins().iconst(pointer_type, 1); - let two = b.ins().iconst(pointer_type, 2); - let three = b.ins().iconst(pointer_type, 3); - let four = b.ins().iconst(pointer_type, 4); - let lt_80 = b.ins().icmp_imm(IntCC::UnsignedLessThan, byte, 0x80); - let lt_e0 = b.ins().icmp_imm(IntCC::UnsignedLessThan, byte, 0xE0); - let lt_f0 = b.ins().icmp_imm(IntCC::UnsignedLessThan, byte, 0xF0); - let tail = b.ins().select(lt_f0, three, four); - let wide = b.ins().select(lt_e0, two, tail); - b.ins().select(lt_80, one, wide) -} - -#[cfg(feature = "cranelift-jit")] -fn builtin_helper_tuple(builtin: BuiltinFunction, argc: u8) -> (i64, i64, i64, i64) { - ( - OP_BUILTIN_CALL, - i64::from(builtin.call_index()), - i64::from(argc), - 0, - ) -} diff --git a/src/vm/tests.rs b/src/vm/tests.rs index acf11f32..b7f0e0d1 100644 --- a/src/vm/tests.rs +++ b/src/vm/tests.rs @@ -317,6 +317,47 @@ fn aot_executes_script_callable_frames_without_interpreter_boundary() { assert!(!vm.aot_interpreter_boundary_hit); } +#[cfg(feature = "cranelift-jit")] +#[test] +fn aot_executes_typed_script_callable_parameter_equality_without_interpreter_boundary() { + let compiled = crate::compile_source( + r#" + fn is_zero(value: int) -> bool { value == 0 } + is_zero(0); + "#, + ) + .expect("typed equality source should compile"); + let mut vm = Vm::new(compiled.program); + vm.compile_aot().expect("aot compilation should succeed"); + assert_eq!( + vm.run().expect("aot execution should halt"), + VmStatus::Halted + ); + assert_eq!(vm.stack(), &[Value::Bool(true)]); + assert!(!vm.aot_interpreter_boundary_hit); +} + +#[cfg(feature = "cranelift-jit")] +#[test] +fn aot_executes_script_callable_bool_return_in_branch_without_interpreter_boundary() { + let compiled = crate::compile_source( + r#" + fn is_zero(value: int) -> bool { value == 0 } + let selected = if is_zero(0) => { 1 } else => { 2 }; + selected; + "#, + ) + .expect("typed branch source should compile"); + let mut vm = Vm::new(compiled.program); + vm.compile_aot().expect("aot compilation should succeed"); + assert_eq!( + vm.run().expect("aot execution should halt"), + VmStatus::Halted + ); + assert_eq!(vm.stack(), &[Value::Int(1)]); + assert!(!vm.aot_interpreter_boundary_hit); +} + #[cfg(feature = "cranelift-jit")] #[test] fn aot_executes_capturing_closure_without_interpreter_boundary() { diff --git a/src/vmbc.rs b/src/vmbc.rs index 9fee6388..77c23eb0 100644 --- a/src/vmbc.rs +++ b/src/vmbc.rs @@ -322,6 +322,26 @@ pub fn decode_program(bytes: &[u8]) -> Result { program.function_regions = function_regions; program.root_callable_bindings = root_callable_bindings; program.exported_callables = exported_callables; + let type_map_local_count = program + .type_map + .as_ref() + .map_or(0, |type_map| type_map.local_types.len()); + let callable_local_count = program + .root_callable_bindings + .iter() + .map(|binding| binding.local_slot as usize + 1) + .chain( + program + .exported_callables + .iter() + .map(|exported| exported.local_slot as usize + 1), + ) + .max() + .unwrap_or(0); + program.local_count = program + .local_count + .max(type_map_local_count) + .max(callable_local_count); Ok(program) } diff --git a/tests/jit/jit_tests.rs b/tests/jit/jit_tests.rs index af1f80e9..456398f0 100644 --- a/tests/jit/jit_tests.rs +++ b/tests/jit/jit_tests.rs @@ -1143,6 +1143,56 @@ fn trace_jit_compiles_hot_loop_and_is_dumpable() { } } +#[test] +fn trace_jit_diagnostics_preserve_public_snapshot_and_machine_code_toggle() { + let source = r#" + let mut i = 0; + while i < 20 { + i = i + 1; + } + i; + "#; + + let compiled = compile_source(source).expect("compile should succeed"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_config(JitConfig { + enabled: native_jit_supported(), + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + assert_eq!(vm.run().expect("vm should run"), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(20)]); + + let snapshot = vm.jit_snapshot(); + let dump_without_machine_code = vm.dump_jit_info_with_machine_code(false); + let dump_with_machine_code = vm.dump_jit_info_with_machine_code(true); + + assert_eq!( + snapshot.metrics.native_trace_exec_count, + vm.jit_native_exec_count(), + "snapshot metrics must retain the public native execution count" + ); + assert!( + dump_without_machine_code.contains("native codegen backend:"), + "diagnostics dump must retain its public native backend line" + ); + assert!( + !dump_without_machine_code.contains(" code:"), + "machine-code-disabled diagnostics dump must omit machine code" + ); + if native_jit_supported() { + assert!( + !snapshot.traces.is_empty(), + "expected a compiled trace, dump:\n{dump_with_machine_code}" + ); + assert!( + dump_with_machine_code.contains(" code:"), + "machine-code-enabled diagnostics dump must include native machine code" + ); + } +} + #[test] fn trace_jit_native_path_honors_fuel_metering() { let source = r#" @@ -4050,6 +4100,42 @@ fn trace_jit_supports_float_and_string_loops_through_ssa() { ); } +#[test] +fn trace_jit_boxes_scalar_before_native_to_string_helper() { + if !native_jit_supported() { + return; + } + + let source = r#" + let mut i = 0; + let mut out = ""; + while i < 4 { + out = i + ""; + i = i + 1; + } + out; + "#; + let compiled = compile_source(source).expect("scalar to_string fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + let status = vm.run().expect("scalar to_string fixture should run"); + assert_eq!(status, VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::string("3")]); + assert!( + vm.jit_snapshot() + .traces + .iter() + .any(|trace| trace.op_names().iter().any(|op| op == "to_string")), + "expected native to_string trace, dump:\n{}", + vm.dump_jit_info() + ); +} + #[test] fn trace_jit_supports_bytes_heavy_call_boundary_exits_without_fallback() { if !native_jit_supported() { diff --git a/tests/wire/wire_tests.rs b/tests/wire/wire_tests.rs index 5eff37ac..ddc3ecfc 100644 --- a/tests/wire/wire_tests.rs +++ b/tests/wire/wire_tests.rs @@ -65,6 +65,25 @@ fn wire_roundtrip_preserves_constants_and_code() { assert_eq!(decoded.type_map, program.type_map); } +#[test] +fn wire_roundtrip_recovers_locals_reserved_by_type_metadata() { + let local_count = 8; + let program = Program::new(Vec::new(), vec![vm::OpCode::Ret as u8]) + .with_local_count(local_count) + .with_type_map(TypeMap { + strict_types: true, + local_types: vec![ValueType::Unknown; local_count], + local_schemas: vec![None; local_count], + callable_slots: vec![false; local_count], + optional_slots: vec![false; local_count], + operand_types: HashMap::new(), + }); + + let encoded = encode_program(&program).expect("encode reserved locals"); + let decoded = decode_program(&encoded).expect("decode reserved locals"); + assert_eq!(decoded.local_count, local_count); +} + #[test] fn decode_rejects_invalid_magic_version_and_truncation() { let program = Program::new(vec![Value::Int(7)], vec![0x01]);