diff --git a/changelog.d/10186-fast-emit-budget-x86-64.md b/changelog.d/10186-fast-emit-budget-x86-64.md new file mode 100644 index 0000000000..aadb6d5fde --- /dev/null +++ b/changelog.d/10186-fast-emit-budget-x86-64.md @@ -0,0 +1,18 @@ +Raise the optimized machine-pipeline budget on x86-64 targets from 100,000 to +600,000 post-optimization instructions. The budget's fallback demotes the whole +codegen unit to LLVM's O0 machine pipeline, not just the over-budget function +(LLVM selects the pipeline per module, and `optnone` on one function frees its +siblings without bounding regalloc time or memory), so on the OpenCode build +61 over-budget functions dragged 140 MiB of ordinary sibling code into O0 +emission with them: 42 % of the binary's text. With the corpus's whole giant +population admitted, the three specimen modules lose 25 % / 42 % / 69 % of +their `.text`, runtime instructions and RSS are unchanged, and compile cost +stays bounded (+13–28 % wall, ~2.5 GB peak on the worst specimen). + +aarch64/arm64 and every other unmeasured target keep the old 100,000 ceiling: +the two observations that set it (a 100k-instruction function past ~10 GiB +RSS; a 277k-instruction function >16 min in register allocation) are arm64 +and have not been re-measured. `PERRY_LL_FAST_EMIT_MAX_INSTRS` still overrides +on every target; on x86-64 `=100000` reproduces the previous output byte for +byte. The fallback diagnostic now names every over-budget function, widest +first, and says how many functions in its unit are demoted alongside it. diff --git a/crates/perry-codegen/src/inprocess.rs b/crates/perry-codegen/src/inprocess.rs index 7b4affa613..f8fe02b633 100644 --- a/crates/perry-codegen/src/inprocess.rs +++ b/crates/perry-codegen/src/inprocess.rs @@ -352,9 +352,12 @@ pub struct UnitCodegenStats { /// Functions stamped `"disable-tail-calls"` because their alloca-walk /// estimate exceeded [`DEFAULT_TRE_MAX_ALLOCA_WALK`] (#8883). pub tail_call_elim_skipped: Vec, - /// The widest function which made this unit use LLVM's bounded O0 machine - /// pipeline after completing the requested IR optimization pipeline. - pub fast_emit_fallback: Option, + /// Every function over the target's ceiling (see + /// [`default_fast_emit_max_instrs`]), widest first — + /// the ones which made this unit use LLVM's bounded O0 machine pipeline + /// after completing the requested IR optimization pipeline. Empty when + /// the unit kept the optimized machine pipeline. + pub fast_emit_fallbacks: Vec, } fn function_instruction_count(function: inkwell::values::FunctionValue<'_>) -> usize { @@ -400,20 +403,85 @@ fn module_instruction_census( /// live-interval and register-allocation pipeline for a unit containing an /// extreme generated function. /// -/// The threshold is bracketed by real arm64/LLVM 22 measurements. Machine-IR -/// expansion depends on CFG shape, so raw IR size is deliberately only a -/// conservative guard: one 161k-instruction function emitted normally in -/// ~19s, while a different 100,152-instruction Claude Code 2.1.259 function -/// grew past ~10 GiB RSS in the optimized machine pipeline. The same function -/// emitted through an O0 target machine in 6s. Another 277k-instruction async -/// state-machine function remained in LiveIntervals / register allocation for -/// more than 16 minutes at ~10 GiB RSS; its already-Os-optimized IR emitted -/// through an O0 target machine in 3.5s at ~550 MiB RSS. 100k is immediately -/// below the smallest observed pathological case. +/// **The demotion is a whole-unit act, so the budget must not be set where +/// ordinary functions pay for it.** A `TargetMachine`'s optimization level is +/// a per-module property: LLVM has no per-function escape from the optimized +/// machine pipeline (`optnone` reaches instruction selection and the optional +/// machine passes, but *not* LiveIntervals or the greedy register allocator — +/// measured below), so every ordinary function sharing the unit with one +/// extreme function is emitted through the O0 machine pipeline too. /// -/// `PERRY_LL_FAST_EMIT_MAX_INSTRS=` raises or lowers the ceiling; `0` / -/// `off` disables the fallback. -const DEFAULT_FAST_EMIT_MAX_INSTRS: usize = 100_000; +/// Measured on `@babel/parser`'s unit 0, LLVM 22 / x86-64 / `-Os` IR pipeline: +/// one 227,108-instruction closure (163,100 of those are `gc.relocate`) and +/// 282 ordinary siblings, each arm emitting the same post-`default` IR: +/// +/// | machine pipeline | unit `.text` | the closure | its 282 siblings | `llc` | peak RSS | +/// |---|---|---|---|---|---| +/// | optimized (`-O2`) | 1,689,851 B | 241,218 B | 1.382 MiB | 10.0 s | 464 MiB | +/// | O0 (this fallback) | 5,862,077 B | 2,253,658 B | 3.441 MiB | 3.9 s | 499 MiB | +/// | `optnone` on the closure only | 2,070,326 B | 621,693 B | 1.382 MiB | 9.5 s | 518 MiB | +/// | the same unit *without* the closure | 1,448,633 B | — | 1.382 MiB | 6.4 s | 208 MiB | +/// +/// So the siblings are pure loss: the fallback costs them 2.06 MiB of machine +/// code (168 of 282 functions change) to save ~6 s, and their emitted code is +/// byte-for-byte what a unit without the extreme function produces as soon as +/// the unit keeps the optimized pipeline. The `optnone` row is why this is a +/// budget and not a per-function demotion: it frees the siblings but bounds +/// neither time (9.5 s of 10.0 s) nor memory (518 MiB — *above* the -O2 arm), +/// because the greedy allocator still runs on the demoted function. +/// +/// On x86-64 the ceiling is therefore set above the whole measured +/// population of extreme generated functions rather than immediately below +/// the smallest pathological one. On the OpenCode corpus 60 of the 61 +/// functions past the old 100k ceiling are under 600k (median 153,455; +/// largest 982,912), and the +/// largest one measured end-to-end — the 522,756-instruction `mime` +/// `types/other.ts` constructor — emits through the optimized machine pipeline +/// in 704 s at 2.26 GB peak RSS, against 393 s at 2.34 GB demoted, for a +/// module `.text` of 13.72 MB against 36.97 MB. +/// +/// **Every measurement above is x86-64, so only x86-64 gets the raised +/// ceiling.** Machine-IR expansion depends on CFG shape *and* on the target's +/// instruction selection and register allocation, and the two observations +/// that set the 100k ceiling in the first place are both arm64/LLVM 22: a +/// 100,152-instruction Claude Code 2.1.259 function grew past ~10 GiB RSS in +/// the optimized machine pipeline (6 s through an O0 target machine), and a +/// 277k-instruction async state-machine function sat in LiveIntervals / +/// register allocation for more than 16 minutes at ~10 GiB (3.5 s at ~550 MiB +/// demoted). Both postdate #8679's shadow-frame retry, so they are current +/// observations, not stale ones — and both sit *inside* the 600k band. Every +/// CI runner and developer build here is macOS arm64, so raising the ceiling +/// there on x86-64 evidence would trade a measured size win for an unmeasured +/// 10 GiB compile. aarch64/arm64 — and every other target nobody has measured +/// — therefore keep 100k until someone measures them the way `x86_64` was +/// measured above, at which point `default_fast_emit_max_instrs` grows a +/// match arm and this comment grows a row. +/// +/// `PERRY_LL_FAST_EMIT_MAX_INSTRS=` raises or lowers the ceiling on every +/// target; `0` / `off` disables the fallback. On x86-64, +/// `PERRY_LL_FAST_EMIT_MAX_INSTRS=100000` reproduces the old behaviour +/// byte-for-byte. +const DEFAULT_FAST_EMIT_MAX_INSTRS_X86_64: usize = 600_000; + +/// The ceiling for every target whose optimized machine pipeline has not been +/// measured against a corpus of extreme generated functions — including +/// aarch64/arm64, where the two pathological observations quoted in +/// [`DEFAULT_FAST_EMIT_MAX_INSTRS_X86_64`] were made. +const DEFAULT_FAST_EMIT_MAX_INSTRS_UNMEASURED: usize = 100_000; + +/// The ceiling for the target this unit is being emitted *for* — not the host. +/// A cross-compile from an x86-64 box to arm64 runs arm64's instruction +/// selection and register allocator, so it is arm64's ceiling that applies. +fn default_fast_emit_max_instrs(effective_target: &str) -> usize { + let arch = effective_target + .split('-') + .next() + .unwrap_or(effective_target); + match arch { + "x86_64" | "x86_64h" | "amd64" => DEFAULT_FAST_EMIT_MAX_INSTRS_X86_64, + _ => DEFAULT_FAST_EMIT_MAX_INSTRS_UNMEASURED, + } +} #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum FastEmitBudget { @@ -421,19 +489,20 @@ enum FastEmitBudget { Cap(usize), } -fn parse_fast_emit_budget(value: Option<&str>) -> FastEmitBudget { +fn parse_fast_emit_budget(value: Option<&str>, effective_target: &str) -> FastEmitBudget { + let default = || FastEmitBudget::Cap(default_fast_emit_max_instrs(effective_target)); match value.map(str::trim) { - None | Some("") => FastEmitBudget::Cap(DEFAULT_FAST_EMIT_MAX_INSTRS), + None | Some("") => default(), Some("0") | Some("off") | Some("false") => FastEmitBudget::Off, Some(v) => match v.parse::() { Ok(0) => FastEmitBudget::Off, Ok(n) => FastEmitBudget::Cap(n), - Err(_) => FastEmitBudget::Cap(DEFAULT_FAST_EMIT_MAX_INSTRS), + Err(_) => default(), }, } } -fn fast_emit_budget() -> FastEmitBudget { +fn fast_emit_budget(effective_target: &str) -> FastEmitBudget { #[cfg(test)] if let Some(budget) = TEST_FAST_EMIT_BUDGET.with(std::cell::Cell::get) { return budget; @@ -442,6 +511,7 @@ fn fast_emit_budget() -> FastEmitBudget { std::env::var("PERRY_LL_FAST_EMIT_MAX_INSTRS") .ok() .as_deref(), + effective_target, ) } @@ -456,23 +526,34 @@ thread_local! { /// other LLVM tests in this binary. #[cfg(test)] fn with_test_fast_emit_budget(cap: usize, run: impl FnOnce() -> T) -> T { + with_test_fast_emit_budget_value(FastEmitBudget::Cap(cap), run) +} + +/// [`with_test_fast_emit_budget`] for a budget that is not a cap — the arm +/// that proves what an undemoted unit emits. +#[cfg(test)] +fn with_test_fast_emit_budget_value(budget: FastEmitBudget, run: impl FnOnce() -> T) -> T { struct Restore(Option); impl Drop for Restore { fn drop(&mut self) { TEST_FAST_EMIT_BUDGET.with(|budget| budget.set(self.0)); } } - let old = TEST_FAST_EMIT_BUDGET.replace(Some(FastEmitBudget::Cap(cap))); + let old = TEST_FAST_EMIT_BUDGET.replace(Some(budget)); let _restore = Restore(old); run() } -/// The extreme function which selected bounded machine-code emission. +/// One extreme function which selected bounded machine-code emission, and how +/// many defined functions in its unit are demoted along with it. #[derive(Debug, Clone, PartialEq, Eq)] pub struct FastEmitFallback { pub name: String, pub instructions: usize, pub cap: usize, + /// Defined functions in the unit — the size of the collateral, since the + /// machine pipeline is selected per module and not per function. + pub unit_functions: usize, } impl std::fmt::Display for FastEmitFallback { @@ -480,43 +561,57 @@ impl std::fmt::Display for FastEmitFallback { write!( f, "`{}` has {} instructions after IR optimization, above the optimized machine-pipeline \ - budget {}; keeping the requested IR optimization, then emitting this unit through \ - LLVM's O0 machine pipeline to bound instruction selection, live intervals and \ - register allocation. Override with PERRY_LL_FAST_EMIT_MAX_INSTRS= (raise) or \ - =0 (disable).", - self.name, self.instructions, self.cap + budget {}; keeping the requested IR optimization, then emitting this unit — all {} \ + of its defined functions, not only this one — through LLVM's O0 machine pipeline to \ + bound instruction selection, live intervals and register allocation. LLVM selects \ + that pipeline per module, so the siblings are demoted too and grow: shrinking this \ + function is what lifts the whole unit back. Override with \ + PERRY_LL_FAST_EMIT_MAX_INSTRS= (raise) or =0 (disable).", + self.name, self.instructions, self.cap, self.unit_functions ) } } -fn fast_emit_fallback( +/// Every defined function over `budget`, widest first. +/// +/// The decision is per function; the consequence cannot be (see +/// [`DEFAULT_FAST_EMIT_MAX_INSTRS_X86_64`]), which is why every offender is +/// returned +/// rather than only the widest: the compile log then names each function that +/// has to shrink before the unit can keep the optimized machine pipeline, +/// instead of naming one and re-reporting a new widest on the next build. +fn fast_emit_fallbacks( module: &inkwell::module::Module<'_>, budget: FastEmitBudget, -) -> Option { +) -> Vec { let cap = match budget { - FastEmitBudget::Off => return None, + FastEmitBudget::Off => return Vec::new(), FastEmitBudget::Cap(cap) => cap, }; - let mut widest: Option = None; + let mut defined = 0usize; + let mut over: Vec<(String, usize)> = Vec::new(); let mut function = module.get_first_function(); while let Some(f) = function { if f.count_basic_blocks() > 0 { + defined += 1; let instructions = function_instruction_count(f); - if instructions > cap - && widest - .as_ref() - .is_none_or(|current| instructions > current.instructions) - { - widest = Some(FastEmitFallback { - name: f.get_name().to_string_lossy().into_owned(), - instructions, - cap, - }); + if instructions > cap { + over.push((f.get_name().to_string_lossy().into_owned(), instructions)); } } function = f.get_next_function(); } - widest + // Widest first, ties by name: one deterministic order for the log and the + // per-unit report, whatever order LLVM holds the functions in. + over.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0))); + over.into_iter() + .map(|(name, instructions)| FastEmitFallback { + name, + instructions, + cap, + unit_functions: defined, + }) + .collect() } /// Instruction budget for ONE function after `rewrite-statepoints-for-gc`. diff --git a/crates/perry-codegen/src/inprocess/optimize_emit.rs b/crates/perry-codegen/src/inprocess/optimize_emit.rs index a0f1e240ec..e7b879b809 100644 --- a/crates/perry-codegen/src/inprocess/optimize_emit.rs +++ b/crates/perry-codegen/src/inprocess/optimize_emit.rs @@ -175,18 +175,25 @@ pub(super) fn optimize_and_emit( // register allocation. Use an O0 target machine only for final emission // of that unit; ordinary units keep `tm`, and the optimized IR is not // rebuilt or demoted. + // + // The selection is per function, the emission cannot be — a TargetMachine + // carries one optimization level for the whole module, and `optnone` does + // not reach LiveIntervals or the register allocator. Every ordinary + // function in the unit is demoted with the offender, which is why the + // budget sits above the measured population of extreme functions and why + // the log below names each of them. let fast_emit = if opt == '0' { - None + Vec::new() } else { - fast_emit_fallback(module, fast_emit_budget()) + fast_emit_fallbacks(module, fast_emit_budget(effective_target)) }; - if let Some(fallback) = &fast_emit { + for fallback in &fast_emit { eprintln!("perry: {fallback}"); } if let Some(stats) = stats.as_deref_mut() { - stats.fast_emit_fallback = fast_emit.clone(); + stats.fast_emit_fallbacks = fast_emit.clone(); } - let fast_tm = if fast_emit.is_some() { + let fast_tm = if !fast_emit.is_empty() { Some( target .create_target_machine( @@ -1014,27 +1021,47 @@ entry: ); } + /// Spellings, on both an x86-64 and an arm64 target: only the default + /// differs between them, and an explicit setting wins on either. #[test] fn fast_emit_budget_spellings() { - assert_eq!( - parse_fast_emit_budget(None), - FastEmitBudget::Cap(DEFAULT_FAST_EMIT_MAX_INSTRS) - ); - assert_eq!( - parse_fast_emit_budget(Some("")), - FastEmitBudget::Cap(DEFAULT_FAST_EMIT_MAX_INSTRS) - ); - assert_eq!(parse_fast_emit_budget(Some("0")), FastEmitBudget::Off); - assert_eq!(parse_fast_emit_budget(Some("off")), FastEmitBudget::Off); - assert_eq!(parse_fast_emit_budget(Some("false")), FastEmitBudget::Off); - assert_eq!( - parse_fast_emit_budget(Some(" 250000 ")), - FastEmitBudget::Cap(250_000) - ); - assert_eq!( - parse_fast_emit_budget(Some("lots")), - FastEmitBudget::Cap(DEFAULT_FAST_EMIT_MAX_INSTRS) - ); + for (target, default) in [ + ("x86_64-unknown-linux-gnu", 600_000), + ("arm64-apple-darwin", 100_000), + ] { + assert_eq!( + parse_fast_emit_budget(None, target), + FastEmitBudget::Cap(default), + "{target}" + ); + assert_eq!( + parse_fast_emit_budget(Some(""), target), + FastEmitBudget::Cap(default), + "{target}" + ); + assert_eq!( + parse_fast_emit_budget(Some("0"), target), + FastEmitBudget::Off + ); + assert_eq!( + parse_fast_emit_budget(Some("off"), target), + FastEmitBudget::Off + ); + assert_eq!( + parse_fast_emit_budget(Some("false"), target), + FastEmitBudget::Off + ); + assert_eq!( + parse_fast_emit_budget(Some(" 250000 "), target), + FastEmitBudget::Cap(250_000), + "an explicit ceiling wins on every target" + ); + assert_eq!( + parse_fast_emit_budget(Some("lots"), target), + FastEmitBudget::Cap(default), + "{target}" + ); + } } /// Two functions: `wide` has 4 allocas across 9 instructions (estimate @@ -1132,32 +1159,47 @@ entry: } /// Selection is per function, the boundary is inclusive, declarations do - /// not count, and the diagnostic names the widest violating function. + /// not count, every violator is reported (widest first) and the + /// diagnostic names the function and the size of the collateral. #[test] fn fast_emit_budget_selects_only_above_the_boundary() { let context = Context::create(); let module = parse_ir_text(&context, alloca_walk_fixture(), "fast_emit_fixture") .expect("fixture parses"); - assert!(fast_emit_fallback(&module, FastEmitBudget::Off).is_none()); - assert!(fast_emit_fallback(&module, FastEmitBudget::Cap(9)).is_none()); + assert!(fast_emit_fallbacks(&module, FastEmitBudget::Off).is_empty()); + assert!(fast_emit_fallbacks(&module, FastEmitBudget::Cap(9)).is_empty()); - let fallback = fast_emit_fallback(&module, FastEmitBudget::Cap(8)) - .expect("wide is one instruction over the budget"); + let fallbacks = fast_emit_fallbacks(&module, FastEmitBudget::Cap(8)); assert_eq!( - fallback, - FastEmitFallback { + fallbacks, + vec![FastEmitFallback { name: "wide".to_string(), instructions: 9, cap: 8, - } + // `narrow` is under the cap and `sink` is a declaration; both + // are still emitted by the demoted machine pipeline. + unit_functions: 2, + }], + "only the function over the budget is selected" ); - let message = fallback.to_string(); + + // Every violator, widest first: one build names the whole set that + // has to shrink instead of re-reporting a new widest each time. + let both = fast_emit_fallbacks(&module, FastEmitBudget::Cap(2)); + let named: Vec<(&str, usize)> = both + .iter() + .map(|f| (f.name.as_str(), f.instructions)) + .collect(); + assert_eq!(named, [("wide", 9), ("narrow", 3)]); + + let message = fallbacks[0].to_string(); for needle in [ "`wide`", "9 instructions", "budget 8", "requested IR optimization", "O0 machine pipeline", + "all 2 of its defined functions", "PERRY_LL_FAST_EMIT_MAX_INSTRS", ] { assert!( @@ -1167,6 +1209,60 @@ entry: } } + /// The ceiling is a calibration, not a round number, and it is only as + /// wide as the target it was measured on. x86-64 must sit above the + /// measured population of extreme generated functions (OpenCode corpus: + /// 60 of the 61 functions past the old 100k ceiling are under 600k, and + /// the largest one measured end to end is `mime` `types/other.ts`'s + /// 522,756-instruction constructor, which emits through the optimized + /// machine pipeline in 704 s at 2.26 GB) and below the RS4GC + /// relocation-fan-out budget, which is the backstop that runs first and + /// re-lowers rather than demotes. Every unmeasured target — aarch64 + /// above all, where both pathological observations were made — keeps + /// 100k. Raising one of those is a measurement, not an edit. + #[test] + fn fast_emit_budget_admits_the_measured_giant_population_on_measured_targets_only() { + assert_eq!(DEFAULT_FAST_EMIT_MAX_INSTRS_X86_64, 600_000); + assert!( + DEFAULT_FAST_EMIT_MAX_INSTRS_X86_64 > 522_756, + "the largest measured-affordable function must not be demoted" + ); + assert!( + DEFAULT_FAST_EMIT_MAX_INSTRS_X86_64 < DEFAULT_RS4GC_MAX_INSTRS, + "a function this wide is handled by the RS4GC budget first" + ); + assert_eq!(DEFAULT_FAST_EMIT_MAX_INSTRS_UNMEASURED, 100_000); + + for target in [ + "x86_64-unknown-linux-gnu", + "x86_64-apple-darwin", + "x86_64h-apple-darwin", + ] { + assert_eq!( + default_fast_emit_max_instrs(target), + 600_000, + "{target} is measured" + ); + } + // The host is irrelevant: a cross-compile runs the *target's* ISel and + // register allocator, so these keep the unmeasured ceiling even from an + // x86-64 box. + for target in [ + "arm64-apple-darwin", + "aarch64-apple-darwin", + "aarch64-unknown-linux-gnu", + "arm64_32-apple-watchos", + "i686-unknown-linux-gnu", + "", + ] { + assert_eq!( + default_fast_emit_max_instrs(target), + 100_000, + "{target:?} has not been measured" + ); + } + } + /// A self-recursive tail call that TailCallElim turns into a loop at /// the pinned LLVM: with no attribute the recursive `call` disappears, /// with `"disable-tail-calls"="true"` (exactly what the budget stamps) @@ -1269,7 +1365,7 @@ entry: /// A tiny test cap proves the shipped path records and successfully uses /// the second, O0 target machine only after running the requested Os IR /// pipeline. The production threshold is pinned by the parser test and - /// the real Claude-Code measurement in its constant's documentation. + /// the real measurements in its constant's documentation. #[test] fn fast_emit_budget_is_applied_by_the_shipped_pipeline() { global_init(&[]); @@ -1290,8 +1386,10 @@ entry: .expect("the already-optimized module emits through the bounded target machine"); assert!(!object.is_empty()); let fallback = stats - .fast_emit_fallback - .expect("the shipped path must report the selected fallback"); + .fast_emit_fallbacks + .first() + .expect("the shipped path must report the selected fallback") + .clone(); assert_eq!(fallback.name, "wide"); assert!(fallback.instructions > fallback.cap); assert_eq!(fallback.cap, 1); @@ -1311,6 +1409,133 @@ entry: ) }) .expect("-O0 emits"); - assert!(stats.fast_emit_fallback.is_none()); + assert!(stats.fast_emit_fallbacks.is_empty()); + } + + /// One function over the budget, one ordinary function that is nowhere + /// near it, and one external callee so nothing folds away. `narrow` + /// holds three values across three calls, which is what makes its + /// machine code differ between the optimized and the O0 register + /// allocators. + fn sibling_cost_fixture(with_wide: bool) -> String { + let wide = r#" +define i64 @wide(i64 %n) { +entry: + %a = call i64 @src(i64 %n) + %b = call i64 @src(i64 %a) + %c = call i64 @src(i64 %b) + %d = call i64 @src(i64 %c) + %s = add i64 %a, %b + %t = add i64 %s, %c + %u = add i64 %t, %d + ret i64 %u +} +"#; + format!( + r#" +declare i64 @src(i64) + +define i64 @narrow(i64 %x, i64 %y) {{ +entry: + %a = call i64 @src(i64 %x) + %b = call i64 @src(i64 %y) + %c = call i64 @src(i64 %a) + %s = add i64 %a, %b + %t = add i64 %s, %c + ret i64 %t +}} +{}"#, + if with_wide { wide } else { "" } + ) + } + + /// The assembly of one function, from its label to the end of its body. + /// Tolerates ELF (`narrow:` / `.size`) and Mach-O (`_narrow:`) spelling. + fn function_assembly(asm: &str, name: &str) -> String { + let label_elf = format!("{name}:"); + let label_macho = format!("_{name}:"); + let mut body: Vec<&str> = Vec::new(); + let mut inside = false; + for line in asm.lines() { + let trimmed = line.trim(); + if !inside { + inside = trimmed == label_elf || trimmed == label_macho; + continue; + } + let next_symbol = trimmed.ends_with(':') + && !trimmed.starts_with('.') + && !trimmed.starts_with('L') + && !trimmed.contains(' '); + if trimmed.starts_with(".size") || trimmed == ".cfi_endproc" || next_symbol { + break; + } + body.push(line); + } + assert!( + body.len() > 3, + "no body extracted for `{name}` — the assertion below would be vacuous:\n{asm}" + ); + body.join("\n") + } + + fn emit_assembly(ir: &str, module_name: &str, budget: FastEmitBudget) -> (String, Vec) { + global_init(&[]); + let target = crate::codegen::default_target_triple(); + let context = Context::create(); + let module = parse_ir_text(&context, ir, module_name).expect("fixture parses"); + let mut stats = UnitCodegenStats::default(); + let asm = with_test_fast_emit_budget_value(budget, || { + optimize_and_emit_module_with_stats( + &module, + &target, + &["-Os".into(), "-S".into()], + false, + Some(&mut stats), + ) + }) + .expect("the fixture emits"); + ( + String::from_utf8(asm).expect("LLVM emits UTF-8 assembly"), + stats + .fast_emit_fallbacks + .iter() + .map(|f| f.name.clone()) + .collect(), + ) + } + + /// What the budget actually costs, and why it is calibrated above the + /// measured population instead of below the smallest pathological case: + /// the demotion is a whole-unit act. An ordinary function emits the same + /// machine code whether or not an extreme function shares its unit — but + /// only while the unit keeps the optimized machine pipeline. Cross the + /// budget and that ordinary function's code changes too, without ever + /// having been over any budget itself. + #[test] + fn the_budget_is_what_makes_ordinary_siblings_pay() { + let with_wide = sibling_cost_fixture(true); + let alone = sibling_cost_fixture(false); + + let (undemoted, none) = emit_assembly(&with_wide, "sibling_cost_ok", FastEmitBudget::Off); + assert!(none.is_empty(), "this arm must not demote: {none:?}"); + let (solo, _) = emit_assembly(&alone, "sibling_cost_alone", FastEmitBudget::Off); + assert_eq!( + function_assembly(&undemoted, "narrow"), + function_assembly(&solo, "narrow"), + "an undemoted unit emits an ordinary function exactly as a unit without the \ + extreme function does" + ); + + // Same module, same IR pipeline, budget crossed: `narrow` is not over + // it and is compiled differently anyway. + let (demoted, over) = + emit_assembly(&with_wide, "sibling_cost_demoted", FastEmitBudget::Cap(7)); + assert_eq!(over, ["wide"], "only `wide` is over the budget"); + assert_ne!( + function_assembly(&demoted, "narrow"), + function_assembly(&undemoted, "narrow"), + "if the demotion did not reach `narrow`, LLVM grew a per-function escape from the \ + optimized machine pipeline and this budget can move back down" + ); } }