Delete the `cfg(not(parallel))` serial compiler by Noratrieb · Pull Request #132282 · rust-lang/rust

Noratrieb · 2024-10-28T18:03:52Z

Since it's inception a long time ago, the parallel compiler and its cfgs have been a maintenance burden. This was a necessary evil the allow iteration while not degrading performance because of synchronization overhead.

But this time is over. Thanks to the amazing work by the parallel working group (and the dyn sync crimes), the parallel compiler has now been fast enough to be shipped by default in nightly for quite a while now.
Stable and beta have still been on the serial compiler, because they can't use -Zthreads anyways.
But this is quite suboptimal:

the maintenance burden still sucks
we're not testing the serial compiler in nightly

Because of these reasons, it's time to end it. The serial compiler has served us well in the years since it was split from the parallel one, but it's over now.

Let the knight slay one head of the two-headed dragon!

#113349

Note that the default is still 1 thread, as more than 1 thread is still fairly broken.

cc @onur-ozkan to see if i did the bootstrap field removal correctly, @SparrowLii on the sync parts

rustbot · 2024-10-28T18:04:01Z

r? @cjgillot

rustbot has assigned @cjgillot.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-10-28T18:04:05Z

rustc_error_messages was changed

cc @davidtwco, @compiler-errors, @TaKO8Ki

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

rustbot · 2024-10-28T18:04:14Z

rustc_error_messages was changed

cc @davidtwco, @compiler-errors, @TaKO8Ki

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

compiler-errors · 2024-10-28T18:10:24Z

cc rust-lang/compiler-team#681

rustbot · 2024-10-28T18:39:31Z

This PR modifies config.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

bors · 2024-11-02T17:36:19Z

☔ The latest upstream changes (presumably #132513) made this pull request unmergeable. Please resolve the merge conflicts.

Noratrieb · 2024-11-02T19:46:15Z

@bors p=1 many changes

cjgillot

I trust you cleaned-up the cfgs correctly.
Some cases of missed formatting for macros there and there, if you don't mind fixing them.
Does this need some kind of heads-up to the team?
r=me if doesn't.

SparrowLii · 2024-11-08T08:10:56Z

From the discussion I think we can land this. @Noratrieb Could you help doing the format work as @cjgillot suggested?

Noratrieb · 2024-11-08T08:57:20Z

Yes, I will do it, but it may take me a few days. Feel free to push the fix to my branch and approve it if you want to.

Since it's inception a long time ago, the parallel compiler and its cfgs have been a maintenance burden. This was a necessary evil the allow iteration while not degrading performance because of synchronization overhead. But this time is over. Thanks to the amazing work by the parallel working group (and the dyn sync crimes), the parallel compiler has now been fast enough to be shipped by default in nightly for quite a while now. Stable and beta have still been on the serial compiler, because they can't use `-Zthreads` anyways. But this is quite suboptimal: - the maintenance burden still sucks - we're not testing the serial compiler in nightly Because of these reasons, it's time to end it. The serial compiler has served us well in the years since it was split from the parallel one, but it's over now. Let the knight slay one head of the two-headed dragon!

bors · 2024-11-12T15:15:08Z

⌛ Testing commit 505b8e1 with merge 6503543...

bors · 2024-11-12T18:03:19Z

☀️ Test successful - checks-actions
Approved by: cjgillot
Pushing 6503543 to master...

rust-timer · 2024-11-12T19:20:32Z

Finished benchmarking commit (6503543): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary -1.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.0%	[1.0%, 1.0%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-3.0%	[-3.0%, -3.0%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-1.0%	[-3.0%, 1.0%]	2

Cycles

Results (secondary 2.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.0%	[1.8%, 2.2%]	3
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 787.726s -> 784.558s (-0.40%)
Artifact size: 335.35 MiB -> 335.34 MiB (-0.00%)

tree-wide: parallel: Fully removed all `Lrc`, replaced with `Arc` tree-wide: parallel: Fully removed all `Lrc`, replaced with `Arc` This is continuation of rust-lang#132282 . I'm pretty sure I did everything right. In particular, I searched all occurrences of `Lrc` in submodules and made sure that they don't need replacement. There are other possibilities, through. We can define `enum Lrc<T> { Rc(Rc<T>), Arc(Arc<T>) }`. Or we can make `Lrc` a union and on every clone we can read from special thread-local variable. Or we can add a generic parameter to `Lrc` and, yes, this parameter will be everywhere across all codebase. So, if you think we should take some alternative approach, then don't merge this PR. But if it is decided to stick with `Arc`, then, please, merge. cc "Parallel Rustc Front-end" ( rust-lang#113349 ) r? SparrowLii `@rustbot` label WG-compiler-parallel

tree-wide: parallel: Fully removed all `Lrc`, replaced with `Arc` tree-wide: parallel: Fully removed all `Lrc`, replaced with `Arc` This is continuation of rust-lang/rust#132282 . I'm pretty sure I did everything right. In particular, I searched all occurrences of `Lrc` in submodules and made sure that they don't need replacement. There are other possibilities, through. We can define `enum Lrc<T> { Rc(Rc<T>), Arc(Arc<T>) }`. Or we can make `Lrc` a union and on every clone we can read from special thread-local variable. Or we can add a generic parameter to `Lrc` and, yes, this parameter will be everywhere across all codebase. So, if you think we should take some alternative approach, then don't merge this PR. But if it is decided to stick with `Arc`, then, please, merge. cc "Parallel Rustc Front-end" ( rust-lang/rust#113349 ) r? SparrowLii `@rustbot` label WG-compiler-parallel

A few cleanups after the removal of `cfg(not(parallel))` I noticed a few small things that are no longer needed after the removal of `cfg(not(parallel))` in rust-lang#132282. One of the later changes adjusts several imports, so viewing the changes individually is recommended. r? SparrowLii (or reroll)

Rollup merge of rust-lang#137922 - Zalathar:sharded, r=SparrowLii A few cleanups after the removal of `cfg(not(parallel))` I noticed a few small things that are no longer needed after the removal of `cfg(not(parallel))` in rust-lang#132282. One of the later changes adjusts several imports, so viewing the changes individually is recommended. r? SparrowLii (or reroll)

…l-queries, r=Mark-Simulacrum,onur-ozkan Remove `RUN_CHECK_WITH_PARALLEL_QUERIES` After rust-lang#132282, I'm pretty sure that this is simply useless? It just runs check with an empty config, lol. CC `@onur-ozkan` r? `@Noratrieb`

…lel-queries, r=Mark-Simulacrum,onur-ozkan Remove `RUN_CHECK_WITH_PARALLEL_QUERIES` After rust-lang#132282, I'm pretty sure that this is simply useless? It just runs check with an empty config, lol. CC `@onur-ozkan` r? `@Noratrieb`

Rollup merge of rust-lang#138452 - Kobzol:remove-run-check-with-parallel-queries, r=Mark-Simulacrum,onur-ozkan Remove `RUN_CHECK_WITH_PARALLEL_QUERIES` After rust-lang#132282, I'm pretty sure that this is simply useless? It just runs check with an empty config, lol. CC `@onur-ozkan` r? `@Noratrieb`

Rust crates are compiled in a serial chain on the critical path - core, bindings, kernel crates and then the drivers - each a rustc invocation. With CONFIG_RUST set and every rust driver enabled, the entire chain takes around 25 seconds on a 128-thread machine, with the associated C code taking 24 seconds. Each time a change is made to any of the bindings, it triggers a rebuild. Use the -Zthreads option to have rustc perform builds in parallel. This has been available since rust 1.84 (rust-lang/rust#132282), and the kernel requires rust 1.85 or above. A future -j/--jobs option is planned for rustc:(rust-lang/compiler-team#1005), so check to see if this available and if so use it. If the user's rustc supports neither, then it falls back gracefully and neither are used. The threads are taken from make's jobserver, so a parallel build is not oversubscribed. It was found that benefits level off at 8 threads (16 was found to be around the same, and 32 slower). Generated output was confirmed byte-for-byte identical. Observed build time changes (using rustc 1.98, clang): before after core.o 7.7s 4.9s bindings.o 4.3s 3.1s kernel.o 2.0s 1.4s clean build 39.3s 34.6s touch rust/kernel/lib.rs 12.9s 12.1s touch rust/bindings/bindings_helper.h 19.5s 17.5s Whole build, 128-thread Threadripper 9980X, best of N runs: before after delta ------------------------------- x86 defconfig+RUST, clean 41.0s 36.4s -4.6s (-11%) x86 defconfig+RUST, touch lib.rs 8.3s 7.7s -0.56s (-7%) Assisted-by: LLM Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>

rustbot assigned cjgillot Oct 28, 2024

Noratrieb force-pushed the it-is-the-end-of-serial branch from fae56ab to 2e9696f Compare October 28, 2024 18:04

This comment has been minimized.

Sign in to view

Noratrieb force-pushed the it-is-the-end-of-serial branch from 2e9696f to 9c4a078 Compare October 28, 2024 18:23

onur-ozkan reviewed Oct 28, 2024

View reviewed changes

Comment thread src/bootstrap/src/core/config/config.rs

Noratrieb force-pushed the it-is-the-end-of-serial branch from 9c4a078 to 2c453de Compare October 28, 2024 18:39

onur-ozkan approved these changes Oct 28, 2024

View reviewed changes

Noratrieb force-pushed the it-is-the-end-of-serial branch from 2c453de to 09d2f62 Compare November 2, 2024 19:45

cjgillot reviewed Nov 3, 2024

View reviewed changes

Comment thread compiler/rustc_data_structures/src/marker.rs Outdated

SparrowLii approved these changes Nov 7, 2024

View reviewed changes

SparrowLii mentioned this pull request Nov 8, 2024

Update parallel-rustc.md rust-lang/rustc-dev-guide#1926

Merged

clubby789 force-pushed the it-is-the-end-of-serial branch from 09d2f62 to 505b8e1 Compare November 12, 2024 13:39

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Nov 12, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 12, 2024

bors merged commit 6503543 into rust-lang:master Nov 12, 2024

rustbot added this to the 1.84.0 milestone Nov 12, 2024

This was referenced Nov 12, 2024

Only use the new DepNode hashmap for anonymous nodes. #109050

Closed

Handle rustc_middle cases of rustc::potential_query_instability lint #131160

Merged

Handle rustc_query_system cases of rustc::potential_query_instability lint #131200

Merged

Noratrieb deleted the it-is-the-end-of-serial branch November 12, 2024 18:50

This was referenced Dec 12, 2024

./x.py test compiler/rustc_abi no longer works #129727

Closed

x test compiler/rustc_borrowck fails and complains about undeclared rayon crate #131093

Closed

safinaskar mentioned this pull request Feb 3, 2025

tree-wide: parallel: Fully removed all Lrc, replaced with Arc #136471

Merged

workingjubilee mentioned this pull request Feb 10, 2025

dev-lang/rust: use/set RPATH instead of ld.so.conf gentoo/gentoo#40500

Closed

Zalathar mentioned this pull request Mar 3, 2025

A few cleanups after the removal of cfg(not(parallel)) #137922

Merged

Kobzol mentioned this pull request Mar 13, 2025

Remove RUN_CHECK_WITH_PARALLEL_QUERIES #138452

Merged

Enselic mentioned this pull request Oct 18, 2025

UI test failures with parallel-compiler=true #75760

Closed

Uh oh!

Conversation

Noratrieb commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Oct 28, 2024

Uh oh!

rustbot commented Oct 28, 2024

Uh oh!

rustbot commented Oct 28, 2024

Uh oh!

compiler-errors commented Oct 28, 2024

Uh oh!

This comment has been minimized.

Uh oh!

rustbot commented Oct 28, 2024

Uh oh!

bors commented Nov 2, 2024

Uh oh!

Noratrieb commented Nov 2, 2024

Uh oh!

cjgillot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparrowLii commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Noratrieb commented Nov 8, 2024

Uh oh!

bors commented Nov 12, 2024

Uh oh!

bors commented Nov 12, 2024

Uh oh!

rust-timer commented Nov 12, 2024

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Noratrieb commented Oct 28, 2024 •

edited

Loading

SparrowLii commented Nov 8, 2024 •

edited

Loading