Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,11 @@ _UpgradeReport_Files/
tools/*/*.i
tools/*/*.i.tmp

# === Rules for pgo artifacts ===
*.profdata
*.profraw
*.gcda

# === Rules for release artifacts ===
/*.tar.*
/*.pkg
Expand Down
30 changes: 30 additions & 0 deletions common.gypi
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
'msvs_multi_core_compile': '0', # we do enable multicore compiles, but not using the V8 way
'enable_pgo_generate%': '0',
'enable_pgo_use%': '0',
'pgo_profile%': '',
'clang_profile_lib%': '',
'python%': 'python',
'emulator%': [],
Expand Down Expand Up @@ -193,6 +194,7 @@
}],
['clang==1', {
'lto': ' -flto ', # Clang
'pgo_use': '-fprofile-use=<(pgo_profile)',
}, {
'lto': ' -flto=4 -ffat-lto-objects ', # GCC
}],
Expand Down Expand Up @@ -247,6 +249,34 @@
},],
],
},],
['OS=="mac"', {
'conditions': [
['enable_pgo_generate=="true"', {
'xcode_settings': {
'OTHER_CFLAGS': ['<(pgo_generate)'],
},
'target_conditions': [
['_type!="static_library"', {
'xcode_settings': {
'OTHER_LDFLAGS': ['<(pgo_generate)'],
},
}],
],
}],
['enable_pgo_use=="true"', {
'xcode_settings': {
'OTHER_CFLAGS': ['<(pgo_use)'],
},
'target_conditions': [
['_type!="static_library"', {
'xcode_settings': {
'OTHER_LDFLAGS': ['<(pgo_use)'],
},
}],
],
}],
],
}],
['OS=="win"', {
'conditions': [
['enable_lto=="true"', {
Expand Down
36 changes: 23 additions & 13 deletions configure.py
Original file line number Diff line number Diff line change
Expand Up @@ -208,14 +208,15 @@
dest="enable_pgo_generate",
default=None,
help="Enable profiling with pgo of a binary. This feature is only available "
"on linux with gcc and g++ 5.4.1 or newer and on windows.")
"on linux with GCC or Clang, on macOS with Clang, and on windows.")

parser.add_argument("--enable-pgo-use",
action="store_true",
dest="enable_pgo_use",
default=None,
help="Enable use of the profile generated with --enable-pgo-generate. This "
"feature is only available on linux with gcc and g++ 5.4.1 or newer and on windows.")
"feature is only available on linux with GCC or Clang, on macOS "
"with Clang, and on windows.")

parser.add_argument("--enable-lto",
action="store_true",
Expand Down Expand Up @@ -2004,18 +2005,9 @@ def configure_node(o):
else:
o['variables']['node_enable_v8_vtunejit'] = 'false'

if (flavor != 'linux' and flavor != 'win') and (options.enable_pgo_generate or options.enable_pgo_use):
if flavor not in ('linux', 'mac', 'win') and (options.enable_pgo_generate or options.enable_pgo_use):
raise Exception(
'The pgo option is supported only on linux and windows.')

if flavor == 'linux':
if options.enable_pgo_generate or options.enable_pgo_use:
version_checked = (5, 4, 1)
if not gcc_version_ge(version_checked):
version_checked_str = ".".join(map(str, version_checked))
raise Exception(
'The options --enable-pgo-generate and --enable-pgo-use '
f'are supported for gcc and gxx {version_checked_str} or newer only.')
'The pgo option is supported only on linux, macOS, and windows.')

if options.enable_pgo_generate and options.enable_pgo_use:
raise Exception(
Expand All @@ -2024,6 +2016,24 @@ def configure_node(o):
'--enable-pgo-generate first, profile node, and then recompile '
'with --enable-pgo-use')

if flavor in ('linux', 'mac'):
if options.enable_pgo_generate or options.enable_pgo_use:
clang_compilers = [try_check_compiler(compiler, language)[1]
for compiler, language in ((CC, 'c'), (CXX, 'c++'))]
if all(clang_compilers):
profile = os.path.abspath('node.profdata')
if options.enable_pgo_use and not os.path.isfile(profile):
raise Exception(
f'PGO profile not found: {profile}. Run llvm-profdata merge first.')
o['variables']['pgo_profile'] = profile
elif flavor == 'mac' or any(clang_compilers):
raise Exception('PGO requires both CC and CXX to use Clang on macOS '
'or the same compiler family on linux.')
elif not gcc_version_ge((5, 4, 1)):
raise Exception(
'The options --enable-pgo-generate and --enable-pgo-use '
'require gcc and gxx 5.4.1 or newer.')

o['variables']['enable_pgo_generate'] = b(options.enable_pgo_generate)
o['variables']['enable_pgo_use'] = b(options.enable_pgo_use)

Expand Down
66 changes: 50 additions & 16 deletions tools/pgo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,18 @@ The process has three phases:

## Platform Support

| Platform | Supported toolchains | Driver |
| -------- | -------------------- | ------------------------- |
| Windows | Clang-CL | `vcbuild.bat` + `pgo.ps1` |
| Linux | GCC | `configure` + `make` |
| macOS | | — |
| Platform | Supported toolchains | Driver |
| -------- | -------------------- | ------------------------------------- |
| Windows | Clang-CL | `vcbuild.bat` + `pgo.py` |
| Linux | GCC, Clang | `configure` + `make`, `pgo.py` for Clang |
| macOS | Clang | `configure` + `make` + `pgo.py` |

The two supported flows differ in how profile data is collected. Clang writes
one `.profraw` file per process, which must be merged into a single
`.profdata` before the optimize phase. GCC's libgcov instead merges counters
into `.gcda` files next to each object file as each process exits, so there is
no merge step.

Clang on Linux and macOS are not supported yet.

## Quick Start: Windows

From a VS Developer Command Prompt, at the repo root:
Expand All @@ -38,25 +36,25 @@ From a VS Developer Command Prompt, at the repo root:
vcbuild.bat pgo-generate

# Step 2: Run workloads to collect profile data
powershell -ExecutionPolicy Bypass -File .\tools\pgo\pgo.ps1
python tools\pgo\pgo.py

# Step 3: Build the optimized binary
vcbuild.bat pgo-use
```

`pgo.ps1` expects the instrumented binary at `Release\node.exe` (produced by
`pgo.py` expects the instrumented binary at `Release\node.exe` (produced by
step 1) and writes `node.profdata` to the repo root (consumed by step 3).

The script is unsigned, so the default execution policy refuses to run it
without `-ExecutionPolicy Bypass`. Use `pwsh` in place of `powershell` on
PowerShell 7.
The script finds `llvm-profdata` in the Visual Studio LLVM toolset, then
`PATH`. Set `LLVM_PROFDATA` to the matching tool when using a different
Clang installation.

```powershell
# Optionally set a longer training duration (default: 15s per script)
powershell -ExecutionPolicy Bypass -File .\tools\pgo\pgo.ps1 -Duration 30
python tools\pgo\pgo.py --duration=30
```

## Quick Start: Linux
## Quick Start: Linux with GCC

```bash
# Step 1: Build the instrumented binary
Expand Down Expand Up @@ -88,6 +86,42 @@ updates from the worker threads and the libuv thread pool race with each
other, and GCC treats the resulting inconsistent profile as an error unless
told to smooth it out.

## Quick Start: Linux and macOS with Clang

From the repo root:

```bash
# Step 1: Build the instrumented binary
./configure --ninja --enable-pgo-generate
make

# Step 2: Run workloads to collect profile data
python3 tools/pgo/pgo.py

# Step 3: Build the optimized binary
./configure --ninja --enable-pgo-use
make
```

`pgo.py` expects the instrumented binary at `out/Release/node` (produced by
step 1) and writes `node.profdata` to the repo root (consumed by step 3).

The script finds `llvm-profdata` through `xcrun` on macOS and `PATH` on Linux.
Set `LLVM_PROFDATA` to the matching tool when using a different Clang
installation.

```bash
# Optionally set a longer training duration (default: 15s per script)
python3 tools/pgo/pgo.py --duration=30
```

## Clang Profile Collection

On all platforms, `pgo.py` collects raw profiles in a fresh directory and
replaces `node.profdata` after a successful merge. It removes raw profiles
after success and preserves them if training or merging fails. Training
failures stop the script so the workloads can be fixed before trying again.

## Training Scripts

All scripts use only Node.js built-in modules (no npm dependencies).
Expand All @@ -110,7 +144,7 @@ Each script is run as a separate process via `fork()`.
### Running the Orchestrator Directly

The orchestrator can also be invoked directly (e.g. for testing individual
workloads). When used with `pgo.ps1`, this is handled automatically.
workloads). When used with `pgo.py`, this is handled automatically.

```bash
# Run all scripts
Expand All @@ -131,7 +165,7 @@ automatically from the `--duration` flag (in seconds).

```
tools/pgo/
├── pgo.ps1 # Windows training driver (collect + merge)
├── pgo.py # Clang training driver (collect + merge)
├── pgo-run-all.js # Training orchestrator
├── pgo-http-server.js # HTTP server + client workload
├── pgo-json.js # JSON parse/stringify workload
Expand Down
Loading
Loading