Skip to content

fix(intel): resume the gap scan past nop padding after a failed candidate - #381

Merged
danielplohmann merged 4 commits into
danielplohmann:masterfrom
r0ny123:fix/gap-scan-resume-without-int3
Oct 14, 2026
Merged

danielplohmann merged 4 commits into
danielplohmann:masterfrom
r0ny123:fix/gap-scan-resume-without-int3

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

The failed-candidate resume from #338 only looks for int3 runs. GCC and Clang pad ELF functions with nops, so on those binaries a gap candidate that fails still drops the rest of its gap. On an Akira ELF from the Malpedia corpus, a 7-byte tail fragment at the start of a 3.9 KB gap fails, and the 17 functions behind it are never offered.

Change: when the code a failed candidate decoded from its start ends in ret/jmp and only nops follow up to a 16-byte boundary, the scan resumes at that boundary. A run that ends anywhere else, or reaches an aligned address with no padding, keeps the old behaviour. Requiring both is what keeps misdecoded candidates on Delphi code (4-byte aligned, padded with lea eax, [eax]) from producing resume points inside functions; without it the Malpedia corpus picked up ~400 such hits.

Measured:

corpus result
18 static x64 ELF builds (lz4, zstd, brotli × gcc/clang × O0/O2/Os), scored against symbol tables +2 TP, +1 FP, 0 lost
Malpedia CI corpus (155 files) 152 unchanged; Akira ELF +109 functions (95 16-byte aligned, mostly opening on a prologue), two Konni samples +1 each; 0 lost

make test-all passes and the bundled fixtures do not move. New tests in tests/testFailedGapResume.py cover the resume target and an end-to-end image where an unreferenced function sits behind a failing fragment and nop padding.

…date

The resume added in danielplohmann#338 only looks for int3 runs. GCC and Clang pad ELF
functions with nops, so on those binaries a failed gap candidate still
dropped the rest of its gap.

If the code a failed candidate decoded from its start ends in ret or jmp
and is followed only by nops up to a 16-byte boundary, resume there.

18 static x64 ELF builds (lz4, zstd, brotli; gcc/clang; O0/O2/Os), scored
against their symbol tables: +2 true positives, +1 false positive.
Malpedia CI corpus: 152/155 files unchanged, one Akira ELF +109
functions, two Konni samples +1 each, nothing lost.
ty cannot see the None check in _failedGapResumeTarget from inside the
helpers it calls, so self.gap_pointer + 1 failed type checking there.

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Since "ret/jmp + nops up to 16 bytes" also marks aligned loop heads inside GCC functions, I wanted numbers on a ground truth before saying anything, so I ran master and this branch over our multiplatform ground truth: 866 unstripped x86/x64 ELFs (busybox, dropbear, inetutils, lighttpd; GCC 4.9.2 / 8.1 and clang, -O2/-O3, none of them CET), each stripped of its symbols, and scored function starts against the original symbol table.

true starts false starts
master 123,022 59,558
this PR 123,484 59,554
difference +464 / -2 +2 / -6

With .eh_frame stripped as well: +454 / -2 true, +3 / -6 false. 133 of 866 files change, all of them GCC builds (dropbear, inetutils, lighttpd; i686 more than x86_64). No clang build changes. So the loop-head concern does not show up in practice: the resume only happens after a failed candidate, and on this data those failures sit between functions far more often than inside them. Happy to merge this.

The first instructions of the 464 new true starts are push (298), sub (103), mov (43) and repz ret (20), so a prologue check at the resume target would throw away over a third of the gain on pre-CET code. Please don't add one.

Small things, none blocking:

  • _straightLineEnd and failed_gap_extent_end live in the shared RecursiveDisassembler / FunctionCandidateManager, so they run after every failed candidate on every backend, sorting the state's instructions each time, while only the intel manager reads the result. Could the computation move into the intel path, or be skipped when the manager doesn't use it?
  • _failedGapResumeTarget can return early (no gap pointer, no binary info, negative offset) before failed_gap_extent_end is cleared, so an old value can stay around. The end <= gap_pointer check keeps this harmless today, but clearing it at the top of the function would be cleaner.
  • Only capstone's nop counts as padding. Old i386 binutils fillers (lea esi,[esi+0], mov esi,esi) aren't covered. I understand why (Delphi lea eax,[eax]), but a sentence in the CHANGELOG would help.
  • There will be a CHANGELOG conflict with #380 once that one lands.

…ger only

The extent was computed in the shared gap loop after every failed
candidate on every backend, sorting the state's instructions each time,
while only the x86 manager reads it. Hand the state to a manager hook
that is a no-op by default, walk the decoded run through an address map
instead of a sort, and drop the stored extent at the top of the resume
so an early return cannot leave it behind.

The CHANGELOG entry now notes that old i386 fillers are not read as
padding and carries the reviewer's ground-truth numbers.
@r0ny123

r0ny123 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for running it over the ground truth. I'll leave the resume target without a prologue check. All four small points are addressed in deb7555:

  • Shared cost. The shared gap loop now just passes the failed candidate's state to noteFailedGapCandidate, which does nothing by default. Only the x86 manager overrides it. It also walks the decoded run through an address map instead of sorting the state's instructions.
  • Stale extent. _failedGapResumeTarget clears the stored extent at its very top, so none of the early returns can leave it behind. There's a test for the no-gap-pointer case.
  • i386 fillers. The CHANGELOG now says only capstone's nop counts as padding, and that lea esi, [esi] / mov esi, esi are deliberately excluded because Delphi emits lea eax, [eax] inside code. I also put your 866-file numbers in the entry as reviewer-reproduced.
  • CHANGELOG conflict with fix(intel): bound jump tables by and masks in Go type switches #380. I'll rebase whichever of the two lands second onto master and resolve it then.

Function sets are unchanged against f5290cb on 18 static x64 ELF builds and the Go reproducer from #363.

@danielplohmann danielplohmann added this to the v4.10.0 milestone Oct 14, 2026
@danielplohmann
danielplohmann merged commit a867148 into danielplohmann:master Oct 14, 2026
25 checks passed
@r0ny123
r0ny123 deleted the fix/gap-scan-resume-without-int3 branch October 15, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants