Skip to content

Widen size checks instead of truncating to size_t - #372

Closed
kev009 wants to merge 9 commits into
InterNetNews:mainfrom
kev009:uintmax
Closed

Widen size checks instead of truncating to size_t#372
kev009 wants to merge 9 commits into
InterNetNews:mainfrom
kev009:uintmax

Conversation

@kev009

@kev009 kev009 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Casting off_t to unsigned int or size_t before comparing bounds drops high bits, so articles and CAF spans larger than the narrow type could pass validation. Compare via uintmax_t so the checks remain value-preserving.

@Julien-Elie

Copy link
Copy Markdown
Contributor

Thanks for these two nits!
Looking at where st_size is used in the source code, shouldn't we also have for instance in storage/tradindexed/tdx-group.c

if ((size_t) st.st_size > sizeof(struct group_header)) {

changed to

if (st.st_size > (off_t) sizeof(struct group_header)) {

We also have functions like index_entry_count which expect a size_t argument, and are given off_t.
And assignments of an off_t to size_t like size = st.st_size in storage/tradindexed/tdx-util.c but I am really unsure these variables have a size exceeding size_t in practice (I guess someone would have already reported an issue otherwise).

@kev009
kev009 force-pushed the uintmax branch 2 times, most recently from 0465b39 to b190c5c Compare August 8, 2026 13:01
kev009 added 9 commits August 8, 2026 07:35
Check wire-format growth before allocation and propagate conversion failures through article readers and callers. Keep the historical tradspool article-size policy while storing representable lengths without narrowing.
Read large descriptors in representable chunks, preserve caller ownership and errno, and reject premature EOF. Keep active-file and buffer sizes tied to the stat used for their allocation.
Validate on-disk CAF bitmap, table, article, and recovery metadata before allocation or seeking. Preserve cleaner recovery for empty and repairable files while rejecting spans outside the backing file.
Route all failures after creating the PID-suffixed CAF path through one cleanup path. This closes the locked descriptor, unlinks the temporary file, and preserves the original errno for callers.
Retry interrupted and partial CAF metadata writes through the standard full-write helper instead of treating them as immediate I/O failures.
Make ovgroupunmap the sole owner of releasing per-group mappings. This removes duplicate munmap paths and lets partially constructed cache lists use the same cleanup path.
Reject unrepresentable or partial group.index files, preserve the current mapping until replacements succeed, propagate lookup failures, and support read-only remapping after concurrent growth.
A full freelist is represented by -1, so using the freelist head as the loop bound skipped hidden-group expiry once all mapped slots had been allocated. Refresh the mapping and scan every slot, relying on the zero count of unused entries to skip them.
Reject index and data sizes that cannot be represented by mapping and entry-count types. Preserve live mappings across remap and expansion failures, and audit the complete on-disk index after concurrent growth.
@kev009

kev009 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Good catch, that comparison was one instance of a broader problem. I audited the related off_t conversions and expanded the PR accordingly.

In particular, the tradindexed paths now validate that file sizes are nonnegative and representable as size_t, that entry counts fit in int, and that header + count * entry_size cannot overflow. Remap and expansion failures also preserve the existing mapping, and audit explicitly refreshes the complete on-disk index after concurrent growth.

I applied equivalent checks to buffindexed, including state-safe read-only remapping and deliberate rejection of partial trailing group.index entries. The PR also now covers the related complete-file read, wire conversion, and CAF metadata arithmetic.

I force-pushed a cleaned-up nine-commit series so the review fixes are folded into their originating changes rather than left as follow-up commits.

Most of these are low-value hardening changes that do not warrant backporting to stable; the possible exception is the buffindexed mapping-state work, which prevents silent group.index corruption and duplicate unmapping on failure, while the rest mainly helps document the file-format invariants and provides regression coverage for them.

@Julien-Elie

Copy link
Copy Markdown
Contributor

Many thanks Kevin for this thorough review of off_t conversions.
The problem may be even broader as there certainly are other unsatisfactory conversions of other types (dev_t, ino_t, uint64_t... even size_t or simple unsigned long... in printf-like functions). It would need a complete check of all conversions but I am unsure it is worth the work unless it uncovers actual bugs. I would not want you to exhaust yourself on that!

@Julien-Elie

Copy link
Copy Markdown
Contributor

Also, many thanks for the addition of new tests! It will help ensuring we do not have regressions between releases.

Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
Check wire-format growth before allocation and propagate conversion
failures through article readers and callers.  Keep the historical
tradspool article-size policy while storing representable lengths
without narrowing.

see #372
Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
Read large descriptors in representable chunks, preserve caller
ownership and errno, and reject premature EOF. Keep active-file and
buffer sizes tied to the stat used for their allocation.

see #372
Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
Validate on-disk CAF bitmap, table, article, and recovery metadata
before allocation or seeking. Preserve cleaner recovery for empty and
repairable files while rejecting spans outside the backing file.

see #372
Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
Route all failures after creating the PID-suffixed CAF path through one
cleanup path. This closes the locked descriptor, unlinks the temporary
file, and preserves the original errno for callers.

see #372
Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
Retry interrupted and partial CAF metadata writes through the standard
full-write helper instead of treating them as immediate I/O failures.

see #372
Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
Make ovgroupunmap the sole owner of releasing per-group mappings. This
removes duplicate munmap paths and lets partially constructed cache
lists use the same cleanup path.

see #372
Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
Reject unrepresentable or partial group.index files, preserve the
current mapping until replacements succeed, propagate lookup failures,
and support read-only remapping after concurrent growth.

see #372
Julien-Elie pushed a commit that referenced this pull request Aug 13, 2026
A full freelist is represented by -1, so using the freelist head as the
loop bound skipped hidden-group expiry once all mapped slots had been
allocated. Refresh the mapping and scan every slot, relying on the zero
count of unused entries to skip them.

see #372
@Julien-Elie Julien-Elie self-assigned this Aug 13, 2026
@Julien-Elie Julien-Elie added bug Something isn't working C: general Related to several components P: low Low priority labels Aug 13, 2026
@Julien-Elie Julien-Elie added this to the 2.8.0 milestone Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working C: general Related to several components P: low Low priority

Development

Successfully merging this pull request may close these issues.

2 participants