Skip to content

Return the exact floor from 3D and 4D Hypot for 32-bit and narrower integers [patch] - #538

Open
matt-edmondson wants to merge 1 commit into
mainfrom
fix/537-narrow-integer-hypot
Open

matt-edmondson wants to merge 1 commit into
mainfrom
fix/537-narrow-integer-hypot

Conversation

@matt-edmondson

Copy link
Copy Markdown
Contributor

Fixes #537

What was wrong

For byte, sbyte, short, ushort, int and uint storage, the 3- and 4-component StorageMath.Hypot overloads nested two double.Hypot calls and then truncated the result with CreateChecked. Each call rounds, so an exact length could come back one ulp low (58.99999999999999), and truncation then made it one too low on a perfect square:

  • Position3D<int>.FromMeter(6, 41, 42).Length() returned 58, not 59
  • new Acceleration4D<int> { X = 3, Y = 3, Z = 3, W = 3 }.Length() returned 5, not 6

This was a regression from the #327 overflow fix.

The fix

A new IsNarrowInteger<T> covers the 32-bit-and-narrower integer primitives (plus nint/nuint on a 32-bit process). HypotOfNarrowInteger widens the components to Int128, which holds the sum of four squares of any 32-bit value (at most 2^66), and takes the exact floor with the existing FloorSqrtOfWideInteger. This is the same approach the wide integers already use since #520. The 2-, 3- and 4-component overloads all use it, so the generated Length, Magnitude, Distance and DistanceTo methods are exact for these types. A length the type can't hold still throws OverflowException, for example Hypot(int.MinValue, 0, 0).

float, double and Half still take the double.Hypot route. The issue notes that double has the same one-ulp error on (6, 41, 42). That isn't changed here, because a correctly rounded floating point result is a separate decision. The wrap-around in generated Distance() (QuantitiesGenerator.cs:1935) is also left for its own issue.

Tests

  • NarrowIntegerHypotenuseIsTheExactFloor: the perfect-square cases from the issue for int, short, uint and byte, plus the extremes and the overflows.
  • NarrowIntegerHypotenuseMatchesTheIntegerRootEverywhere: the issue's brute force, run against an integer square root over every 3D vector with components 0..60 and every 4D vector with components 0..20.
  • IntegerVectorLengthsOfPerfectSquaresAreExact: Length, Magnitude and DistanceTo checked through the generated vector types.
  • TheRootsArePublicAndTheirWorkingsAreNot now also checks that the two new helpers stay private.

With the fix reverted, all three new tests fail. With the fix, the full dotnet test run on net10.0 passes: 2043 passed, 0 failed, 14 skipped as Windows-only. The container has no older .NET runtime, so one test host could not launch.


Generated by Claude Code

…ntegers [patch]

The byte, sbyte, short, ushort, int and uint storage types nested two
double.Hypot calls and truncated the result. Each call rounds, so an exact
length could come back one ulp low (58.99999999999999), and truncation then
answered one too low on a perfect square: Position3D<int>(6, 41, 42).Length()
returned 58 instead of 59, and (3, 3, 3, 3) gave 5 instead of 6.

Those types now sum their squares in Int128, which holds four squares of any
32-bit component, and take the exact integer floor, as the wide integers
already do. Every generated Length, Magnitude, Distance and DistanceTo on
2D, 3D and 4D vector forms goes through this path.

Fixes #537

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017vfJW4YCCXpkCDCmjUcRT6
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3D/4D vector Length() with int/short/byte storage is one too low on perfect squares: Position3D<int>.FromMeter(6, 41, 42).Length() returns 58, not 59

2 participants