Octree broadphase: exact octant classification, depth-blind retrieve()

packages/melonjs/CHANGELOG.md

@@ -65,6 +65,8 @@ The old path scales linearly with vertex count; the new one is flat, because no
 - **the cost of `antiAlias: true` under post effects, quantified** ([#1556](https://github.com/melonjs/melonJS/issues/1556)) — MSAA composing through effect chains (see *Added*) is paid for in memory and bandwidth, and the price is worth knowing. Arithmetic, not measurement: a 4× capture target keeps 4 color + 4 depth-stencil samples per pixel next to its 1× resolve texture — roughly **28 extra bytes per pixel on WebGL (~55 MB of GPU memory at 1080p)** and **~16 bytes per pixel on WebGPU (~32 MB at 1080p)**, where the multisampled depth attachment is shared with the canvas rather than per-target; both scale linearly with resolution. Per frame it adds one resolve blit per effect bracket, and draws inside the bracket write up to 4 samples per covered pixel — bandwidth, not shading cost, since fragment shaders still run once per pixel under MSAA. Only scene **capture** targets pay any of this (ping-pong intermediates stay 1×), and with `antiAlias: false` — the default — no multisampled storage exists at all, so nothing changes
 
 ### Fixed
+- **the 3D broadphase silently dropped collisions between bodies at different depths** — under `Camera3d` the world's broadphase is an `Octree`, and `retrieve()` — the candidate feed for SAT collision, pointer picking, the 2D raycast and `adapter.queryAABB` — descended only into the octant the query item itself classified into. But every one of those consumers decides overlap in the **XY plane**: two bodies at different z that overlap in XY genuinely collide, and were never offered to each other as candidates. Whether a given pair was tested came down to which side of an octant boundary each happened to fall on. Measured on a randomized 300-body scene, **12 of 20 genuinely overlapping pairs were never surfaced**. `retrieve()` is now depth-blind: it classifies on x/y only and walks both depth halves of that quadrant, so x/y pruning still applies at every level and in both halves (an item lying wholly inside a different x/y quadrant cannot overlap, and midpoint-straddling items already live at the parent level). On a 600-body scene this costs nothing at all when the bodies share a gameplay plane — the candidate count is unchanged — and on a depth-spread scene it settles at the same candidate count as the flat one, which is the point: with depth no longer part of the decision, the candidate set depends only on the x/y distribution. The genuinely 3D queries are unaffected and still prune on depth: `queryAABB`, `querySphere`, `queryRay` and `queryFrustum` have their own entry points, and each is now pinned by a differential test against a brute-force scan. Note this removes the incidental "parallax at a distant z drops out of collision for free" behaviour that the 2.5D documentation described as best-effort — it was this defect seen from its good side. Exclude parallax deliberately instead, with `isKinematic = true` or `collisionType` / `collisionMask`, which is what the 2D path has always done
+- **an entire 2.5D gameplay plane sat unpartitioned at the root of the octree** — `getIndex` returned −1 (meaning "straddles a midpoint, keep at this level") for an item sitting *exactly* on one. On x and y that is at least defensible, since an item there may genuinely span the boundary; on z it never is, because items are point-z in the broadphase and a point cannot straddle anything. It mattered because the root box is origin-centred, so its midpoints are `(0, 0, 0)` — the default `pos` of every renderable, and the shared gameplay z that the 2.5D recipe prescribes. Measured: 200 bodies on a `z = 0` plane all stayed at the root and `retrieve()` returned **200 of 200**, degrading the broadphase to a linear scan for exactly the layer holding the most bodies; the same 200 spread across z left only 10 at the root. Classification is now exact on all three axes — a midpoint belongs to the far/right/bottom child, and an item whose far edge merely touches one still counts as wholly inside the near side. Genuine straddlers and out-of-bounds items still stay at the parent, both under regression test. This is invisible to 2D games, which use a `QuadTree` and never construct an `Octree`
 - **a mesh marked `lit` with no usable normals rendered solid black** — normalizing a zero-length normal yields NaN, which the shader turned into black fragments rather than something recognisable. That happens whenever `lit: true` meets geometry with no normals, and on the 2D-camera path generally, where world normals are never written. Such a mesh now degrades to **unlit** on both GPU backends: wrong, but recognisably the model instead of a hole in the scene. Note this makes the failure legible, it does not make a `Camera2d` mesh light — populating world normals on that path is tracked separately as [#1576](https://github.com/melonjs/melonJS/issues/1576) and remains open
 - **`DropShadowEffect` rendered its shadow vertically mirrored (up instead of down) when chained with other effects on WebGL** — the pooled multi-effect path composites through capture FBOs, which are bottom-up under GL, so the y component of any directional UV arithmetic inside an effect body ran inverted relative to the single-effect fast path (and to the WebGPU backend, whose captures are top-down on both paths). Found by cross-backend comparison — earlier pixel-count probes were direction-blind. Effect bodies can now declare a `uUVYDir` uniform that the renderer feeds per draw path (+1 where `uv.y` grows downward, −1 on the GL pooled path); DropShadow uses it, so a positive `offsetY` means *down* on every path of both backends; `ShineEffect` adopts it too, so an angled sweep travels the documented direction (π/2 = top→bottom) on the pooled path as well
 - **a scene containing only meshes stopped clearing its depth buffer after the first frame, and its geometry disappeared** — a regression from the 19.7 mesh state-ownership work ([#1468](https://github.com/melonjs/melonJS/issues/1468)), found while working on [#1552](https://github.com/melonjs/melonJS/issues/1552). The depth clear and the lit-mesh light upload both ran from `MeshBatcher.bind()`, which is a per-*transition* hook, not a per-frame one: `setBatcher` returns early when the requested batcher is already current. A scene with nothing else to draw — no sprites, no UI, no unlit mesh beside a lit one — therefore bound once and never again, leaving the depth attachment on the first frame's values, so anything receding from the camera failed the depth test and was not drawn at all. The same silence froze `Light3d` lighting at its first-frame values on such a scene. Both now refresh on the draw path, at no measurable cost (one boolean test per draw for the depth clear; the light upload is skipped outright when the lights have not changed)

packages/melonjs/src/physics/broadphase/octree.ts

@@ -241,6 +241,70 @@ export default class Octree implements Broadphase<OctreeItem> {
 	 * @returns octant index (0-7) or -1
 	 */
 	getIndex(item: OctreeItem): number {
+		const quadrant = this._quadrantXY(item);
+		if (quadrant === -1) {
+			return -1;
+		}
+		// items are point-z in the broadphase — bounds is 2D, so the
+		// item occupies a single z plane in 3D space. This matches
+		// how the renderer sorts (depth comes from pos.z, not from a
+		// 3D AABB on the renderable). A point cannot STRADDLE the depth
+		// midpoint the way an extended item straddles a vertical one, so
+		// the tie goes to the far child rather than to this level.
+		const rz = itemZ(item);
+		const nodeFront = this.bounds.front;
+		const nodeBack = nodeFront + this.bounds.depth;
+		if (rz < nodeFront || rz > nodeBack) {
+			return -1;
+		}
+		const depthMidpoint = nodeFront + this.bounds.depth / 2;
+		if (rz < depthMidpoint) {
+			return quadrant;
+		}
+		if (rz >= depthMidpoint) {
+			return quadrant + 4;
+		}
+		// unreachable for any real `rz` — catches NaN, which stays at
+		// this level rather than vanishing into an octant no query
+		// will look in.
+		return -1;
+	}
+
+	/**
+	 * Classify an item into one of this node's four x/y quadrants,
+	 * **ignoring z entirely**. Returns the near-half octant index
+	 * (0-3, so `+ 4` gives the far-half sibling), or -1 when the item
+	 * straddles a midpoint or lies outside this node in x/y.
+	 *
+	 * Split out of {@link Octree.getIndex} for {@link Octree.retrieve},
+	 * whose consumers all decide overlap in the XY plane and therefore
+	 * need x/y pruning WITHOUT a depth opinion. Routing that through
+	 * `getIndex` instead would reject the item from its own z-sibling
+	 * on the depth out-of-bounds guard, collapsing that entire subtree
+	 * into an unpruned walk — measured at ~10× the nodes visited on a
+	 * depth-spread scene.
+	 *
+	 * Boundary convention: a midpoint belongs to the RIGHT / BOTTOM
+	 * child, and an item whose far edge merely touches a midpoint still
+	 * counts as wholly inside the near side — `<=` / `>=` rather than
+	 * strict, because an item sitting exactly ON a midpoint lies wholly
+	 * within one child and must descend. Only a genuine STRADDLE
+	 * belongs at this level.
+	 *
+	 * This matters more than it looks: the root box is origin-centred,
+	 * so its midpoints are (0, 0, 0) — the default `pos` of every
+	 * renderable. Under strict comparisons an entire gameplay plane
+	 * classified as -1 and sat unpartitioned at the root.
+	 *
+	 * Safety: two items separated by a midpoint cannot overlap. If A
+	 * lies left (`A.right <= mid`) and B lies right (`B.left >= mid`)
+	 * then `A.right <= B.left`, and touching edges are not an overlap
+	 * under the engine's strict bounds test.
+	 * @param item - the object to classify
+	 * @returns near-half octant index (0-3) or -1
+	 * @ignore
+	 */
+	_quadrantXY(item: OctreeItem): number {
 		const bounds = item.getBounds();
 		let rx: number;
 		let ry: number;
@@ -264,17 +328,10 @@ export default class Octree implements Broadphase<OctreeItem> {
 		}
 		const rw = bounds.width;
 		const rh = bounds.height;
-		const rz = itemZ(item);
-		// items are point-z in the broadphase — bounds is 2D, so the
-		// item occupies a single z plane in 3D space. This matches
-		// how the renderer sorts (depth comes from pos.z, not from a
-		// 3D AABB on the renderable).
 		const nodeLeft = this.bounds.left;
 		const nodeTop = this.bounds.top;
-		const nodeFront = this.bounds.front;
 		const nodeRight = nodeLeft + this.bounds.width;
 		const nodeBottom = nodeTop + this.bounds.height;
-		const nodeBack = nodeFront + this.bounds.depth;
 
 		// Out-of-bounds guard — keep at parent's `objects` so the
 		// spatial pruning in `querySphere` / `queryAABB` stays
@@ -283,34 +340,26 @@ export default class Octree implements Broadphase<OctreeItem> {
 			rx < nodeLeft ||
 			rx + rw > nodeRight ||
 			ry < nodeTop ||
-			ry + rh > nodeBottom ||
-			rz < nodeFront ||
-			rz > nodeBack
+			ry + rh > nodeBottom
 		) {
 			return -1;
 		}
 
 		const verticalMidpoint = nodeLeft + this.bounds.width / 2;
 		const horizontalMidpoint = nodeTop + this.bounds.height / 2;
-		const depthMidpoint = nodeFront + this.bounds.depth / 2;
 
-		const nearOctant = rz < depthMidpoint;
-		const farOctant = rz > depthMidpoint;
-		const topQuadrant = ry < horizontalMidpoint && ry + rh < horizontalMidpoint;
-		const bottomQuadrant = ry > horizontalMidpoint;
-		const leftQuadrant = rx < verticalMidpoint && rx + rw < verticalMidpoint;
-		const rightQuadrant = rx > verticalMidpoint;
+		const topQuadrant =
+			ry < horizontalMidpoint && ry + rh <= horizontalMidpoint;
+		const bottomQuadrant = ry >= horizontalMidpoint;
+		const leftQuadrant = rx < verticalMidpoint && rx + rw <= verticalMidpoint;
+		const rightQuadrant = rx >= verticalMidpoint;
 
-		if (!(nearOctant || farOctant)) {
-			return -1;
-		}
-		const farOffset = farOctant ? 4 : 0;
 		if (leftQuadrant) {
-			if (topQuadrant) return 1 + farOffset;
-			if (bottomQuadrant) return 2 + farOffset;
+			if (topQuadrant) return 1;
+			if (bottomQuadrant) return 2;
 		} else if (rightQuadrant) {
-			if (topQuadrant) return 0 + farOffset;
-			if (bottomQuadrant) return 3 + farOffset;
+			if (topQuadrant) return 0;
+			if (bottomQuadrant) return 3;
 		}
 		return -1;
 	}
@@ -442,9 +491,32 @@ export default class Octree implements Broadphase<OctreeItem> {
 		}
 
 		if (this.nodes.length > 0) {
-			const index = this.getIndex(item);
-			if (index !== -1) {
-				this.nodes[index].retrieve(item, undefined, out);
+			// `retrieve` feeds 2D consumers ONLY — the SAT detector
+			// (`Detector.collisions`), pointer picking, the 2D raycast and
+			// `adapter.queryAABB`. Every one of them decides overlap in the
+			// XY plane, so pruning on z here silently drops real collisions:
+			// two bodies at different depths that overlap in XY do collide
+			// under 2D SAT, but would never be offered to each other as
+			// candidates.
+			//
+			// So classify on x/y ONLY and walk both depth halves of that
+			// quadrant. Going through `getIndex` and walking `index ^ 4`
+			// instead would work, but the sibling would then reject the item
+			// on its depth out-of-bounds guard and fall back to an unpruned
+			// 8-way walk of that whole subtree.
+			//
+			// x/y pruning is preserved at every level and in both halves: an
+			// item lying wholly inside a DIFFERENT x/y quadrant cannot
+			// overlap, and midpoint-straddlers live in `objects` at this
+			// level and were already folded in above.
+			//
+			// Queries that genuinely want depth pruning have their own entry
+			// points (`queryAABB`, `querySphere`, `queryRay`, `queryFrustum`)
+			// and are unaffected.
+			const quadrant = this._quadrantXY(item);
+			if (quadrant !== -1) {
+				this.nodes[quadrant].retrieve(item, undefined, out);
+				this.nodes[quadrant + 4].retrieve(item, undefined, out);
 			} else {
 				for (let i = 0; i < this.nodes.length; i++) {
 					this.nodes[i].retrieve(item, undefined, out);

packages/melonjs/tests/octree.spec.js

@@ -244,10 +244,21 @@ describe("Octree", () => {
 			const item = makeItem({ x: 100, y: 295, z: 0, w: 10, h: 20 });
 			expect(octree.getIndex(item)).toBe(-1);
 		});
-		it("returns -1 for items sitting on the depth midpoint (z=0)", () => {
+		it("classifies items sitting exactly ON the depth midpoint (z=0)", () => {
+			// BEHAVIOUR CHANGE: this used to assert -1. Items are point-z in
+			// the broadphase, so an item can never STRADDLE the depth
+			// midpoint the way it straddles a vertical/horizontal one — it
+			// lies wholly within one half, and must descend. The midpoint
+			// belongs to the far child by convention.
+			//
+			// The old behaviour was costly rather than merely academic: the
+			// real root box is origin-centred, so its depth midpoint is 0 —
+			// the default `pos.z` of every renderable, and the shared
+			// gameplay z the 2.5D recipe prescribes. An entire gameplay
+			// plane therefore stayed at the root, unpartitioned, and every
+			// query degraded to a linear scan over it.
 			const item = makeItem({ x: 100, y: 100, z: 0 });
-			// not strictly < or > midpoint
-			expect(octree.getIndex(item)).toBe(-1);
+			expect(octree.getIndex(item)).toBe(5); // top-left-far
 		});
 		it("classifies top-left-near as octant 1", () => {
 			const item = makeItem({ x: 50, y: 50, z: -50 });