Fail over on malformed 200 responses from Hive RPC nodes

[feruzm]

A Hive RPC node can return 200 with valid JSON that carries neither an error field nor a usable result. HiveRpcClient treated any 200-JSON without an error field as a success, so the health tracker kept such a node ranked first — a poisoned node stayed the preferred node for as long as it misbehaved. Observed twice in production as multi-hour bursts of getAccounts result is not iterable (~1.8k occurrences each), during which token validation failed and authenticated endpoints returned 401 to normal user traffic.

This is the same malformed-200 class #56 fixed on the Hive-Engine side, now applied to the Hive JSON-RPC client.

Changes

• Call() accepts an optional result-shape validator. A failed check throws the internal node-unavailable path: the node is marked failed, the pool advances immediately (no same-node retry — a node serving malformed 200s keeps serving them), and the next node is tried.
• The typed helpers validate their known shapes: get_accounts must yield an array, get_dynamic_global_properties an object. All production call sites go through these two helpers.
• When every node fails validation, the surfaced error names the failing node and method — the previous downstream result is not iterable message could not say which node misbehaved.
• RPC-level errors (error field present) still surface immediately without failover, unchanged.

Testing

• 3 new failover tests (58 total, all passing): malformed node fails over without a same-node retry, is demoted on subsequent calls, and an all-malformed pool throws an error naming the node.
• Local run: portfolio-v2 returns full hive + engine layers across repeated calls (exercises both validated helpers through HiveExplorer).

[coderabbitai]

Warning

Review limit reached

@feruzm, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 56 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 12013d31-12e7-4e5f-9d33-ce0462296d98

📥 Commits

Reviewing files that changed from the base of the PR and between f34ef6a and 9549ff2.

📒 Files selected for processing (3)

• CLAUDE.md
• dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs
• dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/hive-rpc-result-validation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR adds failover for malformed successful responses from Hive RPC nodes. The main changes are:

• Adds optional result-shape validation to HiveRpcClient.Call.
• Validates account results as arrays and global properties as objects.
• Marks malformed responders as failed and immediately advances to another node.
• Adds tests for failover, node demotion, and pool-exhaustion diagnostics.
• Documents the new Hive RPC health behavior.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.
• Legitimate empty account arrays pass validation.
• Validation failures record node failure and skip same-node retries.
• Current production callers use the validated typed helpers.

Important Files Changed

Filename	Overview
dotnet/EcencyApi/Infrastructure/HiveRpcClient.cs	Adds optional result validation and applies array and object checks to the two production helpers.
dotnet/EcencyApi.Tests/HiveRpcFailoverTests.cs	Adds malformed-response coverage for immediate failover, later node ordering, and pool exhaustion.
CLAUDE.md	Documents result-shape validation as part of Hive RPC node health handling.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Call typed Hive helper] --> B[Request preferred node]
    B --> C{RPC error present?}
    C -- Yes --> D[Surface RPC error]
    C -- No --> E{Expected result shape?}
    E -- Yes --> F[Record success and return result]
    E -- No --> G[Record node failure]
    G --> H{Another node available?}
    H -- Yes --> B
    H -- No --> I[Throw node and method error]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Call typed Hive helper] --> B[Request preferred node]
    B --> C{RPC error present?}
    C -- Yes --> D[Surface RPC error]
    C -- No --> E{Expected result shape?}
    E -- Yes --> F[Record success and return result]
    E -- No --> G[Record node failure]
    G --> H{Another node available?}
    H -- Yes --> B
    H -- No --> I[Throw node and method error]

Loading

_{Reviews (1): Last reviewed commit: "fix: fail over on malformed 200 response..." | Re-trigger Greptile}