Skip to content

Resume in-flight rebalancer runs before sizing a fresh one - #1382

Merged
ebma merged 6 commits into
stagingfrom
fix/rebalancer-resume-in-flight
Sep 19, 2026
Merged

ebma merged 6 commits into
stagingfrom
fix/rebalancer-resume-in-flight

Conversation

@ebma

@ebma ebma commented Sep 19, 2026 •

Copy link
Copy Markdown
Member

Why

The automatic-rebalancer cron has failed on every run since 2026-09-17 with Insufficient USDC on Base. Have: 1032.95, need: 2000.

Two defects stacked:

  1. A run from 2026-09-07 is still persisted as in flight. It reached squidRouterApproveAndSwap, the Polygon swap could not be sent (no POL for gas), and the BRLA was swapped back manually the same day. The state file in Supabase Storage was never updated, so it still expects ~5184 BRLA on Polygon.
  2. The opportunistic in-range path could never resume it. It quoted, sized, and balance-checked a fresh USDC→BRLA→USDC run before the state machine looked at the stuck one. With the in-flight USDC gone from the wallet, that check failed every run. The out-of-range path had its own resume check; the opportunistic path had none.

What

  • checkForRebalancing now resumes a non-idle state file for either Base flow before any quote, sizing, daily-limit, or balance check. The duplicated isResuming branches in runUsdcToBrla / runBrlaToUsdc are removed. --restart semantics are unchanged.
  • The profitable USDC→BRLA amount is only selected when the Base USDC balance can fund it; otherwise the standard amount is used instead of crashing after two quote round-trips.
  • bun run reset:usdc-base-state (dry-run by default, --confirm to reset to idle keeping history) for runs whose funds were reconciled manually. README and security-spec invariants updated.

Ops step after merge

The code fix will resume the stale state and fail loudly on the missing Polygon BRLA until that state is reset once (bun run reset:usdc-base-state --confirm with the cron's Supabase env, or set state.currentPhase to idle in the Supabase UI).

Tests

apps/rebalancer/src/rebalanceCycle.test.ts covers the resume-before-fresh-evaluation ordering (coverage read first, nothing fresh after a resume), the dry-run pause, the --restart bypass, both Base flows' non-idle states, and the profitable-amount gate (off mode and equal amounts never read the balance, unaffordable amounts fall back to the standard size). The orchestration was moved out of index.ts into rebalanceCycle.ts behind injected dependencies to make that possible. Rebalancer suite (78 tests), typecheck, lint, and the coverage ratchet pass.

The opportunistic in-range path quoted, sized, and balance-checked a fresh
USDC->BRLA->USDC run before the state machine got a chance to resume a run
that died mid-flow. With the in-flight USDC no longer in the wallet, that
balance check failed on every cron run and the stuck state was never
resumed. The out-of-range path had its own resume check; hoist a single
one to the top of checkForRebalancing for both Base flows.

Also stop selecting the profitable USDC->BRLA amount when the Base USDC
balance cannot fund it: fall back to the standard amount instead of
crashing after two quote round-trips.
Dry-run by default: prints the persisted USDC->BRLA->USDC state from
Supabase Storage. With --confirm it resets the phase to idle while keeping
history, for runs whose funds were reconciled manually.
@netlify

netlify Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for vortexfi canceled.

Name Link
🔨 Latest commit c290abd
🔍 Latest deploy log https://app.netlify.com/projects/vortexfi/deploys/6aaec0d00a788b0008a9ae92

@netlify

netlify Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for vortex-sandbox ready!

Name Link
🔨 Latest commit c290abd
🔍 Latest deploy log https://app.netlify.com/projects/vortex-sandbox/deploys/6aaec0d020a531000871cc23
😎 Deploy Preview https://deploy-preview-1382--vortex-sandbox.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@netlify

netlify Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for vrtx-dashboard canceled.

Name Link
🔨 Latest commit c290abd
🔍 Latest deploy log https://app.netlify.com/projects/vrtx-dashboard/deploys/6aaec0d0782a4b00086ea256

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Resumption bypasses dry-run and coverage safeguards, while the reset procedure has unsafe operational sequencing.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity · 2 Low severity

Open (4)
What changed in this PR

Updates rebalancer recovery so persisted Base runs resume before fresh-run evaluation and adds an operator reset command.

Changes:

  • Resumes either in-flight Base flow before fresh sizing and checks.
  • Gates profitable sizing by available Base USDC.
  • Adds reset tooling and operational documentation.
File Description
apps/​rebalancer/​src/​index.ts Reorders recovery and adds balance-aware sizing.
apps/​rebalancer/​src/​scripts/​resetUsdcBaseState.ts Adds the reset utility.
apps/​rebalancer/​package.json Registers the reset command.
apps/​rebalancer/​README.md Documents reset operations.
docs/​security-spec/​07-operations/​rebalancer.md Updates rebalancer invariants.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/rebalancer/src/index.ts Outdated
Comment thread apps/rebalancer/src/index.ts Outdated
Comment thread apps/rebalancer/README.md Outdated
Comment thread apps/rebalancer/src/scripts/resetUsdcBaseState.ts Outdated
…ming

Resuming a persisted run in dry-run mode would write state and move funds
during an invocation the spec defines as read-only, and resuming before the
coverage read let funds move when the indexer read would have failed.
Both were latent on the out-of-range paths before the resume was hoisted.
The reset script and a live run write the same Supabase object without
locking, and the script's closing message contradicted the README's
reconcile-first order.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Dry-run detection returns the wrong status, allowing fresh quote evaluation despite an in-flight run.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (4)

Comment thread apps/rebalancer/src/index.ts Outdated
Continuing into the fresh evaluation quoted and sized a run that assumes
the in-flight USDC is still in the wallet, and logged "no rebalancing
needed" while a run was paused.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Off mode now performs an unintended balance RPC, and the core resume regression lacks automated coverage.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
Resolved since last review (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Avoid wallet-balance RPC when rebalancing policy is off

apps/​rebalancer/​src/​index.ts:275

When REBALANCING_POLICY_MODE=off and the profitable amount differs from the standard amount, the standard policy decision is non-executing but control still reaches this new wallet-balance RPC. That violates the documented off-mode invariant (docs/security-spec/07-operations/rebalancer.md:255) and can make a disabled cron fail on an unnecessary balance read. Return the standard skip decision before reading the balance in off mode, and cover this configuration with a regression test.

Comment thread apps/rebalancer/src/index.ts Outdated
The affordability gate read the wallet balance even when the policy mode
is off, which the spec defines as returning before any balance read.

The resume-before-fresh-evaluation ordering, the dry-run pause, the
--restart bypass and the profitable-amount gate had no automated
coverage because index.ts runs the cycle at import. Move that
orchestration into rebalanceCycle.ts behind injected dependencies so a
later refactor cannot silently reorder it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It changes automated fund-movement recovery and destructive operational reset behavior, warranting final human validation.

Review effort: Balanced
Findings: None

Resolved since last review (1)

@ebma
ebma merged commit fe8d8b3 into staging Sep 19, 2026
7 checks passed
@ebma
ebma deleted the fix/rebalancer-resume-in-flight branch September 19, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants