[saesch83]

Part of #457. Replaces #464, which I closed; this PR has its changes and the follow-ups below.

Builds on #458. It uses that PR's Months in dashboard charts and reports setting and BudgetPeriod::monthsBetween(). Until #458 is merged, this branch includes its commits. The forecast commits start at feat(forecast): learn from budget months and complete months only.

Why

The live forecast (ForecastService::getLiveForecast(), the Forecast page and the dashboard tile) projects each month as the average of past months. Bills and recurring income only act as a floor under that average. In practice the projected balance was often wrong where it matters most: in the months where something you already know about happens.

• Planned one-off payments were invisible. Examples:

  • an investment you have planned for March, entered as a one-off bill or as a transaction dated in March;
  • a car repair bill due next month;
  • a transfer you have already entered for next week.

  None of these counted, so the balance line ran on as if the money stayed in the account.

• Past one-off payments were spread over every month. When you moved savings into an investment last spring, that amount went into the average and lowered every projected month a little. It never showed as one month that drops.

  • Exclude from forecast already takes such a row out of the history. The future side had no counterpart, so the planned payment was simply missing.

• Irregular income was averaged too. A yearly bonus counted as a twelfth in every month, never in the month it is paid.

• Partial months skewed the average. Bank sync fetches 90 days, so the first month holds only its last days. Five days of September with the salary and almost no spending made the average savings several times too high.

• The start day was ignored. The forecast always used calendar months, so with a start day of 26 a salary paid on the 26th landed in the wrong month compared with the Budget page.

My aim: what you already know about shows in the month it happens, and the average covers only the everyday part that you can't plan.

What changes

1. Complete months only, in budget periods when chosen (feat(forecast): learn from budget months …)
   • With Budget periods chosen in Settings (feat(reports): let dashboard charts and reports count budget months #458), the forecast learns from the complete budget periods before the running one and projects budget periods. Calendar mode keeps calendar months.
   • In both modes a month counts only when every account that held money before its first transaction has history from the month's start.
     • "Held money" means a non-zero opening balance.
     • Up to 3 days without bookings are tolerated, usually a weekend.
     • Accounts opened from nothing, or added after the window, don't limit the history.
   • TransactionMapper::getFirstTransactionDates() reads each account's first date in one grouped query.
2. Planned items in their month (feat(forecast): place bills, recurring income and future rows in their month)
   • Each projected month = the average of the unplanned history plus what is scheduled in that month:
     • bills on the Bills page's own schedule (BillService::occurrencesBetween()), one-off bills included;
     • recurring income on its expected dates;
     • transactions dated after today.
   • The average leaves out bill-linked rows and credits that match a recurring income, so nothing counts twice. A bonus that stopped coming shows as a falling line rather than being propped up by a floor.
   • Items flagged Exclude from forecast stay out, as before.
3. The rest of the running month is its own column (fix(forecast): give the rest of the running month its own column)
   • With a start day, the days left of the running period were folded into the first projected month, which then held a month and a half of bills and income.
   • They are now the first projection, marked partial: true and labelled "{month} (rest)".
   • That column holds the average for the days left plus everything scheduled in them.
4. Recurring income matched word by word (fix(forecast): match a recurring income's pattern word by word)
   • A credit counted as the income's payment only when it held the pattern as one phrase.
   • Banks often write the same words in another order, so the salary stayed in the average and came again on its date: about twice the income.
   • The forecast now needs every word of the pattern, in any order and case. The import matcher (RecurringIncomeService) is unchanged.
5. Income and bills without an account (fix(forecast): count income and bills with no account in the user's forecast)
   • The web UI always passes the visible account ids, and items without an account were then dropped. A yearly bonus entered without an account never appeared.
   • They now count when the forecast holds one of the user's own accounts, and stay out of a forecast made only of accounts shared with the user.

There is a changelog entry under ## [Unreleased] and a separate chore(build) commit with the rebuilt budget-app.js and budget-forecast.js. Strings added: "{month} (rest)" and one hint line on the Forecast page. I left the translation files alone.

Testing

• Unit tests (new): ForecastScheduledTest, with:

  • one-off and monthly bills;
  • recurring income on its dates;
  • future rows;
  • the average without bill-linked and recurring-income rows;
  • start day 27 with the running month's column;
  • pattern words in any order and with a missing word;
  • income without an account in the user's own forecast and in a forecast of shared accounts only.

  Also tests for the history window, complete-month detection and budget-period aggregation in PatternAnalyzer.

• Updated tests: ForecastServiceTest, ForecastHistoryWindowTest, UsersOwnDateTest and the integration tests ForecastHistoryTest and ForecastSpendingFloorTest now expect the running month's column first.

• Full suites:

  • PHPUnit: 7,559 tests, 1 skipped.
  • Integration: 303 tests, 1 skipped (Nextcloud 31, SQLite).
  • vitest: 2,255 tests. Two fail only under full parallel load (dashboardDateRange, importMappingRequiredFields); they time out on dev too and pass when run alone.
  • cs:check, Psalm and npm run lint are clean.

• Manual (Nextcloud 31), on a real data set with start day 27:

  • salary, Christmas bonus and yearly bonus as recurring income, plus a one-off bill;
  • each shows in its own budget period, the running period has its "(rest)" column, and the salary is no longer counted twice.
  • I've been running this on my own instance since 30 October.

🤖 Generated with Claude Code

[otherworld-dev]

I gave this a run on a test instance with a year of history, a few bills and recurring income, in calendar months and with start days 26 and 10. The placement by month works, but a few things need sorting before I can merge it.

• Bills get counted twice. A bill with "Create future transaction" on (the default) has a scheduled row on its next due date, and futureRows() picks that up as well as scheduledTotal() placing the bill. With a 900 rent bill, December came out 900 too high.
• One account can wipe the history. completeHistoryStart() cuts the window for every account to the latest-starting one, so adding a savings account with an opening balance and one transaction on 15 Oct took the forecast from 12 months to 0 - the page says "Not enough data" and the warnings stop. Could it cut per account instead, so only that account's partial months drop out?
• Income booked with Mark Received or auto-create isn't recognised. Those rows have the income's name as vendor and an empty description, so unless the pattern words are in the name isRecurringIncome() misses them and the income counts twice. A 300 pension came out at 600 a month.
• Same for bill payments that aren't linked to the bill - they stay in the average and get placed on their dates as well. The old floor could never double up like that.
• Transfer bills to an account outside the forecast drop out now. Their history rows have a bill_id so they leave the average, and scheduledBills() skips transfers.
• An income with no pattern matches any credit within 20%, and with no account any credit in any account. The import matcher needs a pattern - could this do the same?

Smaller ones: the scheduled totals add up floats where the rest of the app goes through MoneyCalculator, and a bill that's overdue but not paid isn't in any column any more.

The unit tests mock the rows, so none of this shows up there. An integration test with a real bill placeholder and an income marked received would catch the first and third.

[saesch83]

Thanks for the thorough run, all of these were real. Fixed in 4751e82 (dev merged in first, bundle rebuilt in 35435f1):

• Bills counted twice: futureRows() now leaves out rows whose bill_id belongs to a bill that is placed on its dates, and scheduled rows never count as history either. The 900 rent is in December once.
• One account wiping the history: the cut is per account now (completeHistoryStarts() returns a start date per account), so a new savings account only drops its own partial months. The other accounts keep their 12.
• Mark Received / auto-create: a credit counts as the income's payment when the vendor is the income's name (case-insensitive) or the pattern matches, on the income's account and within 20%. The 300 pension stays at 300.
• Unlinked bill payments: the same check for bills, so a payment booked under the bill's name or pattern leaves the average.
• Transfer bills: a transfer to an account outside the forecast is spending, and one from outside into the forecast is income (at the destination account's rate). Transfers between two forecast accounts still drop out, since they don't change the total.
• Income without a pattern: it no longer matches any credit within 20%. Without a pattern it only claims rows booked under its name, which is what Mark Received writes. Requiring a pattern like the import matcher does would bring the doubled pension back for those rows. An income with no account still looks at every account, but only for rows with its name or pattern.
• Floats: the scheduled totals and the row sums go through MoneyCalculator.
• Overdue bills: a bill due today or earlier counts in the first column, unless a non-scheduled row since the due date has already paid it (by bill_id, or by name/pattern as above).

New integration test ForecastScheduledItemsTest with real rows covers the placeholder, an income marked received via createFromIncome, an unlinked payment, an income without a pattern, a new account, a transfer bill out of the forecast, and an overdue bill paid and unpaid. Six of its eight tests fail against the previous version of the branch.

Unit, integration, Psalm and CS are green.

[saesch83]

One more fix from running it on my own data (98b009c): the column for the rest of the running month took a share of the average by days, but spending isn't spread evenly. Most of it leaves in the days after the salary comes in. With start day 27, on 7 November the forecast put 61% of a month's spending in the 19 days left, while the month it learned from had spent 82% in its first 12, so the month looked to end deep in the red.

The rest of the month now gets the income and spending the months learned from had after the same number of days into each, averaged over them. Bills, recurring income and rows dated after today stay on their dates, and the whole months after it are unchanged.

New integration test ForecastRestOfMonthTest (fails against the previous version). Three tests that assumed the share by days now work out what the same days had. Unit, integration, Psalm and CS are green.

[otherworld-dev]

Checked the fixes on a test instance. The placeholder, the new savings account, Mark Received, the transfer bills and the overdue bill all work now, and the new integration tests pass here too.

A few cases still count twice or drop out though:

• An income with no pattern that comes in through the bank. Mark Received without creating a transaction doesn't link the bank row, so a 2500 salary paid by "ACME LTD" came out at 5000 a month. The pattern's optional, so plenty of people will have this. Could it also match a credit in the income's account, within 20%, near one of its expected dates? withinExpectedWindow() in RecurringIncomeService already has the window. Same for bills paid under another name.
• Shared accounts. The rows linked to the owner's bills leave the average, but scheduledBills() only places the viewer's own bills, so for the person it's shared with the owner's 900 rent just disappears. Maybe only drop rows whose bill is one being placed?
• A bill nobody marks paid. With next due back in July, the first column got all five missed dates, 4500, and the balance went negative. Counting just the latest one would be safer.

Smaller ones: a salary that's more than 20% off one month (overtime) becomes all variable income and the trend shoots up, 2500/2500/3250 projected 5208 then 6833. And a bill with a generic pattern like AMAZON takes every Amazon purchase within 20% out of the average. A date window and one row per occurrence would sort both.

Also the placeholder test counts scheduled rows across every user, so a leftover row from another test breaks it. Worth scoping to the test's account.

[saesch83]

Thanks, all fair. Fixed in 10f6572:

• One payment per date: each date a bill or recurring income fell on claims the one row near it, within RecurringIncomeService::expectedWindowDays() (now public), that pays it. A row linked to the bill comes first, then one booked under the item's name or with every word of its pattern, then the nearest date, then the nearest amount. The past dates come from the bill's own schedule, going back before the bill was added (BillService::occurrencesBetween(..., sinceCreated: false)), since bills are usually set up after months of paying them.
• Income without a pattern paid under another name: without a pattern, a row in the item's own account within 20% of its amount and near one of its dates counts as its payment. So "ACME LTD" is the 2500 salary, and the same goes for bills. An item that has a pattern only takes the rows it matches, since the user said what its payments look like. An item with no account still needs its name or pattern.
• Overtime: a row recognised by name or pattern is the item's whatever its amount, so 2500/2500/3250 stays the salary's and doesn't turn into a rising trend.
• Generic patterns: one row per date, so AMAZON takes one purchase a month out of the average, the nearest one, not every purchase near its amount.
• Shared accounts: only rows linked to the user's own bills leave the average. The owner's 900 rent stays in the average of the person it's shared with. I kept the viewer's own paused bills and those excluded from the forecast out of the average, as before.
• Bill nobody marks paid: only its latest date due counts in the first column, and not if a row near that date paid it.
• Placeholder test: it now counts scheduled rows on its own account.

New integration tests for each case (income under another name, bill under another name, overtime, AMAZON, shared account, a bill unmarked since three months ago). All seven fail against the previous version. "Income without a pattern" now checks that the pension under the bank's name is taken and the refund beside it stays. Unit, integration, Psalm and CS are green.