Skip to content

chore: Improve gifts indexing - #255

Merged
danoctua merged 2 commits into
devfrom
chore/improve-gifts-indexing
Sep 25, 2026
Merged

danoctua merged 2 commits into
devfrom
chore/improve-gifts-indexing

Conversation

@danoctua

Copy link
Copy Markdown
Collaborator

Optimize Gift Indexer Resource Allocation and Auto-Block Deleted Accounts

Description

This pull request addresses two significant stability issues within the Telegram unique gifts indexer related to background Celery workers. First, it resolves resource exhaustion (SIGKILL) caused by file descriptor leaks and event loop thrashing when polling batches of users. Second, it gracefully handles "Bad Request: user not found" exceptions by
automatically marking deleted Telegram accounts as inactive, ensuring they are excluded from future syncs.

Fixes: # (add issue number if applicable)

Checklist

Before submitting your pull request, please ensure the following:

  • I have read the contributing guidelines.
  • I ran all tests and they passed successfully.
  • I added/updated documentation (if applicable).
  • I reviewed my own code and followed the project's code style.
  • I included relevant details in the PR description or commit messages.

Changes

  • Change 1: Refactored refresh_users_gifts_batch to execute all 25 users within a single asyncio.run() loop inside IndexerUserGiftAction, sharing a single BotApiGiftIndexer
    instance. This heavily cuts down event loop setup/teardown churn and drops SSL connection creation by 96%.
  • Change 2: Added an asyncio.sleep(0.250) during indexer.close() to give aiohttp the necessary event loop tick to elegantly terminate underlying SSL connections, preventing
    OS-level file descriptor leaks leading to SIGKILL.
  • Change 3: Ensured TelegramBadRequest exceptions correctly raise instead of silently continuing, preventing the indexer from falsely assuming users have lost all their gifts.
  • Change 4: Intercepted "user not found" API errors across both single and batch sync tasks to automatically batch-update those users as is_blocked = True via a new UserService. mark_as_blocked method.
  • Change 5: Updated UserService.get_all_telegram_ids to exclude blocked users entirely, preventing the global dispatcher from unnecessarily re-queueing deleted accounts.

How Has This Been Tested?

  • Verified logic against Celery worker logs locally to ensure the indexer gracefully skips nonexistent users and correctly cascades the batch updates.
  • Ensured Docker container memory limits (1024MB vs 512MB) no longer trigger OOM kills on standard operations due to the mitigated file descriptor bloat.
  • Formatted and validated through pre-commit (Ruff/ESLint).

Screenshots (if applicable)

N/A

Additional Notes

The sync_user_gifts method was updated to accept an optional indexer parameter to allow the single-user priority task (Flow B) to remain unchanged while still benefiting from the
performance optimizations in the master dispatcher (Flow C).


Thank you for contributing! 🎉

…API exceptions

This adds an asyncio.sleep(0.250) on close to allow aiohttp SSL transports to cleanly disconnect, preventing SIGKILL from file descriptor leaks. It also ensures 'user not found' exceptions are correctly raised rather than wiping a user's DB gifts.
@danoctua danoctua self-assigned this Sep 25, 2026
@danoctua danoctua added the gifts label Sep 25, 2026
@danoctua
danoctua force-pushed the chore/improve-gifts-indexing branch 2 times, most recently from e626f79 to 536526c Compare September 25, 2026 13:45
…sers

- Moves batch syncing into IndexerUserGiftAction to share a single BotApiGiftIndexer instance, vastly reducing overhead and event loop thrashing.
- Adds UserService.mark_as_blocked() and excludes blocked users from global fetches.
- Automatically marks users as blocked when Telegram returns a 'user not found' exception during gift sync.
@danoctua
danoctua force-pushed the chore/improve-gifts-indexing branch from 536526c to aabd84b Compare September 25, 2026 13:49
@danoctua
danoctua merged commit a4f84a0 into dev Sep 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant