Skip to content

fix: prevent Actor log-streaming thread from crashing on stream timeout#944

Draft
vdusek wants to merge 2 commits into
masterfrom
fix/streamed-log-stream-timeout
Draft

fix: prevent Actor log-streaming thread from crashing on stream timeout#944
vdusek wants to merge 2 commits into
masterfrom
fix/streamed-log-stream-timeout

Conversation

@vdusek

@vdusek vdusek commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

After a successful ActorClient.call(), the background log-streaming thread could crash with an uncaught impit.TimeoutException, printing a traceback even though the run finished fine. Reported in #945.

Root cause

StreamedLog requested the log stream with a bounded 30s timeout. impit applies its timeout to the whole request, including the streamed response body (not just connect / read-idle), so any run that streamed logs for longer than the timeout tripped impit.TimeoutException mid-stream, and the sync thread had no handler. The async twin caught it but logged a spurious ERROR + traceback. A bounded value could never be large enough anyway: every timeout tier is capped at DEFAULT_TIMEOUT_MAX (360s), so even long would truncate a run past ~6 minutes.

Fix

  • Request the log stream with no_timeout (the only tier above the 360s cap; maps to impit's ~24h ceiling), so it streams until the server closes it with EOF at run finish. The JS client takes the same approach, setting no body timeout on the log stream.
  • Treat impit.TimeoutException as an expected terminal condition in both the sync and async paths (end quietly instead of crashing / error-logging), while still logging any other failure.
  • Run the sync streaming thread as a daemon so a read blocked on a stalled connection can never hold up interpreter shutdown.

Known limitation

A parked synchronous iter_bytes() read cannot be interrupted from another thread, so with no_timeout, StreamedLog.stop() on a stream that is still running but momentarily silent now waits for the next chunk or EOF rather than the old 30s bound. The ActorClient.call() path is unaffected, since run finish sends EOF and the stream ends promptly. This matches the JS client's behavior. Making manual stop() responsive for silent/standby streams would need a reconnect/resume mechanism, which the log API doesn't currently support (no resume offset, and it replays the whole log on reconnect).

Behavior note

On long-running async redirection, the ERROR: Log redirection stopped due to unexpected error: line that previously fired on the 360s timeout no longer fires. The message text, its level, and the logger name are unchanged; it now fires only on genuine, non-timeout errors.

@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Jul 11, 2026
@vdusek vdusek self-assigned this Jul 11, 2026
@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.95238% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.51%. Comparing base (68c0e24) to head (29fcef6).
⚠️ Report is 11 commits behind head on master.

Files with missing lines Patch % Lines
src/apify_client/_streamed_log.py 80.95% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     apify/apify-sdk-python#944      +/-   ##
==========================================
- Coverage   94.56%   94.51%   -0.06%     
==========================================
  Files          48       48              
  Lines        5119     5127       +8     
==========================================
+ Hits         4841     4846       +5     
- Misses        278      281       +3     
Flag Coverage Δ
integration 92.56% <61.90%> (-0.46%) ⬇️
unit 83.42% <80.95%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants