fix(deps): update dependency faust-cchardet to v3 - #177
Open
renovate[bot] wants to merge 1 commit into
Open
Conversation
renovate
Bot
force-pushed
the
renovate/faust-cchardet-3.x
branch
2 times, most recently
from
August 12, 2026 05:50
66556fc to
afa4f16
Compare
renovate
Bot
force-pushed
the
renovate/faust-cchardet-3.x
branch
from
August 12, 2026 10:09
afa4f16 to
658b14b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
==2.2.1→==3.2.0Release Notes
faust-streaming/cChardet (faust-cchardet)
v3.2.0Compare Source
declare the
_cchardetextension free-threading compatible (#55_), soimporting
cchardetno longer re-enables the GIL process-wide onfree-threaded CPython (3.13t / 3.14t). Building now requires Cython >= 3.1,
which is checked at configure time; older Cython ignores the declaration
silently.
UniversalDetectorinstance methods additionally take aper-instance critical section, so sharing one detector across threads can no
longer corrupt the heap on a free-threaded build (it still produces
meaningless results). This is free on ordinary GIL builds.
Note that the published 3.1.0 and 3.0.1
cp314twheels predate this anddo re-enable the GIL on import; 3.2.0 is the first release that carries the
declaration.
fix a memory leak in
UniversalDetector: the underlyinguchardet_thandle was only released by an explicit
close(), so every detector thatwas simply dropped -- the documented pattern, since reading
resultfinalizes detection on its own -- leaked roughly 19 KB.
__dealloc__nowreleases it, and
close()andfeed()'s error path clear the handle soit cannot be released twice. Allocation moved from
__init__to__cinit__, which also fixes a segfault when a detector was built withoutrunning
__init__(via__new__, or a subclass that does not callsuper().__init__());__init__still resets the stream, so calling itagain on a live detector starts fresh as before.
detect()no longerleaks its detector if building the result string raises, and a failed
uchardet_new()now raisesMemoryErrorinstead of dereferencing NULL.stop aborting the interpreter when uchardet runs out of memory. uchardet is
C++ and allocates with plain
new, which throwsstd::bad_allocratherthan returning NULL, so its own out-of-memory checks never fired and the
exception unwound out of the extension into CPython's C frames -- undefined
behaviour, in practice
std::terminate()and aSIGABRT. Everyallocating uchardet entry point is now declared
except +, so anallocation failure surfaces as a normal
MemoryError. Relatedly,close()now releases the handle in afinally, so it cannot returnhaving released nothing when finalizing the stream raises.
document threading expectations for the Python API (
#55_).detect()issafe to call concurrently from multiple threads, while a
UniversalDetectorinstance holds the state of a single stream and mustnot be shared across threads without external synchronization -- use one
detector per thread. See the README.
ship type information (
#71_). The package now carries a PEP 561py.typedmarker, a
_cchardet.pyistub for the compiled extension, and annotationsacross the public API, so
detect()andUniversalDetector.resulttypecheck as
DetectionResultinstead ofAnydownstream. Both members ofthat result are
str | None/float | None, which type checkers nowrequire callers to handle. Also corrects
detect()'s docstring, whichclaimed it took
strwhen it has always requiredbytes... _#55: #55
.. _#71: #71
v3.1.0Compare Source
add a
system-uchardetMeson feature option, so distributions and sourcebuilds can link the system
uchardetinstead of the vendored copy(
#56_ by@mgorny,#64).disabled(the default, and what thepublished wheels use) always builds the bundled copy,
enabledrequiresthe system library and fails if it is missing or too old, and
autofalls back to the bundled copy. The system library must be recent enough to
expose
uchardet_get_n_candidates.Note that a system build currently detects non-UTF-8 input less accurately
than the bundled one, because the encoding-only multibyte prober added in
3.0.1 is part of the vendored copy. See the README for measurements and
guidance for packagers.
.. _#56: #56
.. _#64: #64
.. _@mgorny: https://github.com/mgorny
v3.0.1Compare Source
fix the severe detection slowdown introduced in 3.0.0 (
#57_). freedesktopuchardet 0.0.8 decodes every candidate to Unicode and fans the code points
through its generic language models -- work cChardet never exposes, since it
returns only encoding and confidence. Valid UTF-8 now short-circuits that
path, and the vendored uchardet is built with an encoding-only multibyte
group prober. Measured on the reporter's CC-News corpus: ~595 MB/s, against
~350 MB/s for 2.2.1 on the same runner.
fix non-UTF-8 input being reported as UTF-8. Without uchardet's language
pass,
nsUTF8Probernever rejects invalid byte sequences on its own; theoverlay validates the candidate instead. On the project's non-UTF-8
benchmark corpus this drops the mislabel rate from 16.2% to 0%.
fix heap corruption (
SIGABRT,free(): invalid next size) on longmultibyte input by feeding uchardet in bounded chunks, keeping its internal
1024-entry code-point buffer in range.
add a benchmark CI job that compares each change against the two most recent
releases and fails on a throughput regression or a UTF-8 mislabel rate above
the configured ceiling.
Known limitation: on genuinely non-UTF-8 input, throughput is still about
0.6x of 2.2.1. The remaining cost is the single-byte prober's language
models inside freedesktop uchardet, tracked upstream as
uchardet#38_... _#57: #57
.. _uchardet#38: https://gitlab.freedesktop.org/uchardet/uchardet/-/issues/38
v3.0.0Compare Source
Yanked from PyPI: this release is orders of magnitude slower than 2.2.1
(
#57_), mislabels non-UTF-8 input as UTF-8, and can abort on long multibyteinput. Use 3.0.1 or later.
#50_)UTF-8-SIGfor downstream compatibilityMAC-CENTRALEUROPElabel to protect downstreamopen()/decode().. _#50: #50
Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.