Skip to content

fix(deps): update dependency faust-cchardet to v3 - #177

Open
renovate[bot] wants to merge 1 commit into
mainfrom
renovate/faust-cchardet-3.x
Open

fix(deps): update dependency faust-cchardet to v3#177
renovate[bot] wants to merge 1 commit into
mainfrom
renovate/faust-cchardet-3.x

Conversation

@renovate

@renovate renovate Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

This PR contains the following updates:

Package Change Age Confidence
faust-cchardet ==2.2.1==3.2.0 age confidence

Release Notes

faust-streaming/cChardet (faust-cchardet)

v3.2.0

Compare Source

  • declare the _cchardet extension free-threading compatible (#55_), so
    importing cchardet no longer re-enables the GIL process-wide on
    free-threaded CPython (3.13t / 3.14t). Building now requires Cython >= 3.1,
    which is checked at configure time; older Cython ignores the declaration
    silently. UniversalDetector instance methods additionally take a
    per-instance critical section, so sharing one detector across threads can no
    longer corrupt the heap on a free-threaded build (it still produces
    meaningless results). This is free on ordinary GIL builds.

    Note that the published 3.1.0 and 3.0.1 cp314t wheels predate this and
    do re-enable the GIL on import; 3.2.0 is the first release that carries the
    declaration.

  • fix a memory leak in UniversalDetector: the underlying uchardet_t
    handle was only released by an explicit close(), so every detector that
    was simply dropped -- the documented pattern, since reading result
    finalizes detection on its own -- leaked roughly 19 KB. __dealloc__ now
    releases it, and close() and feed()'s error path clear the handle so
    it cannot be released twice. Allocation moved from __init__ to
    __cinit__, which also fixes a segfault when a detector was built without
    running __init__ (via __new__, or a subclass that does not call
    super().__init__()); __init__ still resets the stream, so calling it
    again on a live detector starts fresh as before. detect() no longer
    leaks its detector if building the result string raises, and a failed
    uchardet_new() now raises MemoryError instead of dereferencing NULL.

  • stop aborting the interpreter when uchardet runs out of memory. uchardet is
    C++ and allocates with plain new, which throws std::bad_alloc rather
    than returning NULL, so its own out-of-memory checks never fired and the
    exception unwound out of the extension into CPython's C frames -- undefined
    behaviour, in practice std::terminate() and a SIGABRT. Every
    allocating uchardet entry point is now declared except +, so an
    allocation failure surfaces as a normal MemoryError. Relatedly,
    close() now releases the handle in a finally, so it cannot return
    having released nothing when finalizing the stream raises.

  • document threading expectations for the Python API (#55_). detect() is
    safe to call concurrently from multiple threads, while a
    UniversalDetector instance holds the state of a single stream and must
    not be shared across threads without external synchronization -- use one
    detector per thread. See the README.

  • ship type information (#71_). The package now carries a PEP 561 py.typed
    marker, a _cchardet.pyi stub for the compiled extension, and annotations
    across the public API, so detect() and UniversalDetector.result type
    check as DetectionResult instead of Any downstream. Both members of
    that result are str | None / float | None, which type checkers now
    require callers to handle. Also corrects detect()'s docstring, which
    claimed it took str when it has always required bytes.

.. _#55: #​55
.. _#71: #​71

v3.1.0

Compare Source

  • add a system-uchardet Meson feature option, so distributions and source
    builds can link the system uchardet instead of the vendored copy
    (#56_ by @mgorny, #64). disabled (the default, and what the
    published wheels use) always builds the bundled copy, enabled requires
    the system library and fails if it is missing or too old, and auto
    falls back to the bundled copy. The system library must be recent enough to
    expose uchardet_get_n_candidates.

    Note that a system build currently detects non-UTF-8 input less accurately
    than the bundled one, because the encoding-only multibyte prober added in
    3.0.1 is part of the vendored copy. See the README for measurements and
    guidance for packagers.

.. _#56: #​56
.. _#64: #​64
.. _@​mgorny: https://github.com/mgorny

v3.0.1

Compare Source

  • fix the severe detection slowdown introduced in 3.0.0 (#57_). freedesktop
    uchardet 0.0.8 decodes every candidate to Unicode and fans the code points
    through its generic language models -- work cChardet never exposes, since it
    returns only encoding and confidence. Valid UTF-8 now short-circuits that
    path, and the vendored uchardet is built with an encoding-only multibyte
    group prober. Measured on the reporter's CC-News corpus: ~595 MB/s, against
    ~350 MB/s for 2.2.1 on the same runner.

  • fix non-UTF-8 input being reported as UTF-8. Without uchardet's language
    pass, nsUTF8Prober never rejects invalid byte sequences on its own; the
    overlay validates the candidate instead. On the project's non-UTF-8
    benchmark corpus this drops the mislabel rate from 16.2% to 0%.

  • fix heap corruption (SIGABRT, free(): invalid next size) on long
    multibyte input by feeding uchardet in bounded chunks, keeping its internal
    1024-entry code-point buffer in range.

  • add a benchmark CI job that compares each change against the two most recent
    releases and fails on a throughput regression or a UTF-8 mislabel rate above
    the configured ceiling.

    Known limitation: on genuinely non-UTF-8 input, throughput is still about
    0.6x of 2.2.1. The remaining cost is the single-byte prober's language
    models inside freedesktop uchardet, tracked upstream as uchardet#38_.

.. _#57: #​57
.. _uchardet#38: https://gitlab.freedesktop.org/uchardet/uchardet/-/issues/38

v3.0.0

Compare Source

Yanked from PyPI: this release is orders of magnitude slower than 2.2.1
(#57_), mislabels non-UTF-8 input as UTF-8, and can abort on long multibyte
input. Use 3.0.1 or later.

  • switch to upstream freedesktop uchardet and its multi-candidate API (#50_)
  • normalize the UTF-8 BOM label to UTF-8-SIG for downstream compatibility
  • normalize the MAC-CENTRALEUROPE label to protect downstream
    open() / decode()

.. _#50: #​50


Configuration

📅 Schedule: (UTC)

  • Branch creation
    • At any time (no schedule defined)
  • Automerge
    • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@renovate
renovate Bot force-pushed the renovate/faust-cchardet-3.x branch 2 times, most recently from 66556fc to afa4f16 Compare August 12, 2026 05:50
@renovate
renovate Bot force-pushed the renovate/faust-cchardet-3.x branch from afa4f16 to 658b14b Compare August 12, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants