Allow writing of incomplete UTF-8 sequences to the Windows console via stdout/stderr

[Count-Count]

Problem

Writes of just an incomplete UTF-8 byte sequence (e.g. b"\xC3" or b"\xF0\x9F") to stdout/stderr with a Windows console attached error with io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" even though further writes could complete the codepoint. This is currently a rare occurence since the linewritershim implementation flushes complete lines immediately and buffers up to 1024 bytes for incomplete lines. It can still happen as described in #83258.

The problem will become more pronounced once the developer can switch stdout/stderr from line-buffered to block-buffered or immediate when the changes in the "Switchable buffering for Stdout" pull request (#78515) get merged.

Patch description

If there is at least one valid UTF-8 codepoint all valid UTF-8 is passed through to the extracted write_valid_utf8_to_console() fn. The new code only comes into play if write() is being passed a short byte slice comprising an incomplete UTF-8 codepoint. In this case up to three bytes are buffered in the IncompleteUtf8 struct associated with Stdout / Stderr. The bytes are accepted one at a time. As soon as an error can be detected io::ErrorKind::InvalidData, "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" is returned. Once a complete UTF-8 codepoint is received it is passed to the write_valid_utf8_to_console() and the buffer length is set to zero.

Calling flush() will neither error nor write anything if an incomplete codepoint is present in the buffer.

Tests

Currently there are no Windows-specific tests for console writing code at all. Writing (regression) tests for this problem is a bit challenging since unit tests and UI tests don't run in a console and suddenly popping up another console window might be surprising to developers running the testsuite and it might not work at all in CI builds. To just test the new functionality in unit tests the code would need to be refactored. Some guidance on how to proceed would be appreciated.

Public API changes

• std::str::verifications::utf8_char_width() would be exposed as std::str::utf8_char_width() behind the "str_internals" feature gate.

Related issues

• Fixes Writing valid UTF-8 to the Windows terminal using stdout().write() can incorrectly return an error #83258.
• PR Switchable buffering for Stdout #78515 will exacerbate the problem.

Open questions

• Add tests?
• Squash into one commit with better commit message?

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @sfackler (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

Should stdout use STDERR handle here?

Should stdout use STDERR handle here?

Cut&Paste mistake, will fix it right away. Thanks for spotting it!

🔔 This is now entering its final comment period, as per the review above. 🔔

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

@bors r+

📌 Commit fbfde7e has been approved by m-ou-se

⌛ Testing commit fbfde7e with merge cc9bb15...

☀️ Test successful - checks-actions
Approved by: m-ou-se
Pushing cc9bb15 to master...