RFI: Rebalancing unicode escaping in str::escape_debug for internationalization

[the8472]

Triggered by discussion in #158286 @rust-lang/libs-api concluded that our debug escapes are escaping too many characters and are not internationalization-friendly.
E.g. ຄົນສົ່ງໜັງສື currently gets escaped to ຄ\u{ebb}ນສ\u{ebb}\u{ec8}ງໜ\u{eb1}ງສ\u{eb7}

We don't have precise rules ourselves what should and shouldn't be escaped and we'd like to gather input and help from unicode/internationalization experts to find a better balance between various debugging goals and learn what the tradeoffs are.
Once we better understand the tradeoffs and have decided on an approach implementation help, testdata and similar would also be appreciated.

Some desiderata that were brought up during discussion:

• living languages¹ should not be excessively mangled by escaping and remain readable.
  A few ligatures getting torn per sentence probably is ok, every other letter getting replaced by an escape wouldn't be
• surprising/unexpected/unprintable characters - e.g. a ZWJ or BOM in the middle of ascii strings/latin script - should get escaped
• debug printing shouldn't mess up terminals → control sequences and directionality overrides should remain escaped
• binary size might also be a concern if additional unicode tables are required

¹ "Any text one could pluck from a random international wikipedia page that isn't about ancient typography"

[clarfonthey]

A few passive thoughts:

• Script detection probably can be done with minimal table sizes since most scripts are clustered close together, with minimal exceptions. We could use this to determine if a character is "surprising" to show up in a string.
• Worst case, we could potentially simplify the debug-printing logic in the debug profile somehow to reduce binary size. That's probably going to require more coordination to make it configurable in cargo, however. This goes along with existing work to reduce the number of Unicode tables present.

[workingjubilee]

cc @Manishearth in the likely event you either have an opinion or know someone who has free time and an opinion.

[Manishearth]

I do, I'll post one when I'm back from vacation next week

[Manishearth]

I agree with the stated principles in their general direction. I'd like to add one more: it would be nice if NFD text were escaped more so that you can tell that something is an NFC vs NFD issue. Most text out there is NFC, and I think some OSes even modify normalization to NFC when you copy paste. The brittleness makes NFD a bad idea for e.g. assert_eq!() checks.

This does mean that living languages' text in NFD will be mangled, and I think that is a desirable property. I've had problems with this many times.

surprising/unexpected/unprintable characters - e.g. a ZWJ or BOM in the middle of ascii strings/latin script - should get escaped

I actually think that escaping ZWJ is generally good. Even the scripts that use ZWJ mostly use it for minor things, well in the realm of "not excessively mangled". The only thing that would get annoyingly mangled is emoji. Which is fine, terminals don't handle emoji well anyway.

I'd generally recommend this be done as a list of code points with no fancy logic. Anything else will increase the binary size. Is there a need for script detection at all?

[Jules-Bertholet]

This does mean that living languages' text in NFC will be mangled, and I think that is a desirable property. I've had problems with this many times.

Is this a typo? Do you think we should mangle living languages?

[Manishearth]

NFD, sorry

[Jules-Bertholet]

I'd generally recommend this be done as a list of code points with no fancy logic. Anything else will increase the binary size. Is there a need for script detection at all?

This is what we do today. But as the Lao example in OP demonstrates, with just a list of codepoints it's impossible to ensure all unnormalized text is mangled without also mangling the NFC of some living languages. The only way to "have it all" is to actually do normalization

[Manishearth]

I don't think we need perfection, aiming for perfect heuristics is a mistake. We should have some low code size heuristics that approximate the principles above.