Look the SSLContext up from the SSL_CTX in the ALPN/NPN callbacks

[jeremy]

Fixes #1088.

ossl_sslctx_mark uses rb_gc_mark_movable, so the SSLContext relocates. Its VALUE is stored in four places:

where	updated on compaction?
SSL_CTX ex_data	yes, by ossl_sslctx_compact
SSL_CTX_set_next_protos_advertised_cb arg (:810)	no
SSL_CTX_set_next_proto_select_cb arg (:814)	no
SSL_CTX_set_alpn_select_cb arg (:830)	no

After a compaction the three callback arguments hold the pre-move address, and each callback does rb_attr_get on it. Registration is one-shot — ossl_sslctx_setup returns early when self is frozen — so the address is captured at the first handshake and never refreshed.

This is a regression from the switch to rb_gc_mark_movable; every release pins with rb_gc_mark and is unaffected.

Approach

Rather than adding three fix-ups to ossl_sslctx_compact, this removes the three extra copies. All three callbacks already receive the SSL, and SSL_get_SSL_CTX(ssl)'s ex_data is the copy that is already maintained — so they can look the object up and pass NULL as the callback argument.

That leaves exactly one stored copy and one place to keep current, which seemed worth more than the alternative: ossl_sslctx_compact would otherwise have to re-register all three callbacks from inside a GC compaction callback, since OpenSSL exposes no getter for these arguments.

The helper sits next to the other ex_data accessors and mirrors the idiom already used in the session-remove callback.

Verification

Built HEAD (9796ee8) and the patched tree in the same step that ran the tests, so the artifact can't drift from the source. Ruby 4.0.6, OpenSSL 3.5.6, in a container.

                          master     with this patch
NPN :810 (server side)    3/3 fail   3/3 pass
NPN :814 (client side)    3/3 fail   3/3 pass
ALPN :830                 3/3 fail   3/3 pass
control (no compaction)   3/3 pass   3/3 pass

The two NPN sites are isolated by putting the server and client contexts in different processes (fork + pipes), so exactly one of them is subject to the compaction — otherwise the server fails first and masks the client site entirely.

Both ends pin max_version to TLS 1.2. That detail is why #1088 recorded the NPN sites as "code reading only": NPN isn't sent at TLS 1.3, so an unconstrained test negotiates 1.3, neither callback fires, and the run looks clean. The scripts now assert the callbacks actually ran before reporting anything.

Failure modes observed on master, both from the same defect depending on what lands in the vacated slot:

• :810 — SEGV in ssl_npn_advertise_cb, faulting at 0x4, i.e. RSTRING_PTR(Qnil)
• :814 — NoMethodError: undefined method 'call' for nil out of SSLSocket#connect, with the peer logging sslv3 alert handshake failure

Reachability

npn_protocols=, npn_select_cb= and alpn_select_cb= are public documented accessors. The trigger is the application's own GC.compact, not anything an attacker supplies, which is why this is a public issue rather than a private report.

One honest caveat: confirmed under GC.verify_compaction_references. I could not get a plain GC.compact to relocate the SSLContext in a small process, so I'm not claiming a spontaneous production rate — every run where the context did relocate failed (30/30), and every run where it didn't survived (60/60).

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.