add PyO3 FFI checks to the suite

[davidhewitt]

I added nox -s ffi-check job from PyO3 which runs some sanity checks that PyO3's FFI definitions match what PyPy is exporting from its headers.

NB from running this locally I think I observed that PyPy needs to implement PEP 623.

[mattip]

I opened PyO3/pyo3#6345 about the c_int failures. The missing _PyInterpreterFrame is typedefed in frameobject.h, i wonder why it was not found

[davidhewitt]

I opened PyO3/pyo3#6374 which should address both issues, once merged we'll see what the next issues are 👀

[davidhewitt]

Ok we're now at a point where we see the wstr fields removed in PEP 623, I guess that's an action on PyPy.

Plus I think CPython changed the PyObject struct to have a union for one field, PyPy might not care about that in which case we can action that on the PyO3 side.

[mattip]

I am working on the wstr change, thanks for pointing it out. Then I will think about the union. We need to be able to parse the struct with pycparser, I don't know if it will handle that and if not how we can have an internal/external declaration. I would like to move to 100% CPython compatibility for the limited API functions and structs.

[bschoenmaeckers]

I would like to move to 100% CPython compatibility for the limited API functions and structs.

This is cool, as RustPython is also aiming to support CPython abi3t!

[davidhewitt]

I would like to move to 100% CPython compatibility for the limited API functions and structs.

This would be nice, you should definitely check out 3.15's abi3t which removes the layout of the PyObject struct from the ABI. I wonder if abi3t would allow PyPy to do more of the optimizations which e.g. HPy also makes possible.

(I realise that PyPy probably has a fair bit of work to catch up with 3.15, however...)

[bschoenmaeckers]

I'm wondering if it's possible to backport abi3t to previous versions. But this probably lacks essential functions

[mattip]

I refactored to remove the wstr field, thanks for pointing out 3.12 removed it. As for the anonymous structure, that is a deeper problem with the PyPy RPython infrastructure. We need to be able to parse the struct definition with internal tools, they cannot handle a union. Is there a way to simplify that from the PyO3 side?

[mattip]

closing/reopening to retrigger CI

[mattip]

I also merged the PR to not mangle names of all the limited-api functions. This will undoubtedly mess up the PyO3 special-casing of function names.

[davidhewitt]

PyO3/pyo3#6389 will hopefully catch up PyO3 to the next phase of changes here.

[mattip]

Closed/reopened to retrigger CI.

[mattip]

PyO3 Prs were merged. I triggered a run against latest PyO3 HEAD, let's see what happens

[mattip]

https://github.com/pypy/binary-testing/actions/runs/35085978334

[davidhewitt]

Looks like next step is PyO3/pyo3#6421

[davidhewitt]

That's now merged; want to rerun here again?

[mattip]

Close/reopen to restart CI

[mattip]

Lots of mangled names still appear in the 3.12 error report.

[davidhewitt]

Hopefully we're finally there after PyO3/pyo3#6422

[davidhewitt]

Locally I now don't get any more mangled name issues.

[mattip]

Close/reopen to trigger CI

[mattip]

Down to one failure and one problem:

• The failure: PyPy does not prefix the class name with builtins. I will fix that for tomorrow's nightly build.
• The problem: the noxfile should use filtered_args here in the join instead of args:

    try:
        filtered_args = [x for x in args if x is not None]
        session.run(*filtered_args, **kwargs)
    except nox.command.CommandFailed:
        failed = True
        raise
    finally:
        if is_github_actions:
            print("::endgroup::", file=sys.stderr)
            # Defer the error message until after the group to make them easier
            # to find in the log
            if failed:
                command = " ".join(args)
                print(f"::error::`{command}` failed", file=sys.stderr)

[davidhewitt]

Thanks, PyO3 side up at PyO3/pyo3#6423

Hopefully we go green with tomorrow's nightly PyPy!

[davidhewitt]

The failure: PyPy does not prefix the class name with builtins. I will fix that for tomorrow's nightly build.

For what it's worth, builtins here will just be the name of the __module__ set by PyO3 for that type - I suspect CPython is using some of the PEP 737 machinery here (though I have not checked the source)

[mattip]

There is a difference in the error message when disallowing new for pure python and for c-extension classes. I fixed it in pypy/pypy@be1d339