Interoperation with other libraries that use FLINT

[magv]

Hi, all. I'm writing a Python library that uses FLINT internally (i.e. there's some C code that uses FLINT and is then interfaced with via Cython), and I'd like to interoperate with python-flint by accepting and/or producing python-flint values (such as fmpq_mpoly). Could you advise on how to organize this best? Currently I'm just converting to Python and/or SymPy native types and back, and this is comically slow.

What I would like to have, ideally is:

1. A way to take a python-flint value like fmpq_mpoly, and extract a pointer to the underlying fmpq_mpoly_struct, which I can pass to my C code directly.
   • Maybe this could be accomplished by shipping some Cython header files.
2. A way to take an fmpq_mpoly_struct and produce a python-flint fmpq_mpoly object from it.
3. A way to guarantee ABI compatibility with the FLINT version that python-flint builds against. Currently I'm building a static FLINT library that I link my Cython extensions with. I guess I would need to match the version of FLINT to the one you've used for this to be seamless?
   • A maximalist solution here would be a separate flint-dev-lib PyPI package that installs FLINT sources and prebuilt libraries to link against, so that both python-flint and other libraries that build against FLINT could be consistent. This package could be a part of python-flint as an alternative.
     Of course, this would require gmp-dev-lib and mpfr-dev-lib packages to exist, but we already build these libraries (same as you do), so it's not a major additional burden.

Anyway, I'd like to learn what you think of how to organize interoperability in Python.

[oscarbenjamin]

• Maybe this could be accomplished by shipping some Cython header files.

This is something that could be done. I don't have experience with how exactly that works though. Shipping the cython headers is easy but I'm not sure how exactly that works in terms of compatibility between separate packages that might be built with different Cython versions or built as separate wheels etc.

2. A way to take an fmpq_mpoly_struct and produce a python-flint fmpq_mpoly object from it.

I guess this would be straight-forward if the Cython headers were available.

• a separate flint-dev-lib PyPI package

This kind of thing has been talked about a lot in Python packaging and I'm not sure that there is yet a solution for it. Supposedly PEP 825 and anticipated followups will make it possible to have wheels with ABI dependencies but that is still a way off.

If it were possible then I think that gmpy2, python-gmp and other packages would want to share the GMP and MPFR libs.

[magv]

I too hope that PEP 825 will help clean up the GPU library mess, but we don't need to go that far for the base implementation of the idea: we already ship PyPI packages that build GMP and MPFR on demand (configured with the most compatible options), so shipping the headers and the libraries is a matter of not deleting them after the build. PEP 825 would make AVX-enabled builds possible as an addition; otherwise just PEP 621 / pyproject.toml is enough -- i.e. this is possible today.

You're right that several packages would benefit from such effort. I think people aren't doing this yet because they haven't gotten around to it, not because the tech is lacking. I think it is inevitable that someone would package a whole Linux distro as a PyPI package in the future :)

Well, anyway. I'm not trying to sell an experimental idea here -- I want to find an easy way forward.

I also have little experience in using Cython this way, but I think the idea is that you ship all structure and function definitions as .pxd files, which you already have, and users should be able to cimport those, and have their Cython figure out the data layout of your objects that way. There is a way to specify minimal Cython version, by naming the files .cython-30.pxd or so, but there's no version pinning as far as I can tell. The implication of the docs (i.e. here and here) seems to be that Cython ABI is backwards compatible.

But in any case, I also worry that FLINT itself is not as backwards compatible as Cython, and I think we could use the same solution to both problems: each version of python-flint could advertise the exact Cython and FLINT versions it uses for the wheels, so users could use the same version. Python-flint would need to commit to these versions in a predictable manner, i.e. have all 0.9.x use the same, and document this commitment. This way users would have the option to be extra sure.

[oscarbenjamin]

we already ship PyPI packages that build GMP and MPFR on demand

What are the PyPI names of these packages?

Where are their repos?

Also who is "we"?

I think the idea is that you ship all structure and function definitions as .pxd files

That sounds fine to me. It has been discussed a few times but I just don't have direct experience of using Cython across separately built package boundaries like that. If we do that then I guess we should at least have a CI job that checks that it works but otherwise I see no problem shipping the pxd files. I guess we would probably need to expose more methods etc in the pxd files as well and think about what counts as "public API" there though.

I also worry that FLINT itself is not as backwards compatible as Cython

For python-flint each minor (3.x) release of either Cython or FLINT has more than 50% change of breakage. That is why they have upper version caps in pyproject.toml and meson.build. Usually that is a build-time breakage but for FLINT I imagine it would also be an ABI break most of the time.

PEP 825 would make AVX-enabled builds possible as an addition; otherwise just PEP 621 / pyproject.toml is enough -- i.e. this is possible today.

Is it though?

I mention PEP 825 because I think this exact topic was discussed there.

Hopefully @rgommers (one of the PEP authors) does not mind being CCed here to correct me if I'm wrong. The situation here is basically the same as NumPy and SciPy trying to share a BLAS library across PyPI wheels. My understanding is that PEP 825 (and/or its successors) is what is needed to bring the missing pieces that can make that work properly.

[magv]

What are the PyPI names of these packages?
Where are their repos?
Also who is "we"?

Sorry for being vague. The packages are pySecDec (repo; this builds GMP), gcad (repo; this builds GMP, MPFR, FLINT), form-bin (repo, build system repo; also builds GMP, MPFR, and FLINT), and more to come. "We" are me and my colleagues (physicists), but also could be you in the near future: you already build FLINT for the wheels, you just do it separately from the build system of this package, but you could do it as a part of it too with a similar effect.

I think the idea is that you ship all structure and function definitions as .pxd files

That sounds fine to me. It has been discussed a few times but I just don't have direct experience of using Cython across separately built package boundaries like that. If we do that then I guess we should at least have a CI job that checks that it works but otherwise I see no problem shipping the pxd files. I guess we would probably need to expose more methods etc in the pxd files as well and think about what counts as "public API" there though.

Yes, everything in the .pxd should be considered stable API if these are to be installed. So anything unstable should be moved out.

Naturally, I'm volunteering to test how this intergration will work out from the user side. I'm mostly interested in the Cython class definitions and functions like create_fmpq_mpoly; I think I should be able to replicate everything else with FLINT.

For python-flint each minor (3.x) release of either Cython or FLINT has more than 50% change of breakage. That is why they have upper version caps in pyproject.toml and meson.build. Usually that is a build-time breakage but for FLINT I imagine it would also be an ABI break most of the time.

I feared this might be the case. Then, strict version pinning is needed.

PEP 825 would make AVX-enabled builds possible as an addition; otherwise just PEP 621 / pyproject.toml is enough -- i.e. this is possible today.

Is it though?

I mention PEP 825 because I think this exact topic was discussed there.

Hopefully @rgommers (one of the PEP authors) does not mind being CCed here to correct me if I'm wrong. The situation here is basically the same as NumPy and SciPy trying to share a BLAS library across PyPI wheels. My understanding is that PEP 825 (and/or its successors) is what is needed to bring the missing pieces that can make that work properly.

I guess I'm not up to date on this discussion. My understanding was that PEP 825 speaks about how different BLAS library variants can be selected based on the CPU arch. My thinking is that if you're OK with building the slowest, most compatible build, and you don't try to provide package variants that depend on different versions of the dependency, then you can just go and do it today. We've certainly made something work for the projects I mentioned.

[rgommers]

My understanding was that PEP 825 speaks about how different BLAS library variants can be selected based on the CPU arch.

BLAS is just an example. You can equally well build x86-64-v1 (works everywhere) and a x86-64-v4 (AVX512 required) wheels with it. And then the installer (pip, uv) select the optimal one based on your CPU capabilities.

You cannot really do that today, you either do a fancier version with dynamic dispatch (a la https://numpy.org/neps/nep-0038-SIMD-optimizations.html), or you have to build the slowest version that works everywhere - and that's then the one distributed on PyPI. Locally you can build whatever you want of course.

---

If you're asking me about the ABI compatibility part though, that's a more complex question. PEP 825 doesn't magically handle ABI breaks, those are always problematic. Rather than dive into that, let me ask what you want to share exactly and why. flint isn't like a runtime with state/threading that are designed for there being only a single instance of it in a process (right?). So you'd only share to get one of these benefits:

1. Shared maintenance of CI, building, patching, etc. of FLINT.
2. Large reduction in binary size.

For (2): I see python-flint is ~10 MB as a wheel and 25 MB on disk, and about 60% (15 MB) of that is libflint/libgmpr/libgmp. You'd only save that 15 MB in case you install two packages that both need/ship those shared libraries. That's nice, but not something to do a lot of work for or introduce == dependencies that are sensitive to ABI.

So, I guess the main reason should be (1), reduce maintenance overhead? That's pretty much exactly why we have a separate repo for OpenBLAS (https://github.com/MacPython/openblas-libs/), which we have wheels for, which in the end get used during the builds of both NumPy and SciPy, and then vendored into numpy/scipy wheels.

[magv]

Some reasons to have a common flint-dev-lib package in the style of your openblas-libs (i.e. with all C headers and prebuilt libs) would be partially (1), and partially ABI compatibility: flint breaks it often, and if we want for different libraries to be able to pass flint objects between each other, we need to make sure all of these libraries were built with compatible (or identical) flint versions.

(But again, I'm happy with just .pxd files and a commitment to a particular flint+Cython version for the wheels).

PS. @rgommers: I was at your CERN talk two weeks ago; it was interesting, thanks a lot.

[oscarbenjamin]

flint isn't like a runtime with state/threading that are designed for there being only a single instance of it in a process (right?).

FLINT has some global state but I'm not sure that it would be problematic if there were multiple instances in a single process. One example of python-flint setting this would be:

In [3]: import flint

In [4]: flint.ctx.threads = 4

Internally that calls flint_set_num_threads which sets a global variable that affects how many threads any given FLINT operation is allowed to use (big operations can spawn threads internally). I'm not sure how that kind of global variable works if there are multiple instances of FLINT in the process.

Apart from a few things like that though most FLINT operations are local and don't use global state. FLINT objects can be shared across threads as long as they are not concurrently mutated.

Reducing maintenance overhead would be a big benefit. There are lots of Python libraries that wrap GMP and MPFR and a few that wrap FLINT. We have multiple patches here for different versions of these. CI in this repo takes about an hour most of which is just building these libraries (maybe we should just cache them though).

The development experience would be nicer for contributors if they didn't need to build these things separately. On Windows it would mean that the upstream package could use MinGW but python-flint contributors could use MSVC to build python-flint itself so they don't need another toolchain.

On the other hand if this stuff was moved out to flint-dev-lib I imagine that it would still be the same people maintaining it for the same amount of work. If it is possible for other libraries to just depend on python-flint and still get the same advantage of flint-dev-lib then maybe it doesn't really help.

(But again, I'm happy with just .pxd files and a commitment to a particular flint+Cython version for the wheels).

I'm not sure exactly what commitment you mean here. Source compatibility for different versions is listed at:
https://github.com/flintlib/python-flint/blob/main/doc/source/build.rst#compatibility-and-supported-versions

Usually python-flint keeps the same oldest supported versions of Cython and FLINT while extending the upper version caps after each minor release of these. One exception to that is that Cython 3.1 is the minimum version for free-threading.

Another exception is that now that Ubuntu 26.04 (ubuntu-latest in GitHub Actions) has FLINT 3.4 I think it is time to drop FLINT < 3.4 which would allow removing most FLINT compatibility code from python-flint.

The PyPI wheels always use the latest version of FLINT and Cython at the time of python-flint release. Ideally there would be a python-flint release after each FLINT release but in the last year or so FLINT sped up and python-flint hasn't kept up (it is up to date now).

[magv]

The commitment could be something like "All 0.9.x wheels will be built with FLINT 3.5.x and Cython 3.3.x". This way I can put my runtime dependency as "python-flint~=0.9.0", build dependency as "cython~=3.3.0", compile my code with FLINT 3.5, upload the wheels, and sleep safely, knowing that if someone will pip install my-package, they will get a consistent set of libraries.

[oscarbenjamin]

The commitment could be something like "All 0.9.x wheels will be built with FLINT 3.5.x and Cython 3.3.x"

That commitment is totally fine.

We should perhaps document it somewhere though I guess.

Then I guess it is just a question of shipping the .pxd files.

[rgommers]

Thanks for the details! And glad you liked the talk:)

The maintenance aspect makes a lot of sense. Sounds like you found a solution that works for other packages that need a Cython interface at least. Which is probably the best interface anyway, since it hides all the C symbol name madness.

I'm not sure how that kind of global variable works if there are multiple instances of FLINT in the process.

Shouldn't be an issue - mangling with auditwheel et al. ensures that there is zero cross-talk.

On Windows it would mean that the upstream package could use MinGW but python-flint contributors could use MSVC to build python-flint itself so they don't need another toolchain.

FYI we just dropped mingw-w64 for SciPy wheels last week, at long last. If the issue is that MSVC is unsupported (as it is for us), switching to clang-cl is probably just fine - and that's a way better compiler. Some problems immediately disappeared, and our binaries got noticeably smaller.

[oscarbenjamin]

If the issue is that MSVC is unsupported (as it is for us), switching to clang-cl is probably just fine -

The basic issue is that MSVC cannot compile GMP with --enable-fat that uses runtime selection of assembly routines. Currently python-flint's Windows wheels are built by first building GMP/MPFR/FLINT using mingw-w64-ucrt, post-processing those for MSVC:

python-flint/bin/cibw_before_all_windows.sh

Lines 15 to 26 in 6708168

	mkdir -p .local/lib
	cd .local/bin
	for dll_file in libgmp-*.dll libmpfr-*.dll libflint*.dll
	do
	lib_name=$(basename -s .dll "${dll_file}")
	def_file=${lib_name}.def
	name=$(echo "${lib_name}" | sed 's/^lib//;s/[-.][0-9].*$//')
	gendef "${dll_file}"
	llvm-lib /def:"${def_file}" /out:"../lib/${name}.lib" /machine:${llvm_machine} /nologo
	rm "${def_file}"
	done

Then MSVC is used to build python-flint itself and the .dlls get bundled.

Actually that is for the x86-64 wheels. The Windows ARM wheels use the mingw clang toolchain then MSVC for python-flint

python-flint/bin/cibw_before_all_windows_arm64.sh

Lines 5 to 8 in 6708168

	pacman -S --noconfirm \
	mingw-w64-clang-aarch64-toolchain \
	mingw-w64-clang-aarch64-llvm-tools \
	mingw-w64-clang-aarch64-tools-git \

GMP/MPFR and FLINT use autotools (although FLINT might change to meson: flintlib/flint#2734) so I'm not sure switching to MSVC or clang-cl helps with that part. We could maybe replace using MSVC with clang-cl just for building the extension modules though I guess.

[oscarbenjamin]

After thinking about it a bit I'm not sure it is a good idea to expose the internal .pxd files directly. Instead there should be a public Cython/C API.

I had GPT put together a quick demonstration of what it might look like to have a C/Cython API in gh-422.

There are two example projects showing how it might look to either link separately against FLINT in the downstream project or just to have python-flint export an API in capsule:
https://github.com/oscarbenjamin/python-flint/tree/fef0070969271d776885d46f6059648a022cdbb7/examples/cython

I'm not sure what makes sense here.

Could a possibility be for python-flint to re-export the whole of FLINT's C API?

[rgommers]

I had a quick peek. The Cython part looks roughly like I'd expect - the one thing I'd add is that Cython has ABI rules that are, eh, interesting, so an ABI stability test would be good to add (see scipy/scipy#24938, I think you could reuse generate_cython_abi_tests.py).

A C API is a fair bit more involved. Although with AI coding tools it should now be feasible to say "do it exactly like numpy" and get something that works.

[oscarbenjamin]

Thanks Ralf.

Yes, I did begin the conversation with GPT by saying "take a look at how ../numpy does it" :)

Okay I see from the scipy PR that Cython ABI compatibility across package boundaries can be flakey. I'll need to look more closely at that to understand what the constraints need to be. I think that the idea here is that minor versions are pinned but I would still need to understand what sort of change could therefore be made in a bugfix release.

I think a C API can be skipped for now.

If we make the assumption that python-flint and downstream library both link against the same (or at least ABI compatible) libflint then the issue is just that the downstream library needs some Cython declarations for the types in python-flint and C-level accessor functions.

I'm thinking that there can be a flint/cython_api.cython-3.1.pxd with something like this for every type and every internal Cython-struct member:

cdef extern class flint.types.fmpz.fmpz [object ..., check_size ignore]:
    pass

cdef const void *fmpz_get_value(fmpz value) except NULL:
    ...

Then the downstream library can work with the Cython fmpz type and has the means to get down to C level access to the member but needs to bring its own C functions to operate on that. We can use accessor functions like fmpz_get_value so that the struct layout is not exposed (although consumers should behave themselves with that pointer access).

A next step after that could be to also expose the entire FLINT C API in Cython via capsule as well. Then downstream library does not need to link against FLINT directly so that part of the ABI problem could go away.