Skip to content

vfs: add materializeSync() for FFI consumers - #65909

Draft
mcollina wants to merge 2 commits into
nodejs:mainfrom
mcollina:vfs-ffi-materialize
Draft

vfs: add materializeSync() for FFI consumers#65909
mcollina wants to merge 2 commits into
nodejs:mainfrom
mcollina:vfs-ffi-materialize

Conversation

@mcollina

@mcollina mcollina commented Sep 8, 2026

Copy link
Copy Markdown
Member

Native addons inside a mounted VFS already load through require(): the loader hands their bytes to an internal helper that writes them to a private image the dynamic loader can open. Libraries that open a binary by path themselves — FFI bindings that dlopen() a shared library, for example — never reach the module loader, so that route cannot help them.

This adds vfs.materializeSync(path), exposing the same mechanism as a public API:

const vfs = require('node:vfs');
const path = require('node:path');

// Paths outside a VFS pass through unchanged, so this is safe to
// call unconditionally.
const libPath = vfs.materializeSync(path.join(__dirname, 'libsqlite3.so'));
const lib = koffi.load(libPath);
  • A path outside any VFS is returned unchanged: it is already openable.
  • A VFS path is read and its bytes are written to a private image — an anonymous in-memory memfd on Linux (nothing touches the file system), a temporary file elsewhere — that stays loadable for the rest of the process lifetime, so the returned path can be handed to any consumer that opens it by path.
  • Repeated calls with the same path return the same image. Unmounting drops the path from the cache while leaving already materialized images alive, since a consumer may still be about to open them. Retained temp files are removed at normal process exit (Windows uses the existing retained delete-on-close handles).

The C++ side reuses AddonImage (the materializer behind dlopenBinary) with a new Retain() consumption mode, exposed as materializeBinary() on the process_methods binding, with the same temp-dir write-permission check and a new ERR_VFS_MATERIALIZE_FAILED error code.

The new test simulates an FFI consumer by calling process.dlopen() on the materialized path directly — the path lies outside the VFS, so the call takes the unhooked route straight to the OS loader, exactly like an FFI library would.

Bug fix included

Writing that test exposed a pre-existing bug, fixed in the first commit: the dlopen hook installed while a VFS is mounted always forwarded its flags parameter, so a two-argument process.dlopen() call for a real file-system path reached the original implementation with undefined as the flags. That coerces to 0, which is not a valid dlopen(2) mode, and loading any addon from the real file system failed with EINVAL while a VFS was mounted.

The dlopen hook installed while a VFS is mounted always forwarded its
flags parameter, so a two-argument process.dlopen() call for a real
file system path reached the original implementation with `undefined`
as the flags. That coerces to 0, which is not a valid dlopen(2) mode,
instead of applying the default flags, and loading any addon from the
real file system failed with EINVAL while a VFS was mounted.

Signed-off-by: Matteo Collina <hello@matteocollina.com>
Native addons inside a mounted VFS already load through require():
the loader hands their bytes to an internal helper that writes them
to a private image the dynamic loader can open. Libraries that open
a binary by path themselves - FFI bindings that dlopen() a shared
library, for example - never reach the module loader, so that route
cannot help them.

Expose the same mechanism as vfs.materializeSync(path): a path
outside any VFS is returned unchanged, while a VFS path is read and
its bytes are written to a private image - an anonymous in-memory
memfd on Linux, a temporary file elsewhere - that stays loadable for
the rest of the process lifetime. The returned real path can be
handed to any consumer that opens it by path. Repeated calls with
the same path return the same image, and unmounting drops the path
from the cache while leaving already materialized images alive.

Signed-off-by: Matteo Collina <hello@matteocollina.com>
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs involving general changes in the lib/ or src/ directories. needs-ci PRs that need a full CI run. labels Sep 8, 2026
@pipobscure

Copy link
Copy Markdown
Contributor

LETM (Looks Excellent To Me 😃 )

@pipobscure

pipobscure commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

The only question I have is whether we want to hide the detail that the path may need materializing inside ffi.load like we did for process.dlopen. There are pro & cons for both. And maybe someone that is using FFI is the kind of person good with handling this manually. However this is also the kind of thing that's easy to forget to do (especially if you don't think of VFS as an option). So hiding it inside ffi.load would eliminate this fault category.

And since node:ffi has both dlopen and dlclose we even have a good place to put it and do reference counting. Making a library Symbol.disposable can call dlclose which would decrement the counter on the path and on the last close we can delete the file. And then just cleanup leftovers atexit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs involving general changes in the lib/ or src/ directories. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants