NFS filehandle recovery loses the INLINED flag, causing duplicate FileInode objects and a use-after-free in CloseFile

[dunci]

NFS filehandle recovery loses the INLINED flag, causing duplicate FileInode objects and a use-after-free in CloseFile

Relation to issue #122

This issue is a follow-up and correction to the production-crash attribution in #122.

Issue #122 remains a valid, independently reproducible bug: reinlining a live inode can clear its
StripePattern*, and the later close path dereferences a NULL pointer.

However, a deeper analysis of the original production core dump shows that the production crash
had a different cause. In that core, StripePattern* was non-NULL but pointed into an already
freed FileInode. The similar top-level stack initially made the two failures appear identical.

This issue describes the actual production failure: an NFS filehandle recovery bug creates two
different FileInode objects for the same EntryID, after which the wrong instance is released.

Describe the bug

The BeeGFS-internal NFS filehandle format V3 stores EntryID, parent EntryID, owner, entry type and
buddy-mirrored state, but does not store ENTRYINFO_FEATURE_INLINED.

When fh_to_dentry() cannot find the corresponding BeeGFS VFS inode through ilookup5(), it
reconstructs an EntryInfo from the filehandle using featureFlags=0. For a buddy-mirrored file
this produces flags BUDDYMIRRORED only, with INLINED missing.

A later open/reference request therefore routes an actually inlined file into the global inode
store. If the normal per-directory instance already exists, meta then holds two distinct
FileInode objects for the same EntryID.

Close and release operations search the global store first and match only by EntryID, without
verifying the supplied object address. Closing the per-directory instance can therefore decrement
and delete the global instance. A SessionFile that still owns the global instance is left with a
dangling pointer, resulting in a use-after-free during a later close.

Impact: beegfs-meta SIGSEGV and metadata-service interruption.

Describe the system

• BeeGFS 7.4.7, commit d197911a18698cae81411953d0a3eaa024a68ce7
• The current master branch still appears to contain the same filehandle recovery code
• Linux 5.15 NFS gateway
• A BeeGFS client mount exported through the Linux kernel NFS server
• Buddy-mirrored metadata
• Regular file with an inlined inode

Required state transition

1. A normal pathname lookup creates or retains the inlined instance in the parent directory's file
   store.
2. The NFS server generates a filehandle for the same file.
3. The corresponding BeeGFS VFS inode is later evicted on the NFS gateway, while the remote NFS
   client retains its filehandle.
4. A later request using that filehandle causes ilookup5() to return NULL.
5. BeeGFS reconstructs EntryInfo with featureFlags=0; buddy handling only adds
   BUDDYMIRRORED.
6. The initial stat fallback may temporarily load the inode from the inlined dentry, but it does
   not persistently insert it into the global store.
7. A subsequent open/reference with the incorrect EntryInfo creates a second instance in the
   global store.
8. Closing the per-directory instance finds the global entry by EntryID and releases the wrong
   object.
9. The NFS SessionFile later closes using the freed global instance and crashes.

The ordering matters: the per-directory instance must already exist before the incorrect NFS open
creates the global instance. No concurrent execution is required.

Source evidence

• FhgfsNfsFileHandleV3 stores isBuddyMirrored but not isInlined:
  client_module/source/filesystem/FhgfsOpsExport.h:98
• Encoding saves buddy-mirrored state but no inlined state:
  client_module/source/filesystem/FhgfsOpsExport.c:171
• ilookup5() is used for local BeeGFS VFS inode recovery:
  client_module/source/filesystem/FhgfsOpsExport.c:412
• On a cache miss, EntryInfo_init() is called with featureFlags=0:
  client_module/source/filesystem/FhgfsOpsExport.c:456
• EntryInfo_init() only derives BUDDYMIRRORED from a group owner:
  client_module/source/common/storage/EntryInfo.h:95
• isInlined=false causes MetaStore::openFile() to load into the global store:
  meta/source/storage/MetaStore.cpp:510
• Loading a non-inlined inode falls back to the inlined dentry:
  meta/source/storage/FileInode.cpp:808
• InodeFileStore::closeFile() and releaseFileInode() look up only by EntryID:
  meta/source/storage/InodeFileStore.cpp:233
meta/source/storage/InodeFileStore.cpp:259
• MetaStore::closeFile() and releaseFileUnlocked() search the global store first:
  meta/source/storage/MetaStore.cpp:586
meta/source/storage/MetaStore.cpp:300

Core dump evidence

The production crash stack was:

StatData::updateDynamicFileAttribs(...)
MsgHelperClose::closeFile(...)
CloseFileMsgEx::closeFilePrimary(...)
MirroredMessage<CloseFileMsg, FileIDLock>::processIncoming(...)
IncomingPreprocessedMsgWork::process(...)
Worker::workLoop(...)

The original fault occurred at the virtual call corresponding to
stripePattern->getNumStripeTargetIDs():

mov  (%rdx), %rax
call *0x30(%rax)

Unlike #122, rdx was not zero. It contained a heap address, but the supposed vtable pointer had
been overwritten with heap data and the indirect call resolved to the unmapped address
0x03e5427c. This is consistent with a freed and reused StripePattern/FileInode, not a NULL
pointer.

The core also contained two distinct object histories for the same EntryID:

• The normal instance had a real filename, EntryInfo flags INLINED | BUDDYMIRRORED, a
  parent-directory handle and FileInode::isInlined=true.
• The NFS-recovered instance used the synthetic filename <nfs_fh>, request flags
  BUDDYMIRRORED only and a global-store handle.
• The NFS instance's allocation already contained glibc tcache/free-list metadata when it was used
  by the close path.
• Its saved StripePattern* was non-NULL, but its virtual-function table was invalid.

The _Unwind_Backtrace frames visible above the fault were entered by the BeeGFS signal handler
while reporting the SIGSEGV; they were not the original failure.

The <nfs_fh> filename is assigned only in the if (!inode) branch after ilookup5(), confirming
that NFS filehandle recovery followed the cache-miss path.

Expected behavior

• NFS filehandle recovery must preserve or authoritatively recover all routing-relevant
  EntryInfo flags.
• One EntryID must not produce independent global and per-directory FileInode objects.
• Close/release must only decrement the exact object that was referenced or opened.
• An ambiguous legacy filehandle must return ESTALE rather than silently assuming
  INLINED=false.

Suggested fixes

1. Introduce a versioned NFS filehandle format that stores featureFlags, including
   ENTRYINFO_FEATURE_INLINED.
2. For legacy V1/V2/V3 handles, recover authoritative flags from meta on an inode-cache miss, or
   return ESTALE.
3. In InodeFileStore::closeFile() and releaseFileInode(), verify that the map entry's
   FileInode* equals the supplied pointer before changing reference counts.
4. Track the originating store in MetaFileHandle and return the handle directly to that store
   instead of searching global-first by EntryID.
5. Before loading from disk, check both loaded stores and prevent insertion of a second object for
   an EntryID already present elsewhere.

Additional context

The missing INLINED flag does not cause ilookup5() to miss: ilookup5() compares the
superblock, inode hash and EntryID, not the inlined state. The cache miss happens first; the
incomplete filehandle becomes dangerous only when BeeGFS tries to reconstruct EntryInfo
afterward.

Please let me know if maintainers need the full core analysis or additional diagnostic data. The
core itself contains private filenames, EntryIDs and session information, so it is not attached
publicly.