Repository navigation
fs::remove_dir_all rarely succeeds for large directories on windows #29497
Description
I've been trying to track this one down for a while with little success. So far I've been able to confirm that no external programs are holding long-standing locks on any of the files in the directory, and I've confirmed that it happens with trivial rust programs such as the following:
use std::fs;
fn main() {
println!("{:?}", fs::remove_dir_all("<path>"));
}I've also confirmed that deleting the folder from explorer (equivalent to using SHFileOperation), or using rmdir on the command line will always succeed.
Currently my best guess is that either windows or other programs are holding transient locks on some of the files in the directory, possibly as part of the directory indexing windows performs to enable efficient searching.
Several factors led me to think this:
fs::remove_dir_allwill pretty much always succeed on the second or third invocations.- it seems to happen most commonly when dealing with large numbers of text files
- unlocker and other tools show no active handles to any files in the directory
Maybe there should be some sort of automated retry system, such that temporary locks do not hinder progress?
edit:
The errors returned seem somewhat random: sometimes it gives a "directory is not empty" error, while sometimes it will show up as an "access denied" error.
Activity
retep998 commentedon Nov 1, 2015
I think this is just race conditions at work. When you delete a file another program might get a notification and check out what happened and that might hold a temporary lock on something. Or it might be that the previous deletions weren't actually done by the time you tried to delete the directory.
petrochenkov commentedon Nov 1, 2015
Hm, https://github.com/CppCon/CppCon2015/blob/master/Tutorials/Racing%20the%20Filesystem/Racing%20the%20Filesystem%20-%20Niall%20Douglas%20-%20CppCon%202015.pdf slide "3. Deleting a directory tree" may be related. I accidentally watched this presentation yesterday. If I understood correctly,
DeleteFileWused inremove_dir_alldoesn't actually delete a file, but only marks it for deletion, it still can be alive when we try to delete the parent directory causing the "directory is not empty" error.Diggsey commentedon Nov 1, 2015
@petrochenkov Great link - that seems like it's exactly the problem I'm experiencing.
It would be great to see some of these portable race-free idioms implemented in std.
alexcrichton commentedon Nov 2, 2015
I'd be interested to dig a bit more into this to see what's going on. According to the
DeleteFiledocumentation the only reason the file would be flagged for deletion would be if there are active open handles, which seems like a legitimate race condition? Note though that the same docs also recommend usingSHFileOperationfor recursively deleting a tree.Those slides are certainly an interesting read though! I'd be a little wary of putting it into the standard library as the "transactional semantics" are a little weird. For example if you successfully rename a file to a temporary directory and then fail to delete it, what happens?
It may be the case that the best implementation for this function on Windows is to just use SHFileOperation, and that may end up just taking care of these sorts of issues on its own.
pitdicker commentedon Feb 19, 2016
I will give fixing this a try.
On Windows XP we could use
SHFileOperation, and on Vista and upIFileOperation. For Windows store applications there is no easy function. But I would still like to try implementing it by walking the directories and deleting recursively.Current problems with
remove_dir_all:And what I hope will fix it:
canonicalise()on the base dir to get support for long pathsFILE_FLAG_OPEN_REPARSE_POINT (for opening reparse points)
FILE_FLAG_DELETE_ON_CLOSE
(normal rename on xp)
(not to %TEMP%, may not be on the same drive)
rename them to rm-[hash]-[nr]
I think we know deleting will fail when opening the file with
FILE_FLAG_DELETE_ON_CLOSEfails, so before moving.pitdicker commentedon Feb 19, 2016
O, and I walked against this bug when rustbuild (is that the current name) failed after make clean because of this.
alexcrichton commentedon Feb 19, 2016
@pitdicker thanks for taking this on! We may want to discuss this a bit before moving much more on a solution. One question is how principled should something like
fs::remove_dir_allbe. For example iffs::remove_dir_allautomatically handled read-only files, shouldfs::remove_filealso do the same? Currently we have this "duality" where somefsfunctions are straight counterparts to the underlying system operations, but some are somewhat fancier likecreate_dir_allandremove_dir_all. We'd just want to make sure to have an explicitly drawn line here. Also note I'm not really thinking of an RFC here, just some more discussion before jumping to a PR.I personally feel that
remove_dir_allis fine to be fancy and do lots of implicit operations. I would continue to want, however, thatremove_fileandremove_dirare as straightforward as possible (e.g. don't mirror what's happening here unless it's a similarly small set of operations as to what happens today). Your proposed strategy seems reasonable to me at first glance, and I wouldn't mind reviewing more over a PR.Curious what others think, though? cc @rust-lang/libs
Diggsey commentedon Feb 19, 2016
@alexcrichton Hmm, I would prefer
remove_fileandremove_dirto try to give the same guarantees they have on linux if possible. Specifically that if they succeed, the file or directory should be completely gone.It's inevitable that people will build their own abstractions over the top of these primitive operations, and I'd very much like to avoid these kind of bugs where it's sufficiently rare that it's almost impossible to track down the true cause of the problem.
retep998 commentedon Feb 19, 2016
This wouldn't really work because files don't necessarily get deleted right away. You can't just move the file somewhere else and then delete it because there isn't always a reasonable place on the same volume to move the file to. Sitting around and indefinitely waiting for the file to vanish isn't ideal either. I'd personally just use
IFileOperationforremove_dir_alland call it a day.Diggsey commentedon Feb 19, 2016
What if the file is opened with
FILE_FLAG_DELETE_ON_CLOSE, no share flags specified, and then is immediately closed? Successfully opening the file means that you have exclusive access, and so closing it should delete the file.retep998 commentedon Feb 19, 2016
No share flags means nobody else can open it to read/write/delete it. However another handle can still be opened without read/write/delete permissions. Also what happens if you fail to open the file with no share flags? Does it error or fallback to the normal deletion method that provides no guarantees?
pitdicker commentedon Feb 19, 2016
Great to see so many comments. Just what I hoped for!
@alexcrichton I completely agree to keep
remove_fileandremove_dirsimple, and only letremove_dir_alldo magic. Actually removing read-only files is pretty difficult when you also don't want to break hard links. http://stackoverflow.com/questions/3055668/delete-link-to-file-without-clearing-readonly-bit is an example, but I am sure I found some better way somewhere.@Diggsey I would also like cross-platform consistency, but sometimes Windows is just to different... I think the standard library should at least expose primitives that only need one system call. Otherwise we lose performance and maybe get more vulnerable to race conditions. Maybe we can add a
remove_no_matter_whatin the future :)@retep998 I have not yet written the code, so I don't know if this will really solve the problem... But if this doesn't work out I am all for
IFileOperation. Meanwhile it is a nice way for me and maybe the standard library to get better at avoiding filesystem races.I would have to test what happens with
FILE_FLAG_DELETE_ON_CLOSE, but in theory if the file opens successfully we only have to move it to avoid a race of a couple of milliseconds. Directly after opening the file no one else can open the file anymore (it is not a share flag).What is difficult is where to move a file / dir to. I think the parent of the dir we are removing is good enough. It should be on the same volume, and I think we also have write permission to it because otherwise deleting a dir is not possible (wild speculation here, it is late :))
brson commentedon Feb 20, 2016
I feel ok about going to extra effort to make
remove_dir_allwork as expected.alexcrichton commentedon Feb 22, 2016
@pitdicker perhaps you could draw up a list of the possible implementation strategies for these functions? That'd also help in terms of weighing the pros/cons of each strategy and we could get a good idea of which terminology is associated with what. So far it sounds like
IFileOperationmay be the best option here, but I'm personally at least not following whatFILE_FLAG_DELETE_ON_CLOSEwould mean, especially in the case that you can't open it as someone else does.25 remaining items
rbtcollins commentedon Mar 30, 2022
@ChrisDenton Looks like
FILE_DISPOSITION_IGNORE_READONLY_ATTRIBUTEis only available for ZwSetInformationFile - Cygwin uses the underlying NT calls, but I had the impression rust is trying to avoid getting that low level.@pwinckles The loop there may reduce the incidence, but as it is reentering into the entire thing its going to repeat IO for the depth of the point where the error occurred, though it will at least bail out fast. I wouldn't want to use that version on rustups deletion of docs trees, for instance; though I am still interested in getting a reliable version into the core.
ChrisDenton commentedon Mar 30, 2022
@rbtcollins
FILE_DISPOSITION_IGNORE_READONLY_ATTRIBUTEis in the standard headers (WinBase.h) and is also in the windows-rs crate.Btw, the code currently in the standard library was written with some amount of haste so isn't necessarily the best example. I do plan to get back to it but lately I've been distracted with other issues.
rbtcollins commentedon Mar 30, 2022
@ChrisDenton thats super interesting - and not visible on the MSDN doc pages :/
I meant no negativity about the current code - it is great the CVE was solved, and there time mattered a lot... and this bug is really (to my mind) about figuring out what is sensible for the stdlib - tradeoffs between complexity, generality, least surprise etc abound.
ChrisDenton commentedon Mar 31, 2022
The newer information classes are listed (e.g.
FileDispositionInfoExandFileRenameInfoEx) but there are not yet platform version information or dedicated pages listing the constants as there are with the others. It probably just needs someone to put in the work of writing the docs (which tbh is rarely the most popular task). In any case being in public headers and used by MSVC STL is quite a vote of confidence that it isn't going away.Sorry! I didn't take what you said as negativity. I was meaning that more as a warning that, while I'm happy it fixes the CVE, I would still like to see the code cleaned up and improved.
rbtcollins commentedon Jan 20, 2023
@ChrisDenton so FileDispositionInfoEx notes that deleting a file with posix semantics - " When FILE_DISPOSITION_POSIX_SEMANTICS is set, the link is removed from the visible namespace as soon as the POSIX delete handle has been closed, but the file's data streams remain accessible by" - does this mean that the directory delete will recieve ERROR_DIR_NOT_EMPTY while those processes with handles open continue to consume the file? Or is it genuinely unlinked from the directory and thus the directory is deletable?
ChrisDenton commentedon Jan 20, 2023
@rbtcollins The
HANDLEthat sets the disposition must be closed for it to take full effect. On settingFILE_DISPOSITION_POSIX_SEMANTICS, but before closing, the file is locked such that the file can no longer be opened. However, it still remains in the directory and so attempting to delete the directory will produceERROR_DIR_NOT_EMPTY. So essentially setting the disposition should be immediately followed by a close.Other file handles to the same file can remain open. They have no effect at all on the POSIX delete (unless of course they lock the file).
rbtcollins commentedon Jan 21, 2023
Thank you @ChrisDenton .
I'm having a little trouble figuring out the version support for this comment of yours:
FILE_DISPOSITION_IGNORE_READONLY_ATTRIBUTE flag (on OS versions that support it).It looks like its definitely present in Windows 10.1809. Is there some easy lookup matrix for these things? In particular to figure out whether to use it conditionally, or just document that successful deletion of readonly trees would require that version ...ChrisDenton commentedon Jan 21, 2023
The documentation is pretty limited atm so the best bet is the C/C++ headers which indeed indicate that it's available on 1809 and later (aka
_WIN32_WINNT_WIN10_RS5).I'd suggest trying the
FILE_DISPOSITION_IGNORE_READONLY_ATTRIBUTEflag first, checking the error returned, and then progressively falling back if necessary. The std currently checks for any ofERROR_NOT_SUPPORTED,ERROR_INVALID_FUNCTIONorERROR_INVALID_PARAMETERand then falls back to non-posix delete. A more involved solution could fallback to droppingFILE_DISPOSITION_IGNORE_READONLY_ATTRIBUTEfirst and if that works then remembering (in an atomic) that the ignore readonly attribute isn't supported on this OS.Diggsey commentedon Jun 1, 2025
The implementation today is much more robust than when I opened this issue - it now has retry logic at the file/folder level, and is a more sophisticated approach in general. I attempted to recreate the issue and didn't have any success.
While I'm sure the issue can still occur, I believe that the remaining issues are caused by real locks on the files rather than simply a race condition in the deletion logic, and that the current Rust implementation is at least as good as the equivalent functionality in other languages.
For that reason I think it makes sense to consider this solved.
fix(mod-install): retry mod folder removal on Windows os error 145