Podman regression in v0.25.1

[fnetX]

Description

I know that podman is not officially supported, but it currently seems to work fine. However, the latest release seems to contain a regression.

Instructions to reproduce the bug

1. Run docuum on rootless podman with $ podman run --init --rm --tty --name docuum --mount type=bind,src=/run/user/1101/podman/podman.sock,dst=/var/run/docker.sock --mount type=bind,source=/home/runner/docuum,target=/root stephanmisc/docuum --threshold '10 GB' --keep 'docker.io/stephanmisc/docuum'
2. get the following log output:

[2025-11-18 18:31:59 +00:00 INFO] Performing an initial vacuum on startup…                                                                                                                                                                    
Error response from daemon: unknown container state: restarting: invalid argument                                      
[2025-11-18 18:32:00 +00:00 ERROR] Unable to list containers.                                                          
[2025-11-18 18:32:00 +00:00 INFO] Retrying in 5 seconds… 
...
[2025-11-18 18:34:28 +00:00 INFO] Performing an initial vacuum on startup…
Error response from daemon: unknown container state: dead: invalid argument
[2025-11-18 18:34:29 +00:00 ERROR] Unable to list containers.              
[2025-11-18 18:34:29 +00:00 INFO] Retrying in 5 seconds…                                                               
...

3. Try with $ podman run --init --rm --tty --name docuum --mount type=bind,src=/run/user/1101/podman/podman.sock,dst=/var/run/docker.sock --mount type=bind,source=/home/runner/docuum,target=/root stephanmisc/docuum:0.25.0 --threshold '10 GB' --keep 'docker.io/stephanmisc/docuum'
4. observe that it works

It would be really great if you could look into this to get podman working again. Thank you very much!

[stepchowfun]

Ah, looks like that was from #341, intended to fix #237 (workaround for a Docker bug). We could revert that PR, but it would bring back the original Docker issue.

If Podman aims to be compatible with Docker, I wonder if this could be fixed upstream there. Or maybe we could fix the original Docker issue that led to that PR.

[fnetX]

It seems like "restarting" is not a valid podman state. I can try to ask upstream, but I wouldn't know if it would be better to simply ignore this or explicitly error.

[fnetX]

I am happy to ask podman to improve compatibility here. But it really feels like an xy-problem. If the problem is that docker containers can get stuck and you exclude one state, I feel like technically there is no guarantee that there won't be changes in states or other stuck containers where inspect would fail that suffer from the same issue. My naive suggestion to fix the problem is ignoring the "container doesn't exist" error from docker in case inspection fails.

Please let me know if I should proceed with discussing with podman anyway.

[fnetX]

Another potential hotfix on the docuum side could be to not filter by status=restarting as I would assume this container state is not super interesting and should only be temporary. But I don't know much about how docuum works and if this is safe enough.

[stepchowfun]

My naive suggestion to fix the problem is ignoring the "container doesn't exist" error from docker in case inspection fails.

I agree with this! The only complication is that Docuum queries Docker about a whole batch of containers at once rather than querying them individually. So if the whole batch fails due to one container, we'd need to add some retry logic that splits up the problematic batch or attempts each container individually. It seems like a relatively small amount of work—I'd be happy to review a PR for it.

Another potential hotfix on the docuum side could be to not filter by status=restarting as I would assume this container state is not super interesting and should only be temporary.

From your logs it seems like dead is also an invalid status for Podman:

Error response from daemon: unknown container state: dead: invalid argument

So we'd need to account for all of these (is Docker's problematic removing status even valid in Podman?).

[stepchowfun]

Additional commentary on previous message:

add some retry logic that splits up the problematic batch or attempts each container individually.

Ideally we'd do the former (e.g., splitting the failing batch into halves). Querying each container individually would be a bummer for performance (the batching optimization was important).

[ntninja]

I was able to current docuum on NixOS with Podman like this:

{ config, pkgs, lib, utils, ... }:

{
	systemd.services.docuum = {
		after = [ "podman.socket" ];
		requires = [ "podman.socket" ];
		wantedBy = [ "multi-user.target" ];
		path = [
			# Workaround script for https://github.com/stepchowfun/docuum/issues/346
			(pkgs.writeShellScriptBin "docker" ''
				echo "Called:" "docker" "$@" >&2
				
				if [ $# -gt 2 ] && [ "$1" = "container" ] && [ "$2" = "ls" ];
				then
					# Filter unsupported “--filter status=dead” and
					# “--filter status=restarting” argument groups
					prev_arg_filter=false
					for arg in "$@";
					do
						shift 1
						if $prev_arg_filter;
						then
							if [ "$arg" != "status=dead" ] \
							&& [ "$arg" != "status=restarting" ];
							then
								# Was not “status=XXX”, re-add both “--filter”
								# and current arg
								set -- "$@" "--filter" "$arg"
							fi

							# If arg was “status=XXX”, neither it nor the
							# preceeding “--filter” is added back to the
							# argument list
							prev_arg_filter=false
						elif [ "$arg" = "--filter" ];
						then
							# Temporarily skip “--filter” in case it is
							# followed by “status=XXX”
							prev_arg_filter=true
						else
							# Other arguments are just readded to the argument
							# list as-is
							set -- "$@" "$arg"
						fi
					done
				fi
				
				echo "Running:" "podman" "$@" >&2
				exec "${lib.getExe config.virtualisation.podman.package}" "$@"
			'')

			# Direct symlink variant (cannot rewrite arguments):
#			(pkgs.linkFarm "podman-as-docker" [{
#				name = "bin/docker";
#				path = lib.getExe config.virtualisation.podman.package;
#			}])
		];
		
		# Make Podman connect to system instance proxy, rather than managing
		# local containers of the “docuum” user
		environment.CONTAINER_HOST="unix:///run/podman/podman.sock";
		
		environment.HOME = "/var/lib/docuum";

		serviceConfig = {
			DynamicUser = true;
			StateDirectory = "docuum";
			SupplementaryGroups = [ "podman" ];
			ExecStart = utils.escapeSystemdExecArgs ([
				(lib.getExe pkgs.docuum)
				"--threshold" "0"  # Always try to evict
				"--deletion-chunk-size" "1"  # Default
				"--min-age" "14d"  # Image retain time
			]);
		};
	};
}

It is based on the default NixOS configuration, but with a wrapper shell script for docker → podman injected into its $PATH that drops the unsupported arguments.