Note: This post will be updated as I run into issues with using podman.
First issue:
podman run --rm -it nvidia/cuda:11.1-devel-ubuntu20.04 bash
ERRO[0000] cannot find UID/GID for user USERNAME: No subuid ranges found for user "USERNAME" in /etc/subuid - check rootless mode in man pages.
Trying to pull registry.opensuse.org/nvidia/cuda:11.1-devel-ubuntu20.04...
name unknown
Trying to pull docker.io/nvidia/cuda:11.1-devel-ubuntu20.04...
Getting image source signatures
Copying blob a254829d9e55 done
Copying blob 29cfce72a460 done
Copying blob 4bb689f629d3 done
Copying blob ba13d3bc422b done
Copying blob 6a5697faee43 done
Copying blob f853e5702a31 done
Copying blob 302d6ee79e94 done
Copying blob fe1233ff1aac done
Copying config 539690cdfc done
Writing manifest to image destination
Storing signatures
Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
Error: unable to pull nvidia/cuda:11.1-devel-ubuntu20.04: 2 errors occurred:
* Error initializing source docker://registry.opensuse.org/nvidia/cuda:11.1-devel-ubuntu20.04: Error reading manifest 11.1-devel-ubuntu20.04 in registry.opensuse.org/nvidia/cuda: name unknown
* Error committing the finished image: error adding layer with blob "sha256:6a5697faee43339ef8e33e3839060252392ad99325a48f7c9d7e93c22db4d4cf": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
Subsequent pulls will have errors too:
podman image pull busybox
Trying to pull registry.opensuse.org/busybox...
name unknown
Trying to pull docker.io/library/busybox...
Getting image source signatures
Copying blob 9758c28807f2 done
Copying config f0b02e9d09 done
Writing manifest to image destination
Storing signatures
Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument
Error: unable to pull busybox: 2 errors occurred:
* Error initializing source docker://registry.opensuse.org/busybox:latest: Error reading manifest latest in registry.opensuse.org/busybox: name unknown
* Error committing the finished image: error adding layer with blob "sha256:9758c28807f21c13d05c704821fdd56c0b9574912f9b916c65e1df3e6b8bc572": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument
The fix:
First, remove some files that cause the error to keep happening rm -rf ~/.{config,local/share}/containers /run/user/$(id -u)/{libpod,runc,vfs-*} (h/t https://github.com/containers/podman/issues/1182#issuecomment-574888124
Then (and this is documented in the podman man pages)
sudo usermod --add-subuids 10000-75535 USERNAME
sudo usermod --add-subgids 10000-75535 USERNAME
And now things should be working!