What about other network-backed FS? SMB? sshfs? HDFS?

Another option (possibly as a follow-up) would be to reverse the list; and only allow running on backing-filesystems that we know to work; other filesystems will error-out, but we can add a --storage-opt overlay.allow-unsupported-filesystems (e.g.) that allows people to override that restriction (in which case we should both log a big-red-warning, and print a warning on docker info as well)

What about other network-backed FS? SMB? sshfs? HDFS?

SSHFS and HDFS are fuse based, we can't really detect those specifically. We would have to switch to a whitelist approach and imo that is too opinionated. NFS is a common setup so users can easily walk into it and expect it to work, not sure anyone expects the same over FUSE or SMB.

SMB is only FUSE on Linux hosts -- on Windows hosts it is about as native as it gets.

Oh, but on Windows we don't support the overlay drivers, only the "Windows" storage driver

As far as I understand recently NFS become suitable for overlayfs https://lwn.net/Articles/736677/ but it requires very fresh kernel and corresponding NFS options. So I suggest to check a kernel version or/and NFS options.

Ah, thanks for that information; I'd not be against doing that check. If possible through feature detection (instead of just comparing kernel versions); perhaps it should be in a follow up.

ping @amir73il any suggestions on how to detect if overlay is supported on NFS?

There is a lot of confusion and inaccurate statements in this thread. Quite understandable with the current state of overlayfs documentation. I'll try to add clarity.

@kazh000 the mentioned article has nothing to do with supporting an NFS mount as upper layer for overlayfs. It is WIP, not anywhere near mainline, to support exporting an overlayfs mount by NFS server.

@MistyHacks the reason that network backed file systems and FUSE are not supported as upper layer is because those file systems can invalidate VFS cache entries (when files are changed remotely) and overlayfs doesn't like things to change under its feet. Incidently, overlayfs doesn't like it when remote file systems change files remotely even when those are used as lower layer, but lower layer files are not supposed to be change at all, while overlay is moutned.

@thaJeztah the text you quoted in the bug from overlayfs.txt has mislead you. Overlayfs will throw a warning to dmesg complaining if d_type is not supported ("overlayfs: upper fs needs to support d_type.").
However, Overlayfs will NOT fail the mount on no d_type support, nor on no xattr support, but it will perform sub optimally (exposing whiteouts and returning EIO in some cases). So w.r.t feature detection, you SHOULD to check d_type support and xattr support before using overlayfs driver, but this is not the reason that overlayfs over NFS is failing to mount.
You do not really need "feature detection" for this case, you just need to try to mount overlay and see that it fails. Anyway, I don't know of an API to check if a file system is a "remote" file system, but FYI, here is the list of such file systems in the kernel, found by 'git grep -lw d_revalidate fs/':
9p
afs
ceph
cifs
coda
ecryptfs
fat
fuse
gfs2
hfs
jfs
kernfs
ncpfs
nfs
ocfs2
orangefs
overlayfs
proc

Thanks for your input @amir73il - detection of d_type support is already in our code base (as running without leads to "interesting side effects"); I'll check/discuss the other options.

We discussed this change, and for now just disallow overlay on NFS, but contributions are welcome if we can implement this using actual detection if its supported

LGTM