debug: chroot fails for distroless images (eg. on minikube)

# How to reproduce 
1. K8s cluster installed/launched via `minikube start --driver=docker` ✔ 
2. `mint` installed from latest release: 
   ```bash
    $ mint --version                                                                                                         
    mint version linux/amd64|ALP|x.1.42.2|29e62e7836de7b1004607c51c502537ffe1969f0|2025-01-16_07:48:54AM|x
    ```
3. try to attach a sidecar container to the pod `storage-provisioner` in namespace `kube-system` using `busybox:latest`

# Output
```bash
$ mint
>>> debug --runtime k8s --namespace kube-system --pod storage-provisioner --target storage-provisioner --debug-image busybox:latest
cmd=debug state=started

cmd=debug info=cmd.input.params namespace='kube-system' pod='storage-provisioner' runtime='k8s' target='storage-provisioner' terminal='true' gid='-1' uid='-1' fallback-to-target-user='true' debug-image='busybox:latest' entrypoint='[]' cmd='[]' run-as-target-shell='true' 
cmd=debug state=kubernetes.runtime.handler.started
cmd=debug info=wait.for.debug.container namespace='kube-system' name='mint-debugger-142661d818b412e5c564825d59bf81b2f5e94b64' pod='storage-provisioner' 
cmd=debug info=wait.for.container namespace='kube-system' type='ephemeral' id='docker://b5469506fae1ff94e74bbab33460e096222918192879c38ae43b3946a8d2b8f3' state='TERMINATED' exit_code='127' reason='Error' name='mint-debugger-142661d818b412e5c564825d59bf81b2f5e94b64' pod='storage-provisioner' 
time="2025-01-28T11:22:18+01:00" level=error msg=waitForContainer app=mint cmd=debug debug.container.name=mint-debugger-142661d818b412e5c564825d59bf81b2f5e94b64 error="Container terminated" op=debug.HandleKubernetesRuntime sid=142661d818b412e5c564825d59bf81b2f5e94b64
cmd=debug error=debug.container.error message='terminated'
cmd=debug info=container.logs.start                                                                                                                          
cmd=debug log='debug.container.logs' event=LOG.START  ====================
chroot: can't execute 'sh': No such file or directory

cmd=debug log='debug.container.logs' event=LOG.END  ====================
cmd=debug info=container.logs.end
cmd=debug state=debug.container.error code=-1
cmd=debug info=report file='slim.report.json'                                                                                                                
cmd=debug info=exit location='/usr/local/bin' code='-1' version='linux/amd64|ALP|x.1.42.2|29e62e7836de7b1004607c51c502537ffe1969f0|2025-01-16_07:48:54AM|x' 
app='mint' message='GitHub Discussions' info='https://github.com/mintoolkit/mint/discussions'
app='mint' message='Join the CNCF Slack channel to ask questions or to share your feedback' info='https://cloud-native.slack.com/archives/C059QP1RH1S'       
app='mint' message='Join the Discord server to ask questions or to share your feedback' info='https://discord.gg/fAvq4ruKsG'   
```
```bash
$ cat slim.report.json
{
  "document": "doc.report.command",
  "version": "ov/command/debug/1.0",
  "engine": "linux/amd64|ALP|x.1.42.2|29e62e7836de7b1004607c51c502537ffe1969f0|2025-01-16_07:48:54AM|x",
  "containerized": false,
  "host_distro": {
    "name": "Ubuntu",
    "version": "22.04",
    "display_name": "Ubuntu 22.04.5 LTS"
  },
  "type": "debug",
  "state": "error"
}
```

# Further info
I tried to manually repeat the steps as described [here](https://static.sched.com/hosted_files/kccnceu2024/05/KubeConEU24%20-%20Building%20a%20Tool%20to%20Debug%20Minimal%20Container%20Images.pdf), i.e.
1. run `kubectl debug -it -n kube-system storage-provisioner -t storage-provisioner --image busybox --profile="sysadmin"`
2. run `ln -s /proc/$$/root /proc/1/.sidecar` -> first failure, because `/proc/1` is read-only; `ln -s /proc/$$/root  /proc/1/root/.sidecar` worked
3. run `chroot /proc/1/root` -> fails with `/bin/sh: no such file or directory` (`chroot /proc/1/root .sidecar/bin/sh` failed also)
4. check content of `/proc/1/root/`:
    ```bash 
    / # ls -l /proc/1/root/
    total 30736
    drwxr-xr-x    5 root     root           360 Jan 28 10:22 dev
    drwxr-xr-x    2 root     root          4096 Jan 28 10:22 etc
    dr-xr-xr-x  505 root     root             0 Jan 28 10:22 proc
    -rwxr-xr-x    1 root     root      31465472 Mar 29  2021 storage-provisioner
    dr-xr-xr-x   13 root     root             0 Jan 28 10:21 sys
    drwxrwxrwt    5 root     root           140 Jan 28 10:29 tmp
    drwxr-xr-x    3 root     root          4096 Jan 28 10:22 var
    ```
4. create symlinks for each missing directory directly:
    ```bash
    # ln -s /proc/$$/root/bin /proc/1/root/bin
    # ln -s /proc/$$/root/usr /proc/1/root/usr
   ```
5. now `chroot /proc/1/root` works fine ✔ 

I do not know whether the root cause is the `read-only` file system or just the missing `/bin`, `/usr` directories themselves.
However, I think the workaround to iteratively link missing directories into the target fs could be implemented as a fallback.

# Update
I found another way to directly `chroot` into the main process' root filesystem:
```bash
$ ln -s /proc/$$/root /proc/1/root/.sidecar
$ export PATH=$PATH:/.sidecar/bin:/.sidecar/usr/bin
$ chroot /proc/1/root /.sidecar/lib/ld-linux-x86-64.so.2 --library-path /.sidecar/lib /.sidecar/bin/sh
```

But now a simple `ls` command inside the new root must be executed as follows:
```bash
/.sidecar/lib/ld-linux-x86-64.so.2 --library-path /.sidecar/lib /.sidecar/bin/ls
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

debug: chroot fails for distroless images (eg. on minikube) #100

How to reproduce

Output

Further info

Update

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

debug: chroot fails for distroless images (eg. on minikube) #100

Description

How to reproduce

Output

Further info

Update

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions