Sideloaded image fails to unpack in a deployment or a run

[halfer]

I may have exhausted my options asking for help on the web, so I am converting my issue to a bug report. Interestingly the core error message is hardly evident in web searches, which seems odd to me, since I think I am doing something very trivial.

I am new to K8S and MicroK8S. I have selected MicroK8S as I believed it would simplify some basics (and, in the case of the K8S dashboard, I think it did). I have a cluster of three 16GB Tiny PCs on a LAN: all are running Ubuntu Server + MicroK8S, everything looks healthy, HA mode is automatically on.

Here is my version of MicroK8S: MicroK8s v1.26.4 revision 5219.

I created a tarball of a local Docker image, thus:

docker save k8s-workload > k8s-workload.docker.tar

I then side-loaded it to all three nodes (which was sent to all nodes without error):

microk8s images import < k8s-workload.docker.tar

This image works in Docker on the leader node, so it seems to be valid (whether it is an OSI image suitable for MicroK8S I cannot say, but if it had a problem I would expect images import to have reported it to me).

After much digging I get errors of this type (with a deployment):

Failed to pull image "k8s-workload": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/k8s-workload:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:e823...45c8: not found

Or if I try a run:

Error: failed to create containerd container: error unpacking image: unexpected media type text/html for sha256:1f2c...753e1: not found

Here is my status:

root@arran:/home/myuser# microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: 192.168.50.251:19001 192.168.50.74:19001 192.168.50.135:19001
  datastore standby nodes: none
addons:
  enabled:
    dashboard            # (core) The Kubernetes dashboard
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    ingress              # (core) Ingress controller for external access
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
  disabled:
    cert-manager         # (core) Cloud native certificate management
    community            # (core) The community addons repository
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated

A couple of thoughts to stimulate fix suggestions. I have not tried these as I am not fond of shooting in the dark, given that the more random stuff I try, the more clean-up of my cluster I will have to do.

• The registry is disabled by default, and I have not enabled it. I am strongly of the view that with the images having been side-loaded, they do not need to be pulled from anywhere. They are already in place.
• Images were converted to a tarball using docker save. I could use docker export to export a container instead, though I'd rather not do that - I want a clean image, and not any runtime cruft accumulated by a locally run container on my dev machine.

Corrections to any of my assumptions are welcome.

[halfer]

OK, I have solved it. I don't know what went wrong, but I found the fix - one of the nodes had acquired a corrupted image in containerd, and by frustrating coincidence, that was the node selected by K8S to run the workload.

I have a high degree of confidence that the image was corrupted by MicroK8S (or something downstream, such as containerd) since the only way I could have imported it is via the microk8s command. I did try per-node image imports (via microk8s ctr image import tarball.tar) and once this was found to fix it, I used sha256sum to verify the local tarball. It was the same as the others.

In my Server Fault update, it can be seen that the faulty image had a wrong MIME type (text/html), a wrong checksum (sha256:5f76...a3aa), and a wrong size (218.4 KiB).

I'll happily help track down a bug if a committer would like to suggest some methods for investigation. However I suspect that the cause in this case is entirely lost to the ether.

I will leave this open so it is seen, but do please close if you wish.

[ktsakalozos]

Hi @halfer. That was an impressive job in debugging and documenting the issue! Thank you.

Before closing the issue, would it be possible to have an microk8s inspect tarball from the misbehaving node? Maybe we could find something in the logs.

[halfer]

Sure thing; there's no PII on these nodes, so it should be safe: inspection-report-20230614_172306.tar.gz. That is a snapshot from today.

For full disclosure, when running the inspection I get this error:

WARNING: IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT
The change can be made persistent with: sudo apt-get install iptables-persistent

But the tarball seems to still be produced.

I wouldn't know where to start looking in this impressive tarball, especially given its size when unpacked. But, it may help to know that the problem would have been caused around 7th June 2023, or perhaps a bit before, when I opened the Server Fault question.

(While the issue was frustrating, it has been great for my K8S learning process - lots of general debugging to get to this point!).

[barrettj12]

I've seen a similar issue, where sometimes the image I've pushed to microk8s is overwritten with text/html. I still don't really understand the conditions that trigger this to happen, but it seems to be happening a lot, especially on more recent versions of microk8s.

[thwhk]

Just hit the same problem, but I'm not side-loading image.

MicroK8s v1.25.13 revision 5876

---

Update:
I noticed that in microk8s ctr images ls results, image without PLATFORMS information had TYPE text/html, and the SIZE is incorrect.

[neoaggelos]

@barrettj12 @thwhk there are typically two reasons why this happens most of the time:

• docker save a.tar, microk8s ctr image import a.tar might not work depending on the actual image that is built. this unfortunately is due to the OCI image formats and comes from containerd.
• sometimes containerd will get back an HTTP 409 error instead of the image (hence the text/html type).

In both cases, importing the images exactly as shown in the docs, or dealing with image pull rate limits should help. If there is a particular image that keeps giving you trouble, please let us know and we can look into it further.

[barrettj12]

@neoaggelos thanks. I'm uploading locally built Juju controller images so I can bootstrap for development. The really strange thing is, sometimes the image is uploaded/pulled correctly, and I can bootstrap fine - but then later it is overwritten with text/html, even though I haven't pushed a new image to microk8s. Maybe it's something on my machine fiddling with the images, but I don't know what would be doing this.

[barrettj12]

For contrast, this is what happens when k3s can't find an image:

Failed to pull image "jujusolutions/jujud-operator:3.3-beta2.11": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/jujusolutions/jujud-operator:3.3-beta2.11": failed to resolve reference "docker.io/jujusolutions/jujud-operator:3.3-beta2.11": docker.io/jujusolutions/jujud-operator:3.3-beta2.11: not found

Versus microk8s which says "unexpected image type: text/html". So the microk8s images are somehow being overwritten with text/html.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[halfer]

I'm not a big fan of stale bots; I get why they're used, but this does not look like it has been solved. Is there any harm in keeping this open?

[wilkmar]

I hit the same problem when I had the built-in registry enabled and tried to run a pod with a local image (that is an image imported to the containerd cache on a node, with the microk8s ctr image import... command). To use local images, I had to disable the local registry (plugin).
It looks to me there is a bug in the microk8s that prevents using local images when having built-in registries enabled.
My version: microk8s v1.32.0 7549 1.32-strict/stable

[pikhovkin]

same issue