exoscale: disk images fail to provision

[dustymabe]

I'm doing some testing on 36.20220325.1.0 for the Fedora CoreOS 36 Test Week. Trying to boot on Exoscale and it's not clear why initially. After grabbing the grub prompt and removing console=ttyS0.. from the kernel arguments I'm able to see it thinks it's getting garbage back from the meta-data service.

It turns out the user-data is now gzip compressed.

Another wrinkle is that you can't boot the nodes without any config at all. The service replies to http://169.254.169.254/1.0/user-data with an empty file. We need to handle that in the code as well. I guess in the past if this was the case the metdata service wouldn't reply with not found or something.

---

Update in #1160 (comment)

[bgilbert]

Another wrinkle is that you can't boot the nodes without any config at all. The service replies to http://169.254.169.254/1.0/user-data with an empty file. We need to handle that in the code as well. I guess in the past if this was the case the metdata service wouldn't reply with not found or something.

We do handle that in the code. What error are you seeing in this case?

[dustymabe]

Another wrinkle is that you can't boot the nodes without any config at all. The service replies to http://169.254.169.254/1.0/user-data with an empty file. We need to handle that in the code as well. I guess in the past if this was the case the metdata service wouldn't reply with not found or something.

We do handle that in the code. What error are you seeing in this case?

More info over at coreos/ignition#1345 (comment)

[bgilbert]

Okay, so you're saying we do handle empty configs correctly now, but the gzip case adds additional complications?

[dustymabe]

Okay, so you're saying we do handle empty configs correctly now, but the gzip case adds additional complications?

Maybe before they converted to compressed user-data it worked when there was an empty config. But certainly now it doesn't work. So I think the answer to your question is "yes".

I can do some more testing around this later and see if going back to non GZIP setting in FetchOptions allows it to work (If I remember correctly it still didn't work right).

[bgilbert]

As I understand it, you're seeing the metadata service provide compressed metadata if a config was provided, and provide zero-length metadata if no config was provided. In that case, an unmodified FCOS image should correctly handle the no-config case, since no gzip is involved there. If not, then our understanding is wrong, and we should track that down. A console log would really help here.

[dustymabe]

As I understand it, you're seeing the metadata service provide compressed metadata if a config was provided, and provide zero-length metadata if no config was provided. In that case, an unmodified FCOS image should correctly handle the no-config case, since no gzip is involved there.

Went back and tested. This is correct. With no config the current next image (36.20220325.1.0) comes up.

[dustymabe]

It turns out the compressed user-data only happens when using the exo CLI. When using the web UI the data isn't compressed.

See coreos/ignition#1345 (comment)

[dustymabe]

For the record here is the error the user would see:

[dustymabe]

OK. The exoscale CLI has been updated to not compress by default exoscale/cli#451 - Maybe we just update our docs to tell people not to use --cloud-init-compress.

[bgilbert]

Yes, and to use a suitable version of the CLI.

[dustymabe]

The exoscale CLI has been updated to not compress by default exoscale/cli#451

This landed in https://github.com/exoscale/cli/releases/tag/v1.54.0

[dustymabe]

docs update: coreos/fedora-coreos-docs#401