Add support for bootc install --with-overlay quay.io/exampleuser/someoverlay:latest

[cgwalters]

I've been advocating general configmap and secret support for a while.

However, the way CoreOS does things today with Ignition is basically allowing the admin to inject "day 1 unmanaged state" that is decoupled from the image/host. And actually, that's how Anaconda kickstart works too.

There's a lot of advantages to configmap-in-registry around supporting "day 2" changes too. However, it will require workflow and tooling changes to adapt.

One key thing I'm narrowing in on is the basic use case of "inject a ssh key into an existing golden image" - a thing I think we really want to support. And we do today via e.g. Anaconda or other installers.

But I think we should support this more natively via bootc install too.

So here's my strawman: the --with-overlay argument takes a container image reference and operates only at install time. A very key difference from the configmap approach is that the image must drop files into /etc and /var. This gives us immediate, direct support for SSH keys by injecting e.g. /var/roothome/.ssh/authorized_keys.

Migrating from Ignition

But actually the neat thing with this is it provides a very seamless adaptation of existing Ignition configuration! We've already prototyped out support for extracting an Ignition configuration to its set of files via ignition-apply, so it'd literally look like:

FROM quay.io/coreos/butane:release as builder
COPY foo.ign foo.ign
RUN ignition-apply --root /config foo.ign # render ignition to filesystem tree, error on partitioning

FROM scratch
COPY --from=builder /config /

Then push that to a registry as quay.io/exampleuser/someoverlay:latest, and one can do bootc install --with-overlay quay.io/exampleuser/someoverlay:latest and then one can switch to installing via bootc without needing to change all the bits in the Ignition config to move to /usr content etc.

So any tools (e.g. OpenShift MCO) that act on "day 2" management for this "unmanaged state" can seamlessly continue to work - but we avoid the need to e.g. host Ignition servers.

[cgwalters]

I think what would also help a lot here is integrated "factory reset" support. The baseline implementation of this idea would be like kickstart - unpacked and applied at install time - it persists but the "original" input is not referenced thereafter or tracked.

But taking it one step up, we could easily also record the reference to the input, and add a bootc factory-reset command that would build on top of reinstallation and when we do that we'd re-fetch the overlay container input and apply.

This would get us really close to a "immutable infrastructure but with easy in-operating system reprovisioning" flow.

[vrothberg]

Would make a nice demo as well 👍

[cgwalters]

Another thing we could consider by default is treating things as if we had VOLUME /var in our base image. So we'd allow content to exist there, but the semantics would be that we copy all the data there only at initial install and don't update it thereafter.

This would certainly unblock a lot of "obvious" flows like RUN useradd foo and getting a /var/home/foo to start with.

I have a bit of a fear it'd lead to confusion down the line, but it could probably be overcome with docs. And probably some sort of command to "re-synchronize"? Like bootc resync-var /home/ to pull the latest updates from the container image? A lot of details there, like whether we delete unknown files, etc.

[vrothberg]

Interesting idea on /var. I do not have an opinion at the moment so I trust your judgement.

[rhatdan]

I think the base image needs to have a bunch of VOLUME and CMD added Specifically VOLUME /var and VOLUME /var/roothome and /var/home. 'CMD /sbin/init'

Since the goal is to run/test images based on the base images as OCI containers we need to ensure that they work easily when run with --privileged mode.

[ondrejbudai]

@cgwalters What's the plan with this one? After we switch to bootc in bootc-image-builder, we don't want to write anything to the disk once bootc image to-filesystem finishes. However, that means that we cannot support any last-mile customization in b-i-b, which isn't great - the custom user one is fairly popular amongst users. The fix that we discussed with you was using --with-overlay, so this issue now blocks adopting bootc install in b-i-b fully.

[cgwalters]

In the short term...today implicitly in a to-filesystem flow, an external process has directly writable access to the filesystem it mounted and can write arbitrary data into the target deployment's /etc and /var. However, the main downsides right now with that are:

• need to dig into the target deployment root
• selinux labeling
• In general case, this requires mounting at deploy time external filesystems (e.g. /var/home) etc...this is what Anaconda does today, and this creates some messy things because one might be mounting partitions via systemd .mount units embedded in the OS image (and not kickstart)

Thinking about this a bit more if we constrain things such that we're just doing things in /etc - in particular e.g. writing systemd-sysusers fragments or custom systemd units which run useradd instead on firstboot, that seems like it gives sufficient power (really, writing systemd units in /etc is arbitrary code on firstboot) without getting into the external partition problem.

By far the biggest thing here that this pushes towards is an architecture where useradd runs on firstboot, not immediately because that has less-than-ideal semantics around /var/home.

For the bib case, this configuration may not be desirable in a registry.

Putting this all together, we could just add bootc install to-(filesystem|disk) --copy-etc /path/to/external/etc? That's pretty trivial.

[cgwalters]

PR in #267

[cgwalters]

One thing I noticed while looking at this is currently install to-filesystem goes to some effort to "finalize" the filesystem today by remounting readonly and using fsfreeze. Of course, nothing stops the caller from remounting writable again, but if we wanted to support arbitrary mutation post-bootc install we'd want an option to suppress that. (And the caller would need to do the same)

Having the /etc writes done via bootc keeps things consistent there.

[cgwalters]

Right now the b-i-b case is really just scoped to users, correct?

(Obviously Anaconda and %post allow basically anything today, but my hope of trying to add structure to that in the near term is low)

That's so tiny that I wonder if we aim to do a general mechanism right now (i.e. "write arbitrary files") versus just chaining together a sysusers.d fragment alongside a tmpfiles.d one to write the SSH key.

Man, SSH keys are a messy topic...while thinking about this I did CentOS/centos-bootc#154 which would notably break with what b-i-b is doing today, but work very nicely if we setup the keys via systemd tmpfiles.d snippets.

This topic of SSH keys also tangentially relates to things we could do by default with AuthorizedKeysCommand - xref discussions in coreos/ssh-key-dir#10

[cgwalters]

The ergonomics around glomming together sysusers.d and tmpfiles.d are pretty low though...we could support something like RUN useradd --sysusers someuser --sshkey <key> and have that write out a requisite snippets.

[jmpolom]

But taking it one step up, we could easily also record the reference to the input, and add a bootc factory-reset command that would build on top of reinstallation and when we do that we'd re-fetch the overlay container input and apply.

This would get us really close to a "immutable infrastructure but with easy in-operating system reprovisioning" flow.

I think it would be extremely desirable if the references could be tracked and subsequently manageable separately (as in removed and updated) from the underlying OS image. This would allow instance specific configuration info to be nicely buildable and manageable and also separating the instance "state" from the runtime environment.