Fold Zincati features into bootc

[jlebon]

Zincati, the update agent used in Fedora CoreOS, currently only understands rpm-ostree.

As we work towards rebasing FCOS on top of bootable containers, we need to also make sure that we don't lose some of the update capabilities that have served us well in FCOS release engineering. This includes the concept of phased rollouts and an update graph.

In the interest of providing a better UX than what we had with Zincati + rpm-ostree, it would be great if bootc directly learned some of these concepts. I think they're also generally useful to other bootc consumers, even if just for the rollout capabilities.

At a high-level, the proposal is to be able to point bootc at an OCI artifact which contains a JSON payload that describes update information, which bootc then knows to routinely check and apply updates based on that. We can dig into the details more once there's overall agreement on the direction.

[cgwalters]

Previously: coreos/zincati#904 (comment)

I am still somewhat skeptical that most users will want to do an update graph versus the path of simply orchestrating updates with a custom agent injected into the image. The core target audience for bootc is just different in that we expect users are running container native infrastructure already. (Yes you can run standalone systems of course, but the point remains)

If we added zincati directly into bootc, where would the dividing line be? Would we also directly merge something like that MQTT support?

IMO we need to just do one thing and do it well here, offering reliable lower level APIs. And yes we have that same "agent problem" here #337
But I think that's tractable.

In the immediate short term, wouldn't it be way simpler to patch zincati to talk to bootc and keep it as is?

[jlebon]

In the immediate short term, wouldn't it be way simpler to patch zincati to talk to bootc and keep it as is?

Hmm, I might've misunderstood. Keeping Zincati separate was the original plan. I opened this based on the last discussion we had about it where I thought you were favourable to folding at least some of the features to provide a better UX. We didn't dig into what specific features exactly, so maybe that's where the misunderstanding is.

I am still somewhat skeptical that most users will want to do an update graph versus the path of simply orchestrating updates with a custom agent injected into the image. The core target audience for bootc is just different in that we expect users are running container native infrastructure already. (Yes you can run standalone systems of course, but the point remains)

Yes, but in practice a systemd timer is just too simplistic for most deployments. That means you have to resort to a custom agent quite quickly. I think keeping that out of bootc is fine. Where I see something like Zincati be useful is providing some features that are simple but powerful enough to meet a lot of custom update behaviour requirements, without having to resort to owning a separate agent. Note BTW that Zincati is also used in clusters, including in Kubernetes.

That said, I need to look more at the other alternatives in this space. If there's something well-established that we could pivot to without losing some of our key capabilities, I think we should consider that also. (Though I suspect a lot of these agents are intended to run in containers and not be part of the host.)

Anyway, sounds like we can just close this and focus on #337!

[cgwalters]

The big fundamental thing going on though IMO (and we covered this extensively in coreos/fedora-coreos-tracker#1263 but relinking for everyone else) is that there are two nearly fundamentally different things:

• Running an OS produced by someone else and just configuring it, letting dnf update or auto-update images (CoreOS) run
• Owning the OS as a container image (here)

In the first case, the fleetlock thing for example just configuring the client side, the admin doesn't need to generally care about the upgrade graph, and they rely on the OS doing it.

It's soooo different as soon as "manage upgrade graph" becomes a load-bearing part of downstream user's workflows. The jump in complexity is very high IMO - too high to be fully baked into the client such that we expect many users to do that versus the alternatives.

For example, a simple alternative is doing "upgrade graph" with container image tags; we should make it ergonomic to do the automatic equivalent of bootc switch across a tag; this can be done manually by injecting a systemd unit into the image, but maybe we should go straight to having /usr/lib/bootc/config.yaml/etc/bootc/config.yaml and get away from relying on the magic ostree origin bits...except the inherent circularity always felt weird.

Anyway, sounds like we can just close this and focus on #337!

Yeah, this is a common thing we need across use cases.