CoreOS "just the containers, ma'am" plus ChromeOS "set in stone" OS (but updateable) approach was so obviously good once you started using it that it was just begging to be picked up again.
Sounds like k3os, although it kind of surprised me when k3os development was stopped. Agreed: something like this almost seems inevitable. When you just want Kubernetes, managing the underlying OS and packages just feels like a waste of time.
The most significant change this acquisition brought to Flatcar was to turn it from a vendor-driven open source project to a community-driven one. Also see Kinvolk's statement at the time: https://kinvolk.io/blog/2021/04/microsoft-acquires-kinvolk/
Since then, the project has opened up and has become fully community driven - releases, features, stewardship, everything. To further underline independent stewardship, Flatcar was recently proposed to join the CNCF as an incubating project: https://github.com/cncf/toc/pull/991 .
Comments like this are unhelpful and unproductive. That team is totally independent and does some great work and has continued to be a totally community driven project.
> Microsoft has done quite a lot to rehabilitate their image
I don't want to sound as a anti-MS fanboy, but that's the standard procedure when you get to the point "If you can't beat them...". Microsoft isn't to be trusted just because they're a business, but because like many others they're after profits above anything else. Just look at how they turned a potentially very decent OS (Windows 7) into worse privacy nightmare at every new version. I simply don't want them to mess with Linux.
Absolutely not. When the paychecks come from a company that fundamentally does not believe in the four freedoms, that is critical information with regards to infrastructure.
Free software is an ideology and philosophy, not just a license.
Agree with sibling comments. Not immediately relevant to me but at my current position we go out of our way to avoid anything and everything Amazon. Very hard to do. And if a project were backed by Amazon I would be very embarrassed to start building on it at work.
We are so serious about this that I almost got a contract canceled because it looked like they were running on AWS, and we got them to switch to Azure.
I agree it does read like the punchline to a joke, that did not escape me when I wrote it, but the larger message is that comments explaining who really owns projects we're talking about are not "unhelpful and unproductive".
Many of the sibling comments were coming from the maintaining freedom perspective, which I personally appreciate as well. I wanted to chime in that this information is also helpful from a corporate/competitive perspective.
Kinda related, but a flatcar was historically a kind of railcar you could mount shipping containers, chassis and large +100 short ton loads onto.
These days we call them intermodal frames or an intermodal chassis. It can go from rail to cargo ship to truck to plane, and its all highly standardized and scalable :) CSX even uses them to test their robotic conductor/ automated locomotive technology
I mean flatcars are still used today (not just historically) for things like tanks, farm equipment, lumber, semi-trailers, and sometimes even standard containers (https://en.wikipedia.org/wiki/Flatcar)
Fedora IoT is an easy step to this as well. It's really "Silverblue server", rpm-ostree based but without making the jump to ignition files right away.
at least to me the declarative setup of ignition was the selling point, not the rpmostree (altho it is nicer than dnf/old school rpm). Another cool thing about fedora-based is that it seems much more bleeding edge in terms of kernel version on other os packages than chromiumos which is what I want for immutable infrastructure
For more on immutable / image-based Linux you might want to check out the UAPI group, a recently founded cross-distro interest group: https://uapi-group.org/
We have contributing members from many FOSS projects gravitating around image-based Linux, including quite a few distros.
Yes this is a direct fork. I happened to have been a CoreOS Berlin employee and know the kinfolk folks (now acquired by Microsoft) that did this fork after the Red Hat acquisition very well. I’m very glad they did, a lot of things I liked a lot about CoreOS container Linux unfortunately didn’t make it into Fedora CoreOS.
The kinfolk folks also run the update server to perform auto updates.
//edit
Continuation wouldn’t be a wrong word either as it all coincided with the Red Hat acquisition. But I guess “fork” is correct since it’s under a different name.
A couple of small things, but primarily I don’t think it actually embraced immutability in the same way. CoreOS container Linux used A/B partitions for updating truly immutably. Partition B didn’t boot? Just boot A again.
Fedora CoreOS and RHEL CoreOS use OSTree [1], no shade on OSTree, it works well, but I slept better with A/B partitions, I just find them easier to reason about.
From Fedora CoreOS docs: “ When an update is complete, the previous OS deployment remains on disk. If an update causes issues, you can use it as a fallback.”
IIRC RHEL now has tooling for automatic rebooting into the old version when checks are failed. Not sure how hard that is to do with Fedora CoreOS.
Guess I’m not sure what functions are missing in these implementations that would cause you to lose sleep.
Appears to be a continuation of that project under a different name. Someone else pointed out that this one was acquired my Microsoft in 2021, but does appear to be receiving regular updates so it’s not sunsetted yet.
I'm working on a AWS Bottlerocket remix with self-hosted capabilities and Azure/GCP support, and we're planning to release it for a ready to use opinionated CNCF stack Kubernetes distro (about 40 different tools), similar to VMWare Tanzu and RedHat's OpenShift.
We were facing a lot of controversy behind Flatcar already - there's no clear Licensing policy for their "Fully OpenSource" release. Kinvolk should've prepped at least a License matrix of what's been hacked out of CoreOS - what is actually behind their "Pro Features" and how exactly it is "Fully Open Source".
"Flatcar Linux Pro Features", in terms of "Linux Kernel Optimization', "GPU instances support" with "Accelerated Networking", are Not So Pro, and are something that everyone use on a daily basis.
So, just from pure Support Monetization perspective, it's impossible to use Flatcar Linux reliably for Free (mic drop).
Thanks for dropping the mic, I'll kindly pick it up.
I'm the initiator of the Flatcar Container Linux project and former CEO of Kinvolk. Thus, I'm rather knowledgeable about the project and was involved in most decisions.
The controversy you speak of is very new to me. If you could point to any references, I'd love to be aware of them.
Firstly, there was nothing "hacked" out of CoreOS. Flatcar is literally the CoreOS Container Linux repos forked and carried on as is. Once the CoreOS EOL was reached we started updating the stale packages. That's it. Any further updates are what any distro would do in the course of maintenance to remain modern and relevant.
Secondly, anything that was previously termed the "Pro" version is now just available in the standard version. So there is no difference. To my knowledge, the project doesn't even produce any Pro versions any longer and I don't think there are even any references to it in our docs. But even when we did have a Pro version, all the work we did was done in the open and was in our source repositories. We just didn't release public builds of those.
Unlike CoreOS, we also developed* and open sourced the update server. It's called Nebraska and available here under an Apache license. https://github.com/kinvolk/nebraska
If you do find anything that is not 100% open source, let me know and I'll follow up to make sure that's corrected.
I'm happy your excited about your project. But I think you'll fine it's better in the open source space to compete on merit and form relationships rather than tear down other projects and the work of the people behind the projects.
My point is that Flatcar positioning is too cryptic and ambiguous for a lot of folks. People don't really get why they should pay for it, and that's a customer reachability issue, - a marketing problem, not an engineering one.
I'm not saying that no one should use it, or it's a bad product, it's just takes too much time and effort to get into it, to understand how the pieces of the puzzle are tied together, and why. It's really great that you've picked up EOL'ed CoreOS services and developed your own. A visual component diagram, with a couple of license icons, and a simple graphical bare metal installer would've been really nice.
> you can find all licenses for each release...
Packages SBOM doesn't give a full answer regarding architecture, existing priorities (if there's any) and project structure: what had been adopted from CoreOS and why, and what had been developed from scratch, and why ?
My primary complaint regarding "Pro" feature set, and further monetization, is that literally every growing business, with a cloud hosting provider, would target those, which makes them a necessity, not an extra option to choose from.
FlatCar should've developed something innovative to differentiate the market a bit, and monetize complex enterprise deployments, instead of sabotaging onboarding for the new folks (customer reachability), with a flat fee and "unpublished builds", of a supposedly free Open Source product. At least that's the feedback I've been able to gather on my own, so far.
> We just didn't release public builds of those.
So, if I get it right, FlatCar monetization may crumble with a "Community build" that will package all the "Pro Features" and let people use 'em for free ?
Not trying to devalue anything, but "just a viable CoreOS fork" doesn't make things magically self sustainable. It's all about the extra services that you can put on top, and real engineering problems that you may solve, for those who are willing to pay.
Ignition is a tire fire and I despise it with all my heart. It's made worse by there being like 5 different incompatible under-documented versions each with their own cutesy bugs
In last contact I had with Flatcar (before we moved to Bottlerocket) one could still use cloud-init yaml even with Flatcar so that's what I'd recommend instead of trying to read through the unbounded number of github issues trying to track down why simple things don't work with ignition
As for "what advantages Flatcar provides," the answer is always "in comparison to what?" I haven't run a general-purpose OS in production in over 8 years, and I don't intend to ever go back to having a mutable OS. There are just too many opportunities for randos to "just ssh in and do this one thing" and then I have to spend hours tracking down why some machines behave different from their peers
> Ignition is a tire fire and I despise it with all my heart.
Could you elaborate on this? I helped [1] create Ignition with Alex Crawford [0], its original author, while @ CoreOS in ~2015. To call what we were making then a "tire fire" seems comically hyperbolic for what was such a rudimentary bootstrapping tool, and generally unexpected for something we'd ship.
Has its scope grown out of control since those early days, or has it been mismanaged? We're coming up on a decade since my involvement...
Ultimately I'm sure I'm going to be sorry I weighed into this, because maybe it's just personal preference and I'm not going to change anyone's mind and you are unquestionably not going to get me to ever touch that again. And I'm sorry if your feelings are hurt, and I appreciate that you and your coauthor know and love the system but my experience with it mirrors that Microservices YouTube joke
My biggest heartburn is that I have[1] to compile a boot configuration document. Some of that may be a concession to Ignition being inexplicably written in JSON, but it's possible the transpiler does some massaging and restructuring of keys, I dunno, I didn't want to compile a boot document
I grant you that https://coreos.github.io/butane/examples/#files may be the example of pre-transpilation using the `inline:` structure but for my use case of authoring that UserData dynamically, I don't have a `!TranspileUsingCuteNameOfTheDay` in CloudFormation
And since I haven't had my hands on the Flatcar ecosystem in a few years, I'm not going to have access to the very specific pain points of "well, this cutesy named thing is broken because it depends on this other cutesy named thing" but what it boils down to is cloud-init behaves sanely, has been around forever, and can be authored easily in a bunch of different circumstances
fn-1: yes, I can appreciate it's "just a JSON document" but no reasonable person is going to author JSON string literals containing data: URIs for contents
“Ultimately I'm sure I'm going to be sorry I weighed into this”
Maybe you wouldn’t be if you didn’t use phrases like “tire fire” at other people’s work?
It sounds like Ignition isn’t your cup of tea, which is perfectly fine and few people could take issue with that. Tire fire is unnecessarily caustic and doesn’t really improve the discussion.
I guess if you found my comment to be "comically hyperbolic" then replying to mine with a "comically reductionist" is fair game
So, anyway, I actually did dig up a concrete example of my experience with it, and I cannot link to the "Additional information" section but that is both why I think the thing was a mess and also why the Miroservices YT joke resonated: https://github.com/flatcar/Flatcar/issues/220
I think the CoreOS boot strategy was decomposed into a bunch of different executables, each responsible for doing their own little slice of the world. Maybe it drew inspiration from systemd in that way. But, just like my real life experience with microservices, it requires keeping a bunch of different projects and their upgrade paths in ones head, knowing their disparate config formats, and when one of them inevitably has a bug, understanding how to troubleshoot what went wrong with the system as a whole
And, again in trying to be reasonable in this discussion[1] I do also understand why one would opt for the data URI, given how much of the rest of Ignition loads content from URLs. I don't believe cloud-init has that remote content paradigm baked into it nearly the same way, so I hear you about that.
And yes, my belief is that JSON is a data-exchange format from _computer to computer_ and making people write them is a poor DX choice, IN MY OPINION. And, to reiterate, I know that CoreOS's perspective is that it is a computer-to-computer transmission from the transpiler-project-o-the-day to the Ignition binary, but that is predicated on one having access to that transpiler binary in all cases, which is quite different from the problem that cloud-init is trying to solve
fn-1: I'm sorry you got hurt by my "tire fire" outburst, and that evidently derailed this whole interaction, but it was my experience
I have one complaint about Ignition: its builtin file download functionality is too limited:
- there is a limited amount of retries (5?), so a network hiccup can easily cause the download to fail
- no control over which responses are fatal, so if you boot a VM with a config that references a URL that's still 404 due to async nature of something, then only a single attempt is made.
The company I work for uses Flatcar exclusively, but we don't use config chaining due to these limitations, and we have moved all file downloads to the shell scripts embedded into the config.
Infrastructure using cloud-init is under risk of suffering from config drift. Cloud-init runs at (each) boot while ignition runs once, at provisioning time.
The benefit is reproducible deployments: Any deployment of a given configuration will reproduce with the same end result. Configuration is kept outside of the instance and does not drift. This is guaranteed across releases / versions (something we very explicitly and thoroughly test, for each of our releases).
(Perceived) drawback can be that it's a bit of a paradigm shift for operators. While Ignition eliminates the need to meddle with instances after provisioning, it can be a bit of a learning curve to produce a node's full config beforehand.
Yes I know, but Ubuntu seems to be more popular as a desktop
What I mean is that people just hack on stuff on their desktop. They install Python NumPy or Rails or whatever. Like gigabytes of dependencies.
And then they want to deploy.
Is it easy to do that with Flatcar Container Linux? How do I do it?
Or is it easier to make a Docker file that does apt-get install X and pip install Y ?
You can do that with Ubuntu as a base image, and Debian usually "just works" if you're on Ubuntu. (I'm doing this with our Debian CI images)
Not saying you should always take the easier path, but I also think it's a good idea to minimize the differences between the dev environment and production
It's intended for a workflow where you build docker containers, and deploy docker containers, so the environment is identical. If all you deploy is containers, it makes sense for the host OS to be as small as possible, and as immutable as possible.
True, but devs like me started with Ubuntu. Then years later we see a Debian docker image and wonder how different it will be. Mostly the same, for the reason you mentioned.
The intention is to do all of your development in containers, in whatever environment you choose, and then deploy the containers into production on a Flatcar cluster.
As Flatcar is a very minimal, immutable distribution to run containers / Kubernetes, our development environment (aka the "Flatcar SDK") is containerised too.
Flatcar releases are tied to git commits in our distro automation repo; so to develop for Flatcar, one would:
1. clone the distro automation repo
2. check out the release version one wants to develop for (tag or branch)
3. starts the SDK container via a wrapper script in the repo
This enables Flatcar development across all distros that run git, and docker or podman (it even works on WSL if one is inclined to use it that way).
My memory is hazy but here's how I remember it: After Red Hat acquired CoreOS they announced their intent to rebase the entire thing around rpm-ostree, which is the CoreOS people know today: https://coreos.github.io/rpm-ostree/
At the time there was some anxiety in the community as to what would happen, as there was no direct upgrade path from old CoreOS to new CoreOS. Theoretically if we all believed the kool-aid we were drinking it's just a redeploy, no pets!
Kinvolk came along, forked it, and made Flatcar Linux, which kept the A/B partitioning system, and more crucially, let you just change a config file and all your old CoreOS nodes would just move to Flatcar and then you were good to go. So now if you wanted to stay on the system you were comfortable with you could just use Flatcar. If the composability of rpm-ostree attracted you then new CoreOS have you covered. Red Hat deserves a hat tip here because in their documentation/blog they explicitly mentioned Flatcar as an option for people who wanted to stick with what they know, which I thought was cool and how I discovered it!
Later on Microsoft acquired Kinvolk and and then people raised eyebrows. I have not checked in a while but the folks involved continued to do their thing and run it like a good OSS project, hold public meetings, all that stuff. Microsoft gets a good hat tip here.
> The Kinvolk Update Service allows for defining instance groups, assigning update channels and controlling the frequency, time of day and rate of updates.
What is this? From a quick glance I don't see it mentioned elsewhere. A hosted service? Something with a different license?
Thanks for the feedback :) Apparently there's always room to improve our docs. A maintainer has taken a stab on that (see https://github.com/flatcar/flatcar-website/pull/263 for progress) and we'll update our docs shortly.
There are multiple ways to start containers, one of them is to define systemd units in Ignition. If that part needs to change, reprovisioning is needed but a new "flatcar-reset" tool (in Alpha) should make the barrier for reprovisioning lower by only requiring a reboot and being able to keep wanted local data. In the past Ansible was a common choice if neither Kubernetes, Nomad or even docker compose are used.
One of the biggest advantages of the UEFI firmware is that you don't have to boot from the SD card. They are really unreliable and also awful for performance. USB3 or even netbooting are much better options.
It's a distro to run containers on. It is configured declaratively before deployment, so there's no config drift. It's fully automatable and the OS is self-updating (atomically, including automated roll-backs to known-good states). The OS cannot be extended after provisioning and all its binaries reside on a read-only partition (under /usr). Updates always replace the whole OS partition (though roll-back is possible). The root remains writable so users can add containerised apps and services etc.
Think of it as rather strong separation between operating system and user-defined workloads (containerised apps, Kubernetes, etc.).
CoreOS "just the containers, ma'am" plus ChromeOS "set in stone" OS (but updateable) approach was so obviously good once you started using it that it was just begging to be picked up again.