Interesting write up, but it would be useful to explain how major graphics APIs fit into this (i.e. Vulkan / OpenGL) and how it all glues together (GPU drivers with KMS/DRM and higher level APIs).
Also note, that Mir is now basically gone, Canonical decided now to stop that rift and stick to Wayland.
Is there a decent write-up of how the graphics stack goes, from userland to the monitor? I have a really hazy picture of how framebuffers, drivers, compositors, GPUs, display servers etc. all fit together. I've read good tutorials about the various pieces, but don't really grok how they inter-relate.
It's a bit outdated and honestly needs replacing at this point. Xplain[0] replaces parts of it, and some of my newer blog posts[1][2] replace other parts. I can also answer any questions people have about the stack.
Also, I'm curious about portable (i.e. across different Unixes) low-level graphics, if such a thing exists.
I assume that the interfaces discussed here are Linux specific, and the BSDs have something different.
Does POSIX have anything to say about graphics interfaces?
What is the lowest-level portable way to draw graphics? X11?
It's funny, because I always thought of X as being more of a windowing abstraction, rather than graphics in general. Perhaps I am making a distinction that does not really exist.
The BSDs and Solaris/Illumos have had DRM for years (where available), but it tends to lag the Linux version significantly because it's not trivial to port the code from Linux. I'm not sure where the compatibility breaks relative to the code given on the page, but most of them have some derivative of the original drm.h header that defines most of the structs and ioctls mentioned (e.g. [1] [2] [3])
> Does POSIX have anything to say about graphics interfaces?
Not at all. Writing 100% compliant POSIX code means sticking to the PDP-11 model of what an application is supposed to be, meaning either a terminal text application or a daemon.
Anything graphical related is a separate standard.
I don't think there is any common low level way, even among the Linux flavors. Android's graphics stack, for example, is completely different and doesn't use any of this.
If you want portable you're going to want something like SDL probably.
Andriod is a weird deviation, and kind of annoying because the rift goes all the way to libc, which makes Android blobs incompatible without translation hacks like libhybris. If in case of GPU you can at least get something like Freedreno to avoid such hacks, but good luck getting open drivers for the modem and the like. So those blobs are a necessary evil for some devices to be usable, and there are none for glibc.
libhybris would still exist if Android had been running glibc. You still have architecture-specific stuff at the end of the day. Just like how XWayland exists to map X11 to Wayland, even though both use the same glibc.
Looking through the source of libhybris it looks like very little of it has anything to do with bionic itself.
I thought its whole point is to work around idiosyncrasies of bionic, and translate it into glibc proper. Or you mean that Android drivers architecture is very different from proper Linux drivers?
Mesa and SurfaceFlinger do two completely different things. You can absolutely use Mesa on Android (the emulator does, for example). But just like on desktop you're typically using the GPU vendors' accelerated GL implementation instead of Mesa. But if the vendors driver uses Mesa, like the Intel GPUs, that still works fine on Android.
bionic is largely just a subset of glibc. There's not really any idiosyncrasies to workaround going from bionic to glibc (the hard part is the reverse as people commonly use non-portable glibc extensions)
libhybris is more things like converting Surfaceflinger calls into Wayland ones. Looks like there's some media, camera, and input stuff, too, which is all outside the realm of bionic/glibc.
Actually Nvidia blob didn't start using DRM/KMS until very recently. I think it's more because of Mesa. It makes sense for FreeBSD to leverage the major open project which does heavy lifting for graphics.
Ok, some corrections first. Wayland is a protocol, not an implementation. Xorg is an implementation of a protocol, not the protocol itself.
Weston is the reference implementation of Wayland, but it is not the only one - KDE's KWin is another.
Xorg is the current most common implementation of the X11 protocol, however - again - it is not the only one (although almost all Linux desktop systems today use it) and there are other implementations.
So how do X11 and Wayland relate to each other? Well, they are both protocols for providing a windowing system. They each do things differently, with pros and cons to each approach. They are different enough so that one can be implemented on top of the other, so compatibility should not be a problem, although obviously features are lost or crippled when doing so.
There is this misconception that Wayland obsoletes X11, but they are very different systems designed by different people and worked by (mostly) different developers that only superficially provide the same features (showing windows on screen) that - at least personally - i do not see that happenning any time soon, if ever. Personally the way i see it is similar to asking if Atom (or VSCode, if that is your poison) obsoletes Emacs (or Vim).
Weston and KWin are Wayland compositors. Wayland should obsolete X11 (it's not a misconception, it's one of the primary goals), but it's a humongous task, especially from applications side. And Wayland is not yet feature complete to replace all common use cases. For example it's still lacking standard protocol for screen recording and screenshots, that fits its security model.
Also, there are no guarantees that any mainstream browser will ever use the Wayland protocol directly without going through an X compatibility layer like XWayland. And XWayland consists of most or all of the X11 code base.
Yes, Firefox and Chrome are open source, but they're atypical of open source projects in that (a) the amount of implementation effort required to make the kind of changes under discussion here are much larger than for typical open-source projects and (b) institutional users and knowledgeable (or simply well-advised) individual users of Firefox and Chrome won't use your fork unless you have a credible story about how you can protect them from 0days. I know of no fork of either Firefox or Chrome that has ever had a credible story, but I've spent only a couple of hours looking.
Google has their own replacement for X11 which they use in some Chromebooks, which reduces the probability that they'll throw in behind Wayland.
Not every OS install needs a mainstream browser (in-vehicle entertainment systems do not seem to: they're heavy users of Wayland) but if your OS install does, and you're waiting for Wayland (more precisely, software that uses the Wayland protocol) to replace xorg or X11, you might be waiting a long time.
"mainstream": provides a decent experience on most popular web sites including e-commerce sites.
> Also, there are no guarantees that any mainstream browser will ever use the Wayland protocol directly without going through an X compatibility layer like XWayland.
Aren't Google using Wayland in ChromeOS? If not, it's pretty bad and just another rift like their SurfaceFlinger on Android.
I'm also waiting for Firefox to start supporting Wayland without X11, but it's progressing really slowly.
Besides browsers another very major work is required for Wine. It's not even moving anywhere now, since developers aren't sure how to address it. They are too dependent on X11 for windows placements and etc.
So XWayland will need to be adequate, to address these cases, including gaming, until the time major projects will complete all that work.
> Aren't Google using Wayland in ChromeOS? If not, it's pretty bad and just another rift like their SurfaceFlinger on Android.
Except SurfaceFlinger predates Wayland by years. Kinda hard to use something that doesn't exist. ;)
At the time the complaint was that Android didn't use X11 but I think it's safe to say that was a good decision at the time. As for merging them at this point they both have too much inertia combined with some fundamental design differences that'd be hard to reconcile.
Their main failure wasn't just the fact that they didn't collaborate with Wayland (which wasn't a lot later by the way), but the fact that they didn't want to work on big problem that benefits Linux as a whole (replacing X11), but made something that benefits only them, and that created a huge and damaging rift. That they can be blamed for.
In contrast, Nokia didn't want to make such rift, and until it went bust, used common graphics stack (with SailfishOS becoming Meego descendant, using Wayland).
Wayland was started the same year that Android shipped and Wayland didn't have it's first release until 4 years later. That's obviously way too late for Android to have used it or collaborated on it.
> but the fact that they didn't want to work on big problem that benefits Linux as a whole (replacing X11), but made something that benefits only them, and that created a huge and damaging rift. That they can be blamed for.
At the time nobody wanted a replacement to X11. That was a big complaint against Wayland, too.
Getting into mailing list wars and trying to make everyone happy ensures you end up like Nokia - too late to matter and with no more money. That's not helpful to anyone.
Also please explain how Google created a "huge and damaging rift"? What damage resulted from this at all?
A major part of that problem is that rift that created incompatibility and prevented reusing of the existing progress. That was the whole reason for making libhybris, to unblock that issue.
That article is long outdated. Mainline and Android kernels have mostly converged at this point.
But that's also wholly unrelated to the specific context here which was Wayland vs. SurfaceFlinger.
Wayland was your "huge damaging rift" here if anything. Divorcing itself from both xorg and surface flinger, for no particular reason at the time in the case of the later.
The article doesn't focus on drivers, but idea is the same. Android did create the rift, and it's still here. It's new "Windows" in hardware compatibility sense.
> Getting into mailing list wars and trying to make everyone happy ensures you end up like Nokia - too late to matter and with no more money. That's not helpful to anyone.
Not arguing about the rest of your post, but the above is incorrect.
Nokia didn't go bust because they tried to cooperate with the rest of the world. You can be cooperative and still say no to people.
If I'm not mistaken Android was developed in submerged mode for a long time, for business reasons.
Well you can also look at how attempts by Google and others to upstream went initially. Mainline outright rejected some of the core tenants of what Android was doing just because it hadn't been an issue on desktop/servers. Things have since slowly changed, but even stuff that shouldn't have been controversial (like ashmem) still took years and when mainline finally added an equivalent (memfd_create) it is just incompatible enough in core behavior that Android can't switch to it.
It was the other way around, it went badly because they didn't collaborate from the start. Android was a huge code drop when it was published and upstreaming wasn't exactly their primary focus.
Vulkan & OpenGL are just really fast and complicated ways of putting pixels into the buffer you get from Wayland, DRM, etc... It's somewhat orthogonal.
Yeah the specifics I can hand wave about but would like more nice details. You get a "native" window surface from Mesa from DRM (the GBM API is one choice here I believe). You pass these returns to EGL for your display, native window, and surface. You then bind the OGLES2 API and you're of to the races.
Also note, that Mir is now basically gone, Canonical decided now to stop that rift and stick to Wayland.