Microsoft's in-house professional networking site is moving to Microsoft's in-house Linux. This could mean that big changes are coming for the former CBL-Mariner distro.
Ievgen Priadka's post on the LinkedIn Engineering blog, titled Navigating the transition: adopting Azure Linux as LinkedIn's operating system, is the visible sign of what we suspect has been a massive internal engineering effort. It describes some of the changes needed to migrate what the post calls "most of our fleet" from the end-of-life CentOS 7 to Microsoft Azure Linux – the distro that grew out of and replaced its previous internal distro, CBL-Mariner.
This is an important stage in a long process. Microsoft acquired LinkedIn way back in 2016. Even so, as recently as the end of last year, we reported that a move to Azure had been abandoned, which came a few months after it laid off almost 700 LinkedIn staff – the majority in R&D.
The blog post is over 3,500 words long, so there's quite a lot to chew on – and we're certain that this has been passed through and approved by numerous marketing and management people and scoured of any potentially embarrassing admissions. Some interesting nuggets remain, though. We enjoyed the modest comment that:
That strikes us as putting it mildly. The way that Red Hat redefined its free distro has caused industry-wide consternation and even outrage, but few would have pegged Microsoft as one of the companies affected. We're also amused that Priadka cites "strong vendor support," defined as "having support from the OS vendor or a reliable support provider," as a motivation. This isn't a shot at Red Hat; it didn't support CentOS anyway. It is, however, praise for the Azure Linux team, and implies that they are on a par with any of the vendors that will give you paid support on CentOS.
There are some interesting technical details in the post too. It seems LinkedIn is running on XFS – also the RHEL default file system, of course — with the notable exception of Hadoop, and so the Azure Linux team had to add XFS support. Some CentOS and actual RHEL is still used in there somewhere.
That fits perfectly with using any of the RHELatives. However, the post also mentions that the team developed a tool to aid with deploying via MaaS, which it explicitly defines as Metal as a Service. MaaS is a Canonical service, although it does support other distros – so as well as CentOS, there may have been some Ubuntu in the LinkedIn stack as well.
Some details hint at what we suspect were probably major deployment headaches. LinkedIn was using DKMS to build drivers into its kernels on the fly, which is a common use case and implies that Nvidia cards are involved. However, the Azure cloud insists on signed kernels, so Microsoft has built a repository of signed kernels incorporating all the hardware in use. Although the section about container images is garbled, Microsoft has also built an internal container image repository for Azure Linux to pull images from.
- Microsoft freshens up its in-house container Linux, CBL-Mariner
- Fujitsu gets ready to eat its own dogfood as company-wide digital transformation project kicks off
- Amazon’s Away Teams laid bare: How AWS's hivemind of engineers develop and maintain their internal tech
- MS Hotmail servers begin switch from FreeBSD to Win2k
Some of the other information covers things the teams did not do, which is equally informative. Development used to be done on "a full-fledged CentOS desktop VM with a window manager" – but Azure Linux doesn't have a GUI or any kind of desktop, and they haven't added one. They've found a way to connect IDEs running locally to remote Azure Linux VMs, complete with GPUs – over 1,500 of them, in four different geographical regions. As you would expect, it mentions Microsoft's own VS Code, but also JetBrains' IntelliJ. LinkedIn was founded back in 2003, and any technology stack that's 21-plus years old is doubtless messy and contains a lot of different vendors' parts.
Microsoft has a long history of "eating its own dogfood," even when the exercise of moving acquisitions to its own stack has been painful. It acquired Hotmail way back in 1997, but even two years later, as the still-young Register was reporting in 1999, it still ran on a mixture of Apache on FreeBSD, with Sun Solaris-powered database servers.
Back then, Windows NT 4 Server simply wasn't up to the task. It wasn't until Windows Server 2000 that it managed to begin the transition, as The Reg reported in 2000. Microsoft even wrote up a detailed report, Converting a UNIX .COM Site to Windows, describing the process. As late as 2003, the replacement was still in progress.
The result of this major undertaking was significant improvements in the abilities, compatibility, and also stability of Windows Server. "Dogfooding" is an important effort. This vulture has in the past been a member of staff at two of the big three enterprise Linux vendors, and I speak from personal experience when I say that those companies relied heavily on external third-party vendors for their IT infrastructure – and, as far as I know, still do.
Microsoft is doing the right thing here, shouldering the burden, and considerable cost, of moving its own internal infrastructure to its own products. This can only be good for Azure Linux, and indeed, for Azure in general. Its fairly obscure and limited little Linux distro is growing up fast, and we'd bet on major functional improvements coming in a future release.
And, indirectly, it's all thanks to Red Hat. ®