Linux

Win The Race: Clear Linux Architecture & Performance Optimizations Review

Paul Nixer

— 7 min read
Win The Race: Clear Linux Architecture & Performance Optimizations Review
Photo by José Pablo Domínguez / Unsplash

Clear Linux Logo Many Linux distributions have really minimal differences between and trying to reach a customer with nice wallpapers, themes and UI decorations in general. Clear Linux is the unique software product backed by Intel, not based on any distribution and provides a lot of interested solutions under and above the hood:

  • Performance is the key: despite the solid synthetic benchmarks handicap, recommending try Clear on several years old laptop and realize the real improvements very fast, so Clear Linux can give new life to old machines and refresh budget gears.
  • Roll back to any previous release? Not a problem.
  • Removed /etc by mistake or just for "hard reset"? Stateless configuration by design.
  • Suddenly deleted a system file or directory? Relax, software manager will help.
  • Even if the state of software update program is removed - OS will be able to get and install updates.

Audience

Clear Linux isn't a typical Linux distribution like Ubuntu or Fedora and never been. Intel introduced the project in 2015 as "the performant Linux distribution for the cloud". The desktop version was released only in 2019. Looking from my experienced point of view, nothing should stop you from using Clear as your everyday gear. Of course, a lot of under the hood details and some software, you never have seen before, will push into an uncomfortable mood. Anyway, no reason to give up fast if you're not even an IT, DevOps, Cloud/Container deployments, or AI professional. Without fighting with the official website position, a typical Linux user with basic experience can handle the easy, spartan-straight installer UI and type one-two swupd commands to install software he needs for everyday tasks. GNOME is provided as default desktop environment, an easy switch to KDE, Xfce or tiling windows managers like Sway and i3wm are available too.

Hardware

Hardware compatibility is also different from popular Linux distributions; performance optimizations cut the list of supported CPU. A user doesn't need to search his CPU specs in internet, the clear-linux-check-config.sh script will check Clear compatibility: 64bit CPU, SSSE3 (Supplemental Streaming SIMD Extensions), SSE4 (Streaming SIMD Extension), VMX (Virtualisation support), CLMUL (Carry-less Multiplication set). Hard to find the oldest processor capable of running Clear, some users had success up to Sandy Bridge CPU launched in 2010.

Performance

When this article was prepared to publish, I did a fast digging on the internet to see what people said about Clear Linux performance. Most of them believe in version "because they use performance governor and GCC -O3 flag!" and nobody can explain why Ubuntu and Fedora can't do the same for years. The answer is simple: they can, of course, but it's far more difficult to do, and they prefer to keep rolling on their way.

Clear docs highlight the optimization aspects:

  • Full stack approach - use software to better support hardware features
  • Runtime performance - build options with longer compilation
  • Server and cloud are first class citizens, desktop & IoT also benefit from the improvements.

Linux kernel has more than one hundred patches at the moment, most interesting from performance aspect:

Inside the compiler's area, Clear Team heavily invests in GCC. Just count the commits: LLVM packaging repository has currently 162 commits, GCC - 1853 and more than 20 patches. It looks strange, because Intel is developing his own proprietary compiler called ICC, now based on LLVM project.

Clear also use LTO (Link-Time Optimization), PGO (Profile guided optimization) with mtune and march=westmere flags. Docs say default mtune is haswell but I've found skylake-avx512 instead, maybe they're a bit outdated or Clear use dynamic flags, depends on CPU micro architecture.

If battery power detected, thermald will be automatically enabled. It's a Linux thermal management daemon forcing CPU C-state (idle state, when CPU has no instructions left to run). clr-power-tweaks invented for setting reasonable power defaults for all supported platforms and energy efficiency improvements on idle state.

AVX (Advanced Vector Extensions) also using actively. For example, let's analyze the ffmpeg package (called not-ffmpeg 'cause reduced codecs support due to US patent anti-troll protection).

%build
export CFLAGS="$CFLAGS -O3 -Ofast -falign-functions=32 -ffat-lto-objects -flto=auto -fno-semantic-interposition -fstack-protector-strong -fzero-call-used-regs=used -mno-vzeroupper -mprefer-vector-width=256 "
%configure...
make  %{?_smp_mflags}

pushd ../buildavx2/
...
export CFLAGS="$CFLAGS -m64 -march=x86-64-v3 -Wl,-z,x86-64-v3"
%configure ...
make  %{?_smp_mflags}
popd
...
...
pushd ../buildavx512/
...
%configure ...
export CFLAGS="$CFLAGS -m64 -march=x86-64-v4 -mprefer-vector-width=256 -Wl,-z,x86-64-v4"
make  %{?_smp_mflags}
...

This shorted RPM spec shows the package compilation actually runs three times. First time with the -O3 -Ofast acceleration flags and LTO -ffat-lto-objects -flto=auto, second time with AVX2 x86-64-v3 micro-architecture level (psABI) and third time with AVX512 and v4 psABI -march=x86-64-v4. Now there's a question how to deliver right build to the customer machine 'cause they are not compatible, you can't run AVX512 build on old CPU. The solution is easy: let the package manager do it. -ffat-lto-objects generate big "fat" link-time optimization object file contains discardable intermediate language and true object: two-in-one on the finish line - LTO linking and normal linking.

/usr/bin/elf-move.py avx2 %{buildroot}-v3 %{buildroot}/usr/share/clear/optimized-elf/ %{buildroot}/usr/share/clear/filemap/filemap-%{name}

The elf-move.py script move AVX* executable files with libraries to filemap RPM sub-package and create manifest with hash sums and paths:

avx2
/usr/bin/ffmpeg
bin0b193e928d304fc1b686be8c9c7b573180685ffa42669564056cb2e7eda5b02c

Then clr-elf-replace trigger will detect host CPU capabilities and install the right ELFs (Executable and Linkable Format). Funny enough, I can't find any source code of clr-elf-replace, only .src.rpm package is available.

Clear Linux also use Glibc HWCAPS functionality. It allows the dynamic linker to load optimized versions of libraries from /usr/lib64/glibc-hwcaps directory. These optimizations depend on CPU micro-architecture, and using the "right" library here will bring performance benefits.

Installer

Installer is the first application you meet when starting to work with any Linux distribution. Clr-installer written in Go and GTK by Clear Linux Team.

Clear Linux Installer, source: official docs

Compact minimalistic style, just a username, partition table, and you're good to go. It's good 'cause super complex installers will scare a lot of potential Linux users - look on the installer developed by SUSE. Three modes are available: console, text-based interface and mass installer for big fleets. The installer can create the required partitions automatically or reuse available partitions with special partition names like CLR_BOOT, CLR_SWAP, CLR_ROOT. This option requires to manually run Gparted or similar application. Fedora and Ubuntu provide better user experience here, of course.

Bootloader

is also written by Intel and very interesting. No BIOS support by design but no reasons for panic: the Clear Server edition can be loaded on non-UEFI machines, thanks to Syslinux. In comparison to way bigger Grub2 it's very easy to work with, see below:

  • Get a list of all installed kernels:
$ clr-boot-manager list-kernels
* org.clearlinux.native.5.16.18-1137

  • clr-boot-manager set-kernel kernel_version - switch to old kernel

  • Kernel boot parameter should be defined in /etc/kernel/cmdline.d/*.conf and requires clr-boot-manager update command to make the changes permanent.

The Grub2 architecture complexity and 10x code size (11MB vs 1.1MB in tar.gz archives) are, probably, the main reasons for reinventing the wheel together with decreased boot time. /proc/rootfstype tell about supported filesystems for root partition: ext4, btrfs, xfs, f2fs. Zfs isn't supported, there's an OpenZFS guide for non-root partitions only.

Package management

At first sight it looks like completely stolen from Fedora/Red Hat 'cause the RPM specs around, and you can even install dnf package manager. But there's a catch: RPM usage ends on software compilation stage and autospec project invented to generate RPM configs and speedup packaging. Mind-blowing, right? So, why dnf and RPM aren't enough? Because of features, for sure, it's easier to invent new tools. All content delivery does swupd with next design patterns:

  • Rolling releases forever
  • The release is monolithic, no software can be added or removed.
  • A bundle contains all libraries needed to run the software, classic situation "one library update breaks the whole distribution"
  • Updates or downgrades only with releases switching.
  • Delta packs for update speedup.
  • Release repair: a broken or deleted files can be automatically restored from server.
  • GPG signed metadata with archives for bundles validation

Swupd syntax is very similar to popular apt or dnf utilities: swupd update, swupd bundle-add, swupd bundle-remove. In contrast with them, swupd supports autoupdate command and can restore OS to previous release with swupd repair --picky -V RELEASE. All available bundles list can be retrieved using swupd bundle list -a or web version.

Stateless concept

Probably the most interesting system design feature is stateless. The "state" means user configuration or any non-default configuration applied, a real split allows completely removing or replacing the user configuration. In classical Linux distributions with the many configuration files mix in /etc directory, hard to tell is this random configuration file provided by default or not. Default /etc is empty and OS will boot successfully even if you wipe it. So where is the default configuration? In /usr/share/defaults. And if application will not find its configuration file in /etc, will be forced to load default from /usr. All files from /usr/, /lib/, /lib64/, /bin/, /sbin/ are immutable for used 'cause swupd will overwrite them with OS updates. /usr/lib/modules, /usr/lib/kernel, /usr/local, /usr/src are whitelisted for kernel drivers and third party software.
Of course, no /etc/fstab by default, systemd can discover and mount all system partitions.

Security and Telemetry

Rolling release model, enabled auto-updates and constant vulnerability scanning allows Clear to fix security issues and deliver patched software updates very fast. All update content signed with GPG key for protection and swupd will ignore all non-HTTPS repository sources by default. Certificate management unified with "Trust" Store - one & only location for certificates. Latest Linux kernel version shipping by Clear ensuring security by providing fresh fixes. Clear Team already disabled old and insecure ciphers like C4, SSLv3, 3DES, SHA-1; SFTP disabled by default due to security considerations. Tallow, a fail2ban alternative, shipped by default and use systemd journal API to filter bad SSH login patterns.

Now telemetry server is down, according to the forum post. Client and backend are open source, this is a positive decision for reputation score.

Conclusion

Absolutely, Clear Linux deserves more attention as a server, cloud and desktop solution too. Hard job has been done to make it fast and stable, a lot of open source software written especially for Clear. Performance is hard, but let's avoid making excuses.

Read more