Image mode for Red Hat Enterprise Linux (RHEL) is designed to simplify building, deploying, and managing Red Hat Enterprise Linux as a bootc container image. It reduces complexity across the enterprise by allowing development, operations, and solution providers to use the same container-native tools and techniques to manage everything from applications to the underlying operating system (OS).
In this article, we will focus on security. Security is a continuous effort where best practices and tooling quickly evolve to keep up with the everlasting threat of vulnerabilities and security breaches.
We will discuss the following ways image mode for RHEL plays a key role in securing our computing environments:
The trusted software supply chain of Red Hat Enterprise Linux.
The high degree of automated security tooling of the cloud-native ecosystem.
The strong defense mechanisms of an immutable operating system.
A deep dive into all of the individual topics is beyond the scope of this article, but we will provide resources where applicable. You can also refer to our series of articles about image mode for RHEL:
Security benefits of RHEL
First and foremost, image mode for RHEL is RHEL. It is the same OS. If you've ever wondered if a specific security attribute or certification for traditional RHEL applies to image mode, the general answer is yes. Is FIPS supported in image mode? Absolutely, FIPS is supported.
Since image mode builds on top of over three decades of work that went into Red Hat Enterprise Linux, we should have a brief look at the security promises of RHEL. Let's start with the following quote:
"Red Hat believes that everyone, everywhere, is entitled to quality information needed to mitigate security and privacy risks, as well as the access to do so. We strive to protect communities of customers, contributors, and partners from digital security threats. We believe open source principles are the best way to achieve this."
In a nutshell, the security approach of Red Hat Enterprise Linux comprises three pillars:
Mitigation: Mitigate the risk of exposing your data, systems, or reputation by applying critical security updates and patches, rigorous and in-depth supply chain security, and scalable vulnerability management.
Automation: Automate security controls at scale and maintain them over time including hardware-based root of trust, network-bound disk encryption, SELinux mandatory access controls, and more.
Compliance: Verified security certifications, FIPS, built-in compliance tools, and automated system roles for managing SELinux, certificates, session recording, SSH, crypto policies, and more.
The bottom line is that Red Hat has you covered, and we have solutions in place to automate securing your computing environments. Going into the details of the various tooling and security technologies is beyond the scope of this article, but if you want to learn more about Red Hat’s security approach, please refer to the Red Hat Product Security page and the following video.
Red Hat Trusted Software Supply Chain
A software supply chain comprises the components, libraries, tools, and processes used to develop, build, and publish a software artifact. If we are looking at a large product such as Red Hat Enterprise Linux, the software supply chain is comparatively big and securing it impacts all stakeholders.
These steps are part of the supply chain which gets slightly more complex when bringing the cloud-native world into the picture:
Establishing best practices when working upstream with the open source community.
Composing an operating system of thousands of components.
Keeping things secure by shipping updates on time to hundreds of millions of users around the world.
Modern cloud-native development workflows use containers as the standard for building, deploying, and managing applications. The pace and frequency of code updates have noticeably increased due to the high degree of automation we can achieve by using these modern cloud technologies. A strict application of DevSecOps principles is crucial to mitigate risks, apply updates on time, and assure compliance across our containers. Fortunately, there is over a decade of innovation and collaboration in the container space with several great solutions (which we will cover in this article).
When it comes to attacks on our supply chains, we should buckle up. According to Gartner®, "Attacks on the software supply chain, including both proprietary and commercial code, pose significant security, regulatory and operational risks to organizations. A source estimates costs from these attacks will rise from $46 billion in 2023 to $138 billion by 2031."
Fortunately, image mode brings the best of RHEL together with the best of the cloud-native solutions:
- Image mode is RHEL, and hence, benefits from over three decades of experience and security hardening of the operating system.
- Image mode applies the best practices and tooling of the cloud-native world to the entire software stack of RHEL. Imagine container security scans on the Linux kernel, firmware, and drivers.
- Image mode enables a unified approach for DevOps and DevSecOps. It lets us manage the entire OS through container-based tooling and concepts, including GitOps and CI/CD pipelines.
In other words, image mode for RHEL bridges the historic gap between the Linux host and application containers running on top of it. By applying DevSecOps practices to our image mode OS, we are well-prepared for the security challenges ahead of us. To learn more, refer to the Red Hat Trusted Software Supply Chain page and watch the following video.
CI/CD pipelines and automation
Combining continuous integration and continuous delivery (CI/CD) in one automated pipeline is the heart of modern software development. The idea is to bridge the gap between development and operations by automating several tasks, such as building, testing, deploying, and releasing software in one connected pipeline.
Automation is crucial in the context of security. We want updates to be published and consumed on time. We want security checks and rigorous testing to be applied to any code or dependency change. We want to reduce human error as much as possible. We need to trust our pipelines.
In our previous article How to create CI/CD pipelines for image mode for RHEL, we elaborated on a CI/CD workflow for image mode for RHEL. Here are a few key takeaways from the article:
- Automation is key: When managing our base images in the form of
Containerfiles, we can use elaborate dependency-management tools to automatically update our build dependencies (e.g., on other container images) and provide information about potential security issues. - Robustness with image digests: Referencing base container images by digests provides reproducible builds, improved security, and full control over changes. It also enables an easier root-cause analysis (e.g., by using git-bisect) of issues and potential rollbacks.
- Reliability of versioning schemes: Versioning our container images will determine how to apply automatic updates of application and bootc container images. Refer to the article, How to name, version, and reference container images, for more information about image versioning and tagging.
Once a pipeline has been created, we can also use it to automate security scanning and reporting.
Vulnerability scanning and reporting
Vulnerability scanning is all about detecting and mitigating vulnerabilities by proactively safeguarding an application and securing its confidentiality, integrity, and availability. That includes scanning for known vulnerabilities by comparing the versions of installed software packages with a Common Vulnerabilities and Exposures (CVE) database. Security scanners can also aid in detecting human errors, such as misconfigurations, mistakenly opened ports, or insecure firewall rules.
There are several certified scanners available for scanning Red Hat Enterprise Linux and Red Hat containers. Clair is a security scanner offered by Red Hat used on the Quay container registry to display vulnerability reports for individual container images. The Red Hat Container Catalog also displays the security rating of container images, such as the Universal Base Image. Security scanning and reporting is consistently applied to all parts of our supply chain.
The great thing about containers is that they are shipped with their workload. That means that we generally install at build time, not run time. Hence, security scanning can be applied once to a given container image. Once we know a given image is insecure, we can easily figure out which containers are impacted by inspecting the used image. As outlined in a previous article, How to use new container events and auditing features in Podman, you can configure Podman to emit events that allow us to trace back which container image has been in use and how it is configured (e.g., volumes, network settings, mounts, etc.).
Image mode offers essentially the same user experience as containers. Updates to the host are shipped in the form of container images. So we can easily detect vulnerable hosts by looking at the security report of the image currently in use. For instance, you can do this by running the bootc-status command and looking at the digest of the used image. The other way around, if we want to figure out if a given vulnerable image is in use, we can easily compare its digest with those in use on our RHEL systems. All of that happens in a fully automated fashion.
Container image signing
Signing container images is the process of adding a digital signature to ensure the authenticity and integrity of the container image. It is a powerful means to ensure the image has not been tampered with by a third party and that it comes from the author we expect. In the case of Red Hat images, the author would be Red Hat.
There are several ways to sign container images, but the most common ones include using a private/public key pair or using a trusted certificate authority. Going into the technical details of the different signing mechanisms is out of the scope of this article, so please refer to the Red Hat documentation.
It is more valuable to highlight these security benefits of signing container images:
- We can trust signed container images. Once verified, we know the origin of the image (e.g., it was built by Red Hat) and that it ships the original contents.
- As outlined in the previously mentioned Podman article, names and tags of container images are not a means to identify a single image. For that, we should use digests. The signature of an image extends that identity.
- We not only benefit from signed images as consumers but can also sign them ourselves and let our customers enjoy the same benefits.
- Signing is easy. Using Podman with Sigstore is simple and portable. Signatures are stored alongside the image on the registry.
- You can automate and integrate signing into a CI/CD pipeline.
Security has always been like an onion where we wrap one layer of security into another. The same applies to signing container images. Using the Red Hat Universal Base Image, the installed software packages and their dependencies are signed along with the entire container image. We can use that base image, build our custom image, and ship it to customers with our signature. It is a lovely example of a trusted and secure supply chain, isn’t it?
An immutable OS increases security
An important attribute of image mode for RHEL is immutability. An immutable operating system follows a different paradigm than traditional package-based systems. Once deployed, the entire filesystem is mounted read-only, except for /etc and /var. This means that not even the root user has write privileges. You can apply updates to the system by downloading a new version of the bootc image from a container registry and rebooting into the new state. It's a different way of approaching updates than using a package manager to update the system at runtime. It forces you to be intentional about changes to the operating system and gives you full control while increasing the security of the system.
In the previously listed article on managing workloads, we argued that the new paradigm of image mode approaches Linux systems with a fresh pair of eyes. We strongly believe that most workloads should not run directly on the host but as application containers on top of the host. This allows a clear separation of concerns between providing and managing the resources of the system (host) and the workload and application logic (container). It further prevents polluting the host system with continuously changing applications. From a technical perspective, we also prevent rebooting the entire image mode system just for application updates.
An immutable OS in combination with containerized applications creates a strong line of defense as well. In the extremely rare case of a successful attack on a container in combination with an even rarer chance of bypassing SELinux, there is reduced potential for damage. The system is immutable, and most parts of the filesystem are read-only. This kind of architecture has been used in cloud computing for over a decade. Thanks to image mode for RHEL, we can now apply it to the entire Linux enterprise space.
Zero-drift infrastructure
One of the more elusive challenges in enterprise IT is the notion of systems drifting from their desired state. In practice, there can be many causes for this, ranging from manual changes to scripting mistakes. The challenge here isn’t merely the detection of deviations, but the cumulative effects that compound over the life of a system and the magnification across an enterprise environment. While many tools exist to help combat drift, including Red Hat Ansible Automation Platform, an ideal posture is to remove or reduce the ability to change the system at run time, thus reducing the surface area where drift can occur.
Let’s consider troubleshooting, a scenario that commonly leads to drift and how image mode directly helps. When determining an issue of a system, a common approach is to install additional software to help analyze the system and change the configuration to enable more verbose logging. Image mode offers several tools to help ensure the system returns to the proper and desired state after identifying the issue:
Users with administrative privileges can enable a non-persistent overlay on the system to allow the installation of RPMs. None of these will survive a reboot, thus providing just enough non-immutability to assist with the issue and a surefire means to restore the correct state of the system with a reboot.
We also provide toolbox, a utility that enables privileged containers to diagnose system issues. You can find more details in our Red Hat blog post, Debugging image mode hosts.
We can restore config files in
/etc, which is one of the writeable areas of the filesystem. Now we can restore the original state of files by simply copying the original from the image stored in/usr/etc. On an image mode system, it’s not necessary to copy the original file to[filename].origand make edits because it already exists in/usr/etcas a read-only object. This is a powerful addition which can make check summing files in/etcfast and easy, ensuring alteration of only the desired, machine-specific files.
The immutable state of image mode for RHEL largely sets the foundation for zero-drift systems. Just as we looked at the cumulative effects of drift, the inverse is true. Following our guidance and connecting your software supply chain and CI/CD infrastructure to operating system deployments means that we can remove the historic opportunities for drift to occur.
In other words, the discipline of making changes at build time rather than run time accrue across operational and security domains. Image mode largely solves patching issues, discourages manual changes, and provides administrators with new tools for troubleshooting that ensure the intended state of the system isn’t altered.
Next steps
We focused on security in this article. Image mode for RHEL combines the best of both worlds: the trusted software supply chain of Red Hat Enterprise Linux and the high degree of automated security tooling of the cloud-native ecosystem. We also discussed how image mode simplifies building, deploying, and managing RHEL.
Hungry for more? Explore an interactive lab at red.ht/imagemode and get started with image mode for RHEL.
[Source: Gartner, Leader’s Guide to Software Supply Chain Security, Dale Gardner, Manjunath Bhat, 20 June 2024. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.]
