This article is for people interested in long-term Linux kernel maintenance. It introduces you to tools that can help keep the binary interfaces between the kernel and its loadable modules stable during the entire lifetime of a supposedly stable kernel, while the code is modified. As these tools are essentially analysis tools, they can be used not only by kernel developers, but also by quality assurance engineers and advanced kernel users (system programmers).
Upstream in-tree kernel modules: the ideal situation
In the canonical development model of the Linux kernel, the source code of all dynamically loaded modules is hosted alongside the source code of the core kernel. In this model, whenever the core kernel changes the interface it exposes to its modules, the compilers detects that the interface changed, making it easy to adjust the code of the modules accordingly.
This model is referred to as the in-tree kernel module model and the binary interface between the kernel and its module as the kernel module interface, or KMI. In the in-tree kernel module development model, nothing needs to be done when the KMI changes. Adaptation of the source code of the modules happens naturally. I believe this is the best model to follow. Source code of kernel modules should just be upstream.
Out-of-tree kernel modules: life can be tough sometimes
Source code of kernel modules should just be upstream. I like that sentence. And it would be great if real life just complied all the time.
But it doesn't.
There are cases where some people might develop a kernel module without posting its source code upstream, at least for a certain period of time. I call this situation the out-of-tree kernel module model.
In this model, suppose the kernel module was initially developed against the interface exposed by an upstream kernel with a version number of 3.10. When the upstream kernel moves to version 3.11, the authors of the module might want to know if the binary KMI the module was developed against changed. And if it did, what the changes were. With that knowledge, module authors can assess the need to re-compile their module to make it compatible with the new 3.11 kernel.
Likewise, a Linux distributor might want to ensure that updates to the stable kernel they ship does not modify the initial KMI in an incompatible way. To that end, tools that help visualize and review changes to that binary KMI can be really useful.
In the end, users of that stable kernel might have developed private, non-distributed modules for their own use. Chances are that would they greatly appreciate that updates to that stable kernel don't break their private modules.
Visualizing KMI changes
This article shows you how to visualize and analyze the changes in the KMI between two binary kernel packages available in a Linux distribution, namely CentOS.
In practice, a set of functions and global variables (as well as their types) define the binary KMI to be compared across two binary kernel packages.
Note that Red Hat defines the binary KMI in a text file called the "kabi whitelist file". You can find it at /lib/modules/kabi-rhel70/kabi_whitelist_x86_64 if you are on a RHEL or CentOS 7 system sporting an x86_64 machine. On those systems, that file is contained in an RPM named kernel-abi-whitelists.
To perform the actual KMI comparison, this example uses the abipkgdiff tool, which is based on the Libabigail framework. This framework has been recently improved to analyze Linux kernel ELF binaries. As a result, the abipkgdiff tool has been improved to support Linux kernel packages in the RPM format.
Setting up your environment for KMI change analysis
Getting Libabigail tools
This example assumes you are either on CentOS 6 or 7, RHEL 6 or 7, or a supported version of the Fedora Linux distribution.
To install Libabigail tools on CentOS or RHEL you need to type:
$ yum install libabigail
On a supported Fedora distribution you need to type:
$ dnf install libabigail
Getting the kernel packages to compare
abipkgdiff needs the two kernel packages to compare the KMI, as well as some ancillary packages; namely, the associated debug info and KMI definition packages.
Here's how to compare the binary KMIs of the kernels of CentOS 7.0 and CentOS 7.1.
First get the 7.0 kernel packages:
$ mkdir 7.0
$ pushd .
$ cd 7.0
$ wget http://vault.centos.org/7.0.1406/os/x86_64/Packages/kernel-3.10.0-123.el7.x86_64.rpm \
http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-3.10.0-123.el7.x86_64.rpm \
http://vault.centos.org/7.0.1406/os/x86_64/Packages/kernel-abi-whitelists-3.10.0-123.el7.noarch.rpm
$ popd
Next, get the 7.1 kernel packages:
$ mkdir 7.1
$ pushd .
$ cd 7.1
$ wget http://vault.centos.org/7.1.1503/os/x86_64/Packages/kernel-3.10.0-229.el7.x86_64.rpm \
http://vault.centos.org/7.1.1503/os/x86_64/Packages/kernel-abi-whitelists-3.10.0-229.el7.noarch.rpm \
http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-3.10.0-229.el7.x86_64.rpm
$ popd
Comparing the KMIs
Now compare the KMI of those two kernels and save it into a file conveniently name kernel.kmidiff.txt. To do so, you need to type the following command:
$ abipkgdiff --d1 7.0/kernel-debuginfo-3.10.0-123.el7.x86_64.rpm \
--d2 7.1/kernel-debuginfo-3.10.0-229.el7.x86_64.rpm \
--wp 7.1/kernel-abi-whitelists-3.10.0-229.el7.noarch.rpm \
7.0/kernel-3.10.0-123.el7.x86_64.rpm \
7.1/kernel-3.10.0-229.el7.x86_64.rpm \
> kernel.kmidiff.txt
This command takes a little while as it needs to unpack the two kernel packages, analyze the two vmlinux binaries as well as the thousands of associated binary modules, compare the interfaces, and produce a meaningful report. For the record, on my machine, it takes around two minutes. Your mileage might vary.
Analyzing the results
For good measure and for the sake of keeping this article concise enough, you can get the resulting text file report of the comparison by clicking here. Let's go through interesting snippets of that report and analyze them.
For a start, here's the summary of the report:
== Kernel ABI changes between packages '7.0/kernel-3.10.0-123.el7.x86_64.rpm' and '7.1/kernel-3.10.0-229.el7.x86_64.rpm' are: === Leaf changes summary: 38 artifacts changed (16 filtered out) Added/removed functions summary: 0 Removed, 0 Added functions Added/removed variables summary: 0 Removed, 0 Added variable
So, the 38 artifact changes are types that changed. The types are reachable from the type of the functions and global variables that make up the KMI, as defined.
The artifacts could have been functions or variables as well, but then we see that no functions or variables were removed or added. We see that 16 of those artifact changes have been filtered out, presumably because the tool deemed those changes not meaningful from a KMI perspective. There are options in the tool that let you see all of the changes, unfiltered. You can learn about those options in the online manual.
Here are some of the type changes reported:
'struct netns_nftables at nftables.h:8:1' changed: type size hasn't changed 1 data member insertion: 'unsigned int netns_nftables::base_seq', at offset 608 (in bits) at nftables.h:19:1 there are data member changes: 'unsigned long int netns_nftables::__rht_reserved1' size changed from 64 to 32 (in bits) (by -32 bits)
You can see that a data member named 'base_seq', of type int (32 bits) was added to the struct netns_nftables. At the same time, the size of a data member named __rht_reserved1 changed and was reduced by 32 bits. As a result, the report says that the size of struct netns_nftables did not change. So presumably, code that uses instances of type and that expect its size to stay constant, will keep working.
Now, lets look at another kind of reported change:
'struct swap_info_struct at swap.h:182:1' changed: type size changed from 1152 to 1792 bits 2 data member insertions: 'plist_node swap_info_struct::list', at offset 1152 (in bits) at swap.h:225:1 'plist_node swap_info_struct::avail_list', at offset 1472 (in bits) at swap.h:226:1
Notice that the size of the struct changed. It was 1152 bit before, and it's now 1792 bits. And it grew because two new data members were added at its end - starting at offset 1152.
After reviewing this change, it would be fair to think that the kernel developers thought that no code using this struct was expecting its size to stay at 1152. Note that no offset of any existing data member changed, precisely because the data members were added at the end of the struct.
And there is also another interesting kind of change:
struct cfg80211_connect_params at cfg80211.h:1587:1' changed: type size changed from 1728 to 1856 bits 2 data member insertions: 'ieee80211_channel* cfg80211_connect_params::channel_hint', at offset 64 (in bits) at cfg80211.h:1775:1 'const u8* cfg80211_connect_params::bssid_hint', at offset 192 (in bits) at cfg80211.h:1777:1 there are data member changes: 'u8* cfg80211_connect_params::bssid' offset changed from 64 to 128 (in bits) (by +64 bits) 'u8* cfg80211_connect_params::ssid' offset changed from 128 to 256 (in bits) (by +128 bits) 'size_t cfg80211_connect_params::ssid_len' offset changed from 192 to 320 (in bits) (by +128 bits) ... ...
In this change report (trimmed for the sake of simplicity), the size of this struct cfg80211_connect type changed because two data members were inserted in the middle of it. That caused the offsets of subsequent data members to change as well.
Clearly, this struct, even though reachable from the functions and variables defined in the kabi whitelist file at /lib/modules/kabi-rhel70/kabi_whitelist_x86_64, is not meant to be part of the KMI.
Said otherwise, any code that uses this structure is meant to be (re)compiled together with the file cfg80211.h. And, presumably, this change has been reviewed and accepted by kernel developers knowingly, as it doesn't negatively impact the binary KMI stability in practice.
So we see here that this libabigail-based tool doesn't do away with the need for doing a peer review of the binary KMI changes. Rather, it provides a hopefully useful basis for that review to be productive.
It also shows how some tailored automatic KMI stability verification tools could be built using the libaigail library.
Possible improvements
This Linux kernel support is a new development. As a result, there is still room for improvement.
Filtering non-public types changes out of the report
As shown above, some types are reachable from the set of functions and global variables that make up the binary KMI, but they are meant to be private to the module (or vmlinux binary) where they are defined.
Users willing to remove those types can define a suppression specification file in which they basically define a blacklist of types that changes should not be reported by the tool. That suppression specification can then be provided to abipkgdiff by using the --suppressions option.
Details of what type is put in that blacklist is a matter of defining a particular review process in which things are thoroughly discussed and weighted.
Filtering out changes that don't impact any size or offset
Whenever a type change doesn't impact the offset of any data members or the size of the structure, we could probably just avoid showing that change, by default.
This particular filtering feature hasn't been implemented yet, but I believe it could be an interesting one.
Supporting other packaging formats
At the moment, abipkgdiff supports RPMs and DEB package formats in general. Note that support for the DEB format was added by the Debian maintainer of the GCC package. Ah, the beauty of free and open source software. OK, let's not digress too much.
But then, for this particular Linux kernel analysis feature, abipkgdiff only supports the RPM format. Even for RPMs, it only supports the particular file layout used by the RHEL and Fedora kernels. We would happily accept patches adding support for kernels of other distributions.
Performance
Performance of the libabigail library is acceptable today, but we are always looking for faster (yet robust) ways to do things, especially now that it must analyze massive sets of binaries like the Linux kernel and its modules. This is an area where we plan to improve things as well.
Conclusion
The recent addition of Linux kernel binaries support in libabigail is providing a new set of analyzing features to the set of tools that it powers; abipkgdiff is one example of these tools that can now analyze Linux kernel packages.
There is also a new tool (named kmidiff) aimed at comparing KMIs from source trees as opposed to RPM packages. Developers compiling their own source tree might use kmidiff rather than abipkgdiff.
As already said above, this is all new development and we expect to improve things even further in terms of performance, filtering, and ease of use in general.
Last updated: May 26, 2022