When moving an application that you've compiled on Red Hat Enterprise Linux (RHEL) 7 to RHEL 8, you will likely encounter issues due to changes in the application binary interface (ABI). The ABI describes the low-level binary interface between an application and its operating environment. This interface requires tools such as compilers and linkers, as well as the produced runtime libraries and the operating system itself, to agree upon the following:
- The application's data type, size, and alignment.
- The calling conventions, which define how function arguments are passed and how return values are retrieved.
- The function and symbol names and their versions.
As the toolchain and operating environment evolve with successive versions of Red Hat Enterprise Linux, the details of the ABI will change. Red Hat makes certain guarantees regarding application compatibility between releases. Those guarantees are outlined in the RHEL 7 Application Compatibility Guide (ACG) and the RHEL 8 Application Compatibility Guide. This article is a companion to the ACG guides.
Migrating compatible C and C++ applications
The most straightforward way to avoid compatibility problems when migrating your C and C++ application code from RHEL 7 to RHEL 8 is to rebuild the code on RHEL 8. Users can leverage a containerized or virtualized environment to build and test in a RHEL 8 environment on a RHEL 7 system before migrating to the newer version.
In some cases, it might be possible to deploy C and C++ applications built on RHEL 7 to RHEL 8 without first rebuilding. If you've followed the guidance in the RHEL 7 ACG, and depend only on the C and C++ libraries in compatibility level 1 (CL1), then Red Hat provides a compatible version of those libraries in RHEL 8. Indeed, we maintain stable versions of those libraries for three major RHEL releases. Note, however, that there are no guarantees that RHEL 8 will provide RHEL 7-compatible libraries beyond CL1.
Migrating shared libraries
Unless a C application loads shared libraries via dlopen(), the ldd command shows what libraries a given application depends on (in the case of dlopen(), you can use LD_DEBUG=all my_application or the strace command to observe which libraries are being loaded). However, shared libraries for C++ applications are more complicated and restricted.
C++ ABI incompatibility between RHEL 7 and RHEL 8
C++ libraries from RHEL 7 are likely to be incompatible with the corresponding libraries from RHEL 8. The reason is that the system compiler for RHEL 8 uses a new ABI for C++ code, which is different from the ABI used by the RHEL 7 system compiler. The RHEL 7 version of the C++ libraries uses the old ABI, whereas the RHEL 8 version uses the new one. Applications linked to an RHEL 7 version of a C++ library cannot use the RHEL 8 version of that library. The only C++ libraries that are in ACG CL1 are the C++ standard library (libstdc++) and Thread Building Blocks libraries (libtbb and libtbbmalloc).
Changes in the C++ ABI between RHEL 7 and RHEL 8 are due to introducing new versions of std::string and std::list in the C++ standard library. For details about the changes to the std::string ABI, see Jason Merril's article, GCC5 and the C++11 ABI. The changes to the C++ ABI apply to all language modes, so it doesn't matter whether you are compiling for -std=c++11 or any other -std option.
Workarounds that do not work
If your application code depends on libraries not in ACG CL1, it might be tempting to simply copy those other binary dependencies over; however, this is neither supported nor likely to work. It might also be tempting to statically link all dependencies on RHEL 7 and deploy to RHEL 8. Statically linking glibc prevents some features from functioning, so this option is also unsupported and not recommended by Red Hat. It is possible to statically link to libstdc++, but doing that shouldn't be necessary. The libstdc++.so.6 that shipped with RHEL 8 is backward compatible. Statically linking to other C++ libraries could be a solution if the versions of those libraries on RHEL 8 are incompatible with the RHEL 7 versions.
Build guidance
Finally, the ACG provides a recommended approach to building applications on RHEL 7 that will run on RHEL 8. This essentially means building your application against your own version(s) of library dependencies outside of those specifically listed in ACG CL1, rather than those provided by the base system. See the ACG for guidance on providing compatibility libraries for applications that have been built with libraries that are outside of the desired compatibility level.
Frequently asked questions (F.A.Q.)
Before closing, I'll answer the most pertinent questions that we've received since releasing RHEL 8.
Q: What does Red Hat recommend if I want to deploy an application on RHEL 8? Where should I build my application?
A: The most straightforward way to avoid compatibility problems is to build your application on RHEL 8.
Q: Can I build my application on RHEL 7 and deploy it on RHEL 8?
A: Yes, if you follow the guidance in the Application Compatibility Guide.
Whether a given application built on RHEL 7 can be deployed on RHEL 8 depends on the system libraries the application is linked with. For libraries within the RHEL 7 ACG's CL1, a compatible version of the library is provided in RHEL 8. For all other libraries, there is no guarantee that RHEL 8 provides a compatible version.
For C++ libraries, this advice is not hypothetical, as C++ libraries from RHEL 7 are likely to be incompatible with the corresponding libraries from RHEL 8. To build applications on RHEL 7 that will run on RHEL 8, the ACG suggests the following:
Provide compatibility libraries for applications that have been built with libraries that are not at the desired compatibility level.
In other words, build your application against your own version of the libraries, rather than the RHEL 7 versions. That way, your application depends on a library that you control, not one that belongs to RHEL, and will be incompatible between major releases.
Q: Why are C++ libraries built with the system compiler on RHEL 7 not compatible with RHEL 8?
A: The system compiler on RHEL 8 uses a new ABI for C++ code, which is different from the ABI used by the RHEL 7 system compiler. The RHEL 7 versions of C++ libraries use the old ABI, and the RHEL 8 versions use the new one. Applications linked to the RHEL 7 version of a C++ library will not be able to use the RHEL 8 version of that library.
Q: What has changed in the C++ compiler and libraries to cause ABI incompatibilities between RHEL 7 and 8?
A: Starting with RHEL 8, new versions of the std::string and std::list types are defined in the C++ standard library. As a result, C++ binaries (applications and libraries) built on RHEL 7 could fail to link with C++ binaries built on RHEL 8—or they might link successfully but fail at runtime.
Q: If I want to build a C++ application on RHEL 8, do I need to compile with -std=c++11?
A: No. The changes to the C++ ABI apply to all language modes, whether you compile with -std=c++11 or -std=c++98 or any other -std option.
Q: Can I fix library ABI compatibility problems by copying shared libraries from RHEL 7 over to RHEL 8?
A: Copying a RHEL 7 library to RHEL 8 might work, but Red Hat does not support it. There are downsides of copying RHEL 7 libraries to RHEL 8, such as the library not being automatically updated by yum/dnf when Red Hat issues an erratum for the package to fix bugs or security flaws.
Q: I built my application on RHEL 6, and it ran fine on RHEL 7. Will my application work on RHEL 8 as well?
A: It depends. If the application only uses libraries from CL1 in the RHEL 6 ACG, then it should work on RHEL 8 because those libraries are stable for three major releases (RHEL versions 6, 7, and 8). If the application uses RHEL 6 libraries that are outside of CL1, but which still happen to work on RHEL 7, it might also run on RHEL 8, but it might not. There was never a guarantee that such an application would run on RHEL 7.
Q: Which packages are guaranteed to be ABI compatible across major RHEL releases?
A: See the Application Compatibility Guides for each version of RHEL. Libraries in compatibility level 1 are guaranteed to be compatible. On RHEL 7, the only C++ libraries in CL1 are libstdc++ and the Thread Building Blocks (TBB) libs.
Q: Are there libraries that support both the old and new C++ ABIs?
A: Yes, libstdc++ does. The TBB libraries are not affected by the ABI change, so they are also compatible with both ABIs.
Q: How can I find out which libraries my application depends on?
A: The ldd command displays all the shared libraries that a binary depends on (including indirect dependencies from other libraries).
Q: Can I build an application for RHEL 8 (with the RHEL 8 system compiler) on RHEL 7?
A: While theoretically possible, doing so would be awkward and error-prone. If developing on RHEL 7 is a requirement for applications to be deployed on RHEL 8, Red Hat recommends running RHEL 8 in a container or virtual machine.
Q: If I link statically to glibc/libstdc++ on RHEL 7, will the resulting binary work fine on RHEL 8?
A: Red Hat does not support statically linking applications. Statically linking glibc prevents some features from functioning and is not recommended. Statically linking libstdc++ is possible, but it should not be necessary because the libstdc++.so.6 library on RHEL 8 is backward compatible. Statically linking to other C++ libraries could be a solution if the versions of those libraries on RHEL 8 are incompatible with the RHEL 7 versions.
