Welcome back to our blog series about the evolution of mass-prebuild. In our first post in 2022, we introduced this powerful open source tool designed for streamlining package builds across multiple architectures and Linux distributions. This article continues that journey, exploring how mass-prebuild has been continually refined, improved, and grown in response to user feedback and technological advancements. We will dive into specific updates that have shaped the tool over time, highlight its significant role in improving the stability of essential packages like Autoconf and GCC, and showcase its impact on the open source community. Fasten your seat belts for a comprehensive look at mass-prebuild's progress and future prospects!
Recap: What is mass-prebuild?
Need a summary of the previous blog? OK, here is a short version. The Mass Prebuilder (MPB) is a set of tools aimed to help the user to create mass rebuilds around a limited set of packages, in order to assess the stability of a given update.
The way it works is fairly simple. Once you have it installed through its COPR repository, you can create a simple configuration file as follows:
$> cat > my_config << EOF
packages:
glibc:
src_type: file
src: glibc-2.40.9000-16.fc41.src.rpm
EOF
And then execute `mpb -c my_config`; this will automatically rebuild the corresponding component in a dedicated COPR project, calculate the list of packages that depend on this component, and rebuild them against the new build. If they fail, these failed packages are rebuilt against the version of the component that was already available in the distribution, and give a report about failures that are likely due to the new version that way. By default, this will happen for x86_64, on Fedora Rawhide, unless specified otherwise.
A long version of this is available in the Git repository.
Navigating the path to stability: The early days of mass-prebuild
August–November 2022
In this section, we delve into mass-prebuild's initial journey, from its first release as a more stable alpha version to the improvements that paved the way for further refinements. Despite lingering bugs and heavy development, the tool quickly gained essential functionalities such as enabling GitLab CI support, handling reverse dependencies, and refining error tracking. As the project progressed, developers focused on addressing user issues and improving functionality. Reverse dependency calculations were enhanced to accommodate virtual packages and runtime dependencies, while support for non-public COPR and Koji instances was introduced. The introduction of scripts aiding in user-specific repositories configurations and handling failed package lists marked significant strides toward enhancing the overall user experience during this phase.
It is now possible to provide sub-packages in the configuration. The following configuration would rebuild packages that depend on the gcc-gnat package only, instead of all the packages that depend on GCC:
packages:
gcc-gnat:
committish: '@last_build'
The `@last_build` ensures that the package is rebuilt from a known state. The mass-prebuild tool will ask Koji what is the commit hash of the latest known buildable version of the package for a given Fedora release.
It is also possible to provide groups. The following configuration would rebuild the packages listed in the `anaconda-tools` group:
packages:
'@anaconda-tools':
committish: '@last_build'
The end of this early period was marked by the release of GCC13 in Fedora and initial attempts from the community to release Autoconf 2.72, both of which were crucial for testing mass-prebuild's capabilities. As the tool played a vital role in assessing these components' stability, it paved the way for new features that would greatly enhance its potential.
That led to the support for COPR specific configurations. If need be, one may specify another COPR repository where some packages may already have been built:
packages:
gcc:
deps_only: True
copr:
additional_repos:
- copr://fberat/gcc-14_smoketest
With such a configuration, instead of wasting time in rebuilding GCC, the one already available in the `gcc-14_smoketest` project would be used. That is also in this period that the tool moved from a personal GitLab repository to its new location: https://gitlab.com/fedora/packager-tools/mass-prebuild
Expanding horizons, accelerating growth: Innovations and upgrades
December 2022–March 2023
In this phase of development, mass-prebuild underwent significant changes with an emphasis on improving user experience. Notable additions included dynamic retry on failure, package rebuild capabilities, support for generating man-pages from AsciiDoc files, a report utility, and autocomplete in Bash/Tsh environments. The v1.0.0 release marked a milestone as most major features originally required had been implemented, solidifying mass-prebuild's place as an essential tool for streamlining package builds across multiple architectures on Fedora and RHEL distributions.
There may be some cases where you know that the build order of some packages is important for them to succeed, but don't know what is the exact order that is needed. That's when the dynamic retry feature comes handy:
packages:
gcc:
deps_only: True
retry: dynamic
This will build the packages that depend on GCC, and then automatically rebuild all failed packages, until there is no more successful build twice in a row. That way, if some failures were due to some package needing other packages in the list to be rebuilt, they wouldn't be marked as failed unless they really could not be built.
Bug fixes, enhancements, and new features: The continued evolution
July–December 2023
In this chapter of development, the focus was on refining and strengthening mass-prebuild. From v1.1.0 to v1.3.1, updates included support for the `copr.ownername` key in configuration files, eliminating duplicates from package groups, setting checker default values, storing full paths in the `src` directory, new tests for the mock backend, and continuous integration speedups for CLI tests. Notable releases during this time included substantial additions such as updates to the mock backend, remote mock backend, and daemonize option, which enabled users to build packages more efficiently and with greater control over their build environments. Although these updates mainly consisted of bug fixes, minor improvements, and tweaks, they all contributed significantly to maintaining a high level of performance and reliability in the mass-prebuild project.
If thousands of packages depend on your component, you may not want to rebuild all of them to have a first glance on whether an update would be stable enough. In this case, rebuilding a random sample of packages, in addition to a specific list of critical packages may be what you need. Here is an example with a sample of 500 packages:
packages:
gcc:
deps_only: True
revdeps:
sample: 500
append:
rpm:
committish: '@last_build'
mock:
committish: '@last_build'
kernel:
committish: '@last_build'
firefox:
committish: '@last_build'
anaconda:
committish: '@last_build'
copr:
additional_repos:
- copr://fberat/gcc-14_smoketest
A milestone with GCC 14: Uncovering critical issues through comprehensive rebuilds
As a recent milestone in mass-prebuild's journey it was used with GCC 14 to assess the compiler’s performance and stability across multiple platforms. This testing process took place during the integration of GCC 14 into Fedora 40, which began in December and continued through January as the team prepared for the Fedora 40 mass rebuild.
By rebuilding a random subset of 500 and 1,000 packages initially, developers were able to identify any critical failures that needed immediate attention. Once these initial tests proved successful, all 9,000 packages were rebuilt on all architectures, revealing issues in both GCC pre-release versions like the aarch64 critical failures, as well as problems in around 300 components. As a result of this rigorous testing process, GCC 14 was improved ahead of its integration into Fedora 40, which ultimately contributed to enhancing the stability and quality of the release.
Future directions
Throughout its remarkable evolution, mass-prebuild has become an indispensable tool to assess the stability of a given component update. By incorporating numerous improvements, updates, bug fixes, and new features, it has solidified its position as a vital resource within the open-source community. Yet, there will be some improvement coming for the tool. An example of this is the backends. The mass-prebuild currently supports COPR and mock for the builds, support for Koji is still not yet integrated and will be part of a future release.
Join us on this ongoing journey of progress by using mass-prebuild for your own package build requirements or contributing to its development as part of our vibrant open source community. Together, we can continue shaping and improving this powerful tool while fostering collaboration and innovation within the wider ecosystem.
