CentOS 7 enters end-of-life June 2024, which gives a little more than a year to migrate out accounts or perform an in-place OS upgrade using Leapp and Bootstrapper.

Bootstrapper forms the self-healing/self-configuration component of ApisCP while Leapp is a tool from Red Hat that performs an OS upgrade in a RAM disk at boot. That is to say, Leapp downloads required packages, restarts the server, then before it boots into multi-user mode, replaces all system packages with the new OS. This solves a historical problem of overwriting glibc or OpenSSL libraries during a package update marathon.

This guide covers experiences migrating a CentOS 7 machine to Rocky Linux with notes about migrating to AlmaLinux. Both distributions are bug-for-bug copies of Red Hat Enterprise Linux 8 and therefore would be equivalent upgrade experiences. If anything goes swimmingly, upgrades should take ~1 hour.

⚠️
Before getting started it's strongly advised to have a VNC, KVM, DRAC, SOL, or another out-of-band console available. This will allow you to view upgrade progress when it's in a recovery shell as well as react to any challenges. During both updates, eth0 was not automatically renamed to ens3 following Consistent Network Device Naming in RHEL8.
SPICE client for KVM (virt-viewer) during upgrade.

In-place upgrades aren't for the faint of heart. First, should an issue arise, you're against the clock to fix it. Second, while I have covered issues during my upgrades there could always be other issues with your hardware combination. Server-to-server migrations are always recommended for thoroughness. Not only do you have time to explore the server setup but can easily pull back a botched migration by reverting the IP address. Updating an OS is a one-way ticket.

Download Leapp

yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el7.noarch.rpm
# Convert to AlmaLinux
yum install -y leapp-upgrade leapp-data-almalinux
# Or to convert to Rocky Linux
yum install -y leapp-upgrade leapp-data-rocky

Preupgrade check – 10 min

Leapp performs a system check to ensure everything can be replaced without incident. For items that require action, it uses an answer file in /var/log/leapp/answerfile. We'll need to confirm it's OK to remove pam_pkcs11 from the stack as it is removed in RHEL8.

# Perform pre-upgrade check
leapp preupgrade
# Acknowledge removal of pam_pkcs11
leapp answer --section remove_pam_pkcs11_module_check.confirm=True
Off we go!

Addressing issues

Precheck may fail one - or in this case - several times depending upon the accretions your system has developed over the years. Here's some from an upgrade to Rocky:

  • ModelViolationError: The value of "name" field is None, but this is not allowed
    Confirm by running yum list | grep 'missing name'

    In this case, remove invalid [pgdg13] section from /etc/yum.repos.d/pgdg-redhat-all.repo
  • Inhibitor: Newest installed kernel not in use
    Ensure the system kernel is installed, then reboot. --var=x=y sets a temporary setting that expires at the end of the run.

    upcp -sb --var=always_update_kernel=True --var=prefer_experimental_kernel=False system/kernel
  • Inhibitor: Missing required answers in the answer file
    Check /var/log/leapp/answerfile, in this case pam_pkcs11 required confirmation.

    leapp answer --section remove_pam_pkcs11_module_check.confirm=True
  • RPM transaction errors if present
    file /usr/lib64/.libcrypto.so.1.1.1k.hmac from install of openssl-libs-1:1.1.1k-7.el8_6.x86_64 conflicts with file from package openssl11-libs-1:1.1.1k-4.el7.x86_64

    Remove supplemental openssl11 RPMS installed with nodejs dependency.
    (rpm -qa | grep openssl11 ; echo nodejs nodejs-libs v8-devel nodejs-full-i18n nodejs-devel npm) | xargs rpm -e

    file /etc/yum/pluginconf.d from install of yum-4.7.0-11.el8.noarch conflicts with file from package etckeeper-1.18.20-1.el7.noarch

    More of the same, this time etckeeper which will be reinstalled during the final scrub.
    rpm -e etckeeper
💡
Any RPMs removed to satisfy upgrade will be reinstalled during platform scrub.

Run leapp preupgrade each time this is resolved until it completes successfully.

Upgrade – 30 min

Now we can proceed with the actual upgrade tasks. This is where having a remote console comes in handy!

leapp upgrade
System primed to upgrade

After the upgrade completes, reboot for the package swap process.

reboot

The system will boot into a special kernel called ELevate-Upgrade-Initramfs.

Almost completed successfully but complete enough to reboot into Rocky 8 
  • Once the upgrade completes - and all goes according to plan - you'll be returned to multi-user mode running its shiny new OS. Let's cleanup any orphan packages from RHEL7 before moving onto a scrub.
rpm -qa | grep -E 'el7[.-]' | xargs rpm -e
⚠️
If your network device was named ethXX before, run ip addr list to look at the new device name. Rename /etc/sysconfig/network-scripts/ifcfg-ethXX to /etc/sysconfig/network-scripts/ifcfg-ensXX as well as update the device name within the file. RHEL8 names network devices differently in 8.

Scrubbing – 30 min

Now the hard part is over. Reinstall Ansible, remove a couple bad packages from AppStream, then run Bootstrapper to scrub your new AlmaLinux or Rocky Linux platform. Scrubbing will reapply installation, correcting any configuration drifts. This includes reinstalling missing packages, adjusting system configuration, synchronizing your filesystem, and rebuilding all PHP releases with system optimizations. yum-post.php is a helper that's run whenever a package updates to synchronize system packages into FILESYSTEMTEMPLATE.

Lastly, rebuild site configuration using EditDomain --reconfig, which does a more thorough rebuild than without.

# Reinstall Ansible + JSON parser
yum install ansible python3-jmespath
# Remove Apache RPMs from AppStream
rpm -e mod_http2 httpd mod_ssl --nodeps
# Scrub
upcp -sbf
# Update filesystem packages
/usr/local/apnscp/bin/scripts/yum-post.php resync --force
# Rebuild site configuration
EditDomain --reconfig --all

Congratulatory pat on the back – 5+ min

🤗
You've done it. Enjoy!

Lastly reinstall any RubyGems or Node packages that were compiled on the old OS. Often times these are linked against older library features that no longer exist.