So sadly the big news this week is the outage of two of our servers for needed upgrades. The status of the services that are down can be read here. Our sysadmins have been hard at work, already pushing 30 patches to our infrastructure and working on porting the PostgreSQL database used by most services to a new server.
The first priority is to get the mirrorlist service running again quickly as it has the largest impact on users. Our Bugzilla, which will not only be moved to a machine running Mageia 5 but will also be upgraded to Bugzilla 5, will be the next step. The homepage has already been moved to a server that is fully up to date and had no downtime.
Note that the mirrorlist and Bugzilla were restored shortly after publishing, thanks to the sysadmins for finishing the upgrades so quickly.
While the updates on the two servers we preventively took down were severely needed for security reasons, they have been in planning for a significant time, and require non-trivial development work to port scripts, templates and configurations to the updated Mageia 5 ecosystem.
The outage only affected two of our many servers. All of our other servers, including the build nodes, for i586, x86_64 and arm, which I struggled to count from memory, 6 arm and 3 x86 at last count – I swear the arm nodes are multiplying, ISO builder and the repository server have remained active and fully up to date on Mageia 5, busily building for Cauldron and Mageia 5, you can see the queue here. Nevertheless, it must be noted that the two servers which had to be taken down were the most exposed as they hosted most web-facing services (and since they ran outdated software, also the most vulnerable).
We can only apologise that these upgrades have not happened sooner, even if the impact on the development has been minimal, we will have to continue to make changes in our sysadmin procedures to ensure that upgrades are simpler and timelier in the future.
Once the services migration is complete, we will publish a blog post to give a better overview of what the components of our infrastructure are, what software they run, how the sysadmin team maintains it and the evolution in our sysadmin team that has been happening to for some time now.
ISO building and testing for Mageia 6 RC has been making good progress. The release blockers are being fixed nicely, a new drakxtools included on the latest ISOs fixed a number of partitioning bugs so the ISOs are starting to become satisfactory. The Live ISOs are now ready for wider QA testing, and an EFI issue that was holding up the 64-bit Classical ISOs has hopefully been fixed.
There were updates to many packages, the highlights of which were:
- drakconf 13.15 aka Mageia Control Center – fixes 3 bugs, improved message for missing packages (mga#20614), dropped legacy loaders (mga#18572) and improved Gtk+ animations (mga#19827)
- webkit2 2.16.1 fixes several crashes and rendering issue, numerous CVEs
- darktable 2.2.4 adds a few new features, more supported cameras and lots of bugfixes
- mate 1.18.1 fixes multiple memory leaks as well as improving support for status-notifier
- dnf 2.2.0
- enlightenment 0.21.7
- kernel 4.9.20 added firmware for Intel 6030 wifi cards and added more Polaris 12 PCI IDs.
There were updates to a number of packages for Mageia 5 this week, including:
- webkit 2 2.16.1 fixes several crashes and rendering issue, numerous CVEs
- python-django 1.18.16 with numerous CVE fixes
- nvidia-current 375.39 with CVE fixes and new GPU support
- phpmyadmin 220.127.116.11 with numerous CVE fixes
- wget 1.15 fixes various CVEs
These updates are going through validation, so will be pushed to a mirror once that process is completed. The Bugzilla downtime makes the QA team’s work slightly harder, but they are keeping testing via their mailing list, so updates should keep coming as usual 🙂