[Solved] Our other servers are down again

As you can see, our blogs are still up and running.

However, the servers hosting most of the rest of our infrastructure, like the forums, wiki and bugzilla, are not.

I assume we are under attack from bots again, but that needs to be confirmed by one of our sysadmins.

Anyway, the issue is known, we will try to update this message as soon as we know more.

1st update : a switch in the datacenter is broken, it should be replaced. A new one is on its way to Marseille. (2025-09-18, at 9:45 CEST)

2nd update: The outage is still ongoing, but we have set up a replica of the mirror list so that updates and new installations are no longer blocked (2025-09-24, at 8:00 CEST).

3rd update: After the mirrorlist replication, www.mageia.org was redirected to blog.mageia.org. (2025-09-24, 08.49 CEST)

4th update: Unfortunately, it appears that users using DNF instead of URPMI (the legacy tool in Mageia for updates) still cannot use the mirrorlist. The reason is, that DNF uses a different mirrorlist, created for downloading ISOs. That mirrorlist has not been replicated. As a workaround you can, in a terminal/konsole as root:

cp -a /etc/yum.repos.d/ /etc/yum.repos.d_original/
cd /etc/yum.repos.d/
sed -i 's/^mirrorlist/#mirrorlist/' *
sed -i 's/#baseurl/baseurl/' *

Doing that will set https://mirrors.kernel.org as your fixed mirror, so that you can install packages again. (2025-09-24 15:57 CEST) (Edit: add the forgotten -i switch to sed, 16:20 CEST)

5th update: We were notified that the issue at the datacenter will be fixed ultimately tomorrow (2025-09-25 11:50 CEST) (Edit: put the code above in a code block instead of in a list, 13:31CEST)

6th update: We received no update from the datacenter, but we are sure there is a good reason for that. Note that they have always hosted us for free (2025-09-27 10:50 CEST).

7th update: There was certainly a good reason, they had not yet received the switch. They now expect to receive it today (2025-09-30 12:53 CEST).

8th update: According to the datacenter people, the ordered switch is now expected to arrive at the datacenter tomorrow. It will be then installed and we hope to recover the servers back to normal Monday or Tuesday at the latest (2025-10-02 16:00 CEST).

9th update: the router has been delivered in the DC and the old one replaced, last week someone went in the DC to cable stuff, but there seems to be an issue with fibers now, so our servers are still not reachable. We have not yet information on when the next operation is scheduled in
order to fix that remaining issue (2025-10-07 9:00 CEST).

10th update: The servers are back! (2025-10-07 15:00 CEST)

This entry was posted in sysadmin and tagged , . Bookmark the permalink.

Curious about Mageia? Download it, give it a try and tell us how you feel about it.

Want to bring something to it? Learn how you can contribute and donate.

70 Responses to [Solved] Our other servers are down again

  1. filochard says:

    the problem concerns gmane.linux.mageia.devel too

  2. filochard says:

    This explains why there’s no message inside
    news.gmane.io for
    gmane.linux.mageia.bugs
    gmane.linux.mageia.changelog
    since 17/09/2025 09:57

    But that doesn’t explain why there are not any message inside
    gmane.linux.mageia.devel
    since 17/09/2025 06:59
    even if I I sent messages on 17/09/2025 10:30

    • Marja says:

      It does explain it, because gmane gives access to some of our mailing lists. It does not have Mageia mailing lists of its own.
      Our mailserver is hit by the outage, too, so there are no new mails that can be retrieved by gmane.

  3. Chuck Spaeth says:

    This is not a good situation at all. Obviously the datacenter lacks required redundancy. They should consider having a secondary datacenter site.

  4. Marc Paré says:

    I am just back from a 3 week trip and updating 7 of my Mageia laptop boxes. Just as an FYI, just in case this adds to your search for resolution:

    *I have updated 3 systems with the same results
    ** the red Mageia update ball in the bottom KDE lower-right-hand taskbar indicated an update was waiting
    ** I clicked on the update icon, the updates went through without a problem (If I remember correctly, the repository servers where I am getting my updates are those somewhere in FR)
    ** then, all of the awaiting updates were done successfully (it appears the updates were fetched from the repositories)
    ** then, as the usual final step, Mageia searched for any other updates, but could not reach any repositories

    My point being that, the last updates somehow managed to retrieve the updates on the repositories(???) and installed them. I am not sure if Mageia usually pre-fetches updates and readies them on Mageia boxes in order to save time, but could it be that something in the last updates could have salted/helped enable the ddos attacks somehow?

    I still have another 5’ish systems with the Mageia red icon in the taskbar signalling waiting updates. Let me know if you would like to test anything related to this.

    • marja says:

      Thanks for your feedback. I understand that you are not using the mirrorlist, but fetch your updates from one or more fixed mirrors. All our updates remain available from our mirrors, even if all of our own servers are down.
      The problem was for users depending on the mirrorlist on our website, that they could not reach that list. Luckily a duplicate mirrorlist is now activated and reacheable.

  5. Marc Paré says:

    Is all of this costing you extra unplanned expenses?If so, you could set up something on some site calling on all of we, Mageians, to send you some extra funds $$$ to help pay for any extra costs? We could then help with propagating the link to others to help out.

    Marc

    • papoteur says:

      Thanks for the proposition. We have still some funds for current expenses. We will call for donations in the future, but this this is not the today task.

    • Pascal Drabik says:

      Hi,
      I do believe that collecting some extra funds would be a good idea.
      Not only to cover unexpected expenses, but also to thank all the persons for their time in making Mageia work.

  6. Bostaurus says:

    Now, it is tuesday. Is there a chance, that Mageia comes. back?
    Aujourd’hui, nous sommes mardi. Y a-t-il une chance que Mageia revienne ?

  7. JanKusanagi says:

    Thanks for all your hard work on this!

  8. Gukin says:

    If you’re like me, and I know you are, you’d like to install a new package or perhaps build a new system but urpmi keeps failing on “Could not find a mirror from mirrorlist $MIRRORLIST”, you can work around this by running two commands from as root:

    first, clear out your /etc/urpmi/urpmi.cfg by running:
    urpmi.removemedia -a
    Then add in a specific mirror thusly:
    If you’re in the USA, this is a good one:
    urpmi.addmedia –distrib http://mirrors.kernel.org/mageia/distrib/9/x86_64/
    If you’re in Europe:
    urpmi.addmedia –distrib http://distrib-coffee.ipsl.jussieu.fr/pub/linux/Mageia/distrib/9/x86_64/
    Please note that the “argument after “urpmi.addmedia” has two dashes.

    This has been one tough, long outage, http://www.mageia.org really needs a backup site.

    • papoteur says:

      Yes, this was a workaround when the configured sources were with mirrorlist. We have connected a replacement for the mirrorlist, thus this step is no more needed.

  9. Serguei says:

    Sad news but THANK YOU anyway For your work

  10. Jamal says:

    I feel sad that Mageia’s website is down currently.

  11. speller822 says:

    It seems the switch hasn’t arrived yet, but those updates are good news, thank you.

    We can join the #mageia IRC channel, for example with the web client https://web.libera.chat/gamja

    Choose a nickname and when connected, type /join #mageia

  12. Germ says:

    I have my sources set to the princeton mirror in the US. I have not received any updates and none show on the server. So will switching to mirrorlist provide updates?

  13. JeffR says:

    I had to manually replace the single quotes (apostrophes). The copy and paste from the webpage changed them to a character that sed did not recognize

    • marja says:

      Yes, WordPress automatically turned them into a wrong type. I haven’t found time to find out how to make WordPress show the correct type. Edit: It is fixed now

  14. Pascal Drabik says:

    Dear Mageia Team,
    I am very sad to see mageia.org down…
    But, thanks a lot to all of you for the incredible work you are doing.

  15. Fabio says:

    Keep on strugglin’, guys!

  16. filochard says:

    Mageia has been disappearing from internet for 12 days
    It’s not so easy for many people to know where to get some information about the reason of this … if they don’t access to Mageia blog (fortunately remaining spared from the problem in the datacenter in Marseille).
    They may think that Mageia is dead
    Or at least that Mageia is not a trustworthy and solid distribution

    It’s very sad that such a long time is needed to repair the cause of this disappearance

  17. rh says:

    Has the data centre provided an update regarding the progress of the repair/replacement of equipment?

  18. filochard says:

    How is it possible : the supplier of the datacenter switch needs 2 weeks to send it to its client ? Isn’t it considered as an emergency ?

    • marja says:

      When I recently checked the webshop of a supplier of professional switches in my country, I saw that delivery times were up to two weeks for most, if not all of them. If it’s like that in France, too, then I’m not suprised.

      • trollie says:

        How about using less professional (or used) switch for 2 weeks? I know doing swap and configuration twice would be *going the extra mile*, but downtime would be noticeably shorter. (Humph… don’t they have a spare (old) switch at the datacenter just in case? Are these switches specialized that much, that swapping one for another would not work well?)
        And besides I really hope to see a photo- or videostory of the switch’s way to Marseilles – I’m buying popcorn today.

  19. filochard says:

    The delay proposed by the supplier depends on which supplier you ask and what stocks it has
    I found a German supplier that have stocks and that engages itself to deliver the switch free of charge with UPS
    But it’s true that the delay depends on which switch you order and on the place where are the stocks :
    – this can be 48 hours to deliver in France if the stock is in Germany…
    – but this can be 3 weeks if the stock is far from Germany (USA Singapour Australia…)

  20. trollie says:

    Ok, a switch went out of order and we’re all waiting for fix-up (ie. equipment replacement) – it happens.
    But what’s going on with the communication?
    1st update: “a switch in the datacenter is broken, it should be replaced. A new one is on its way to Marseille. (2025-09-18, at 9:45 CEST)”;
    5th update: “We were notified that the issue at the datacenter will be fixed ultimately tomorrow (2025-09-25 11:50 CEST)”;
    7th update: “There was certainly a good reason, they had not yet received the switch. They now expect to receive it today (2025-09-30 12:53 CEST).”
    and 8th update: “According to the datacenter people, the ordered switch is now expected to arrive at the datacenter tomorrow. It will be then installed and we hope to recover the servers back to normal Monday or Tuesday at the latest (2025-10-02 16:00 CEST).”.
    Is the switch hitchhiking?

    • marja says:

      Sometimes, anything that can go wrong does go wrong.
      The people at the datacenter do everything within their power, but they aren’t omnipotent.
      I’m sure you have experience with deliveries arriving too late, in the wrong place, with the wrong content etc.
      Or even being sent back to the sender without ever arriving at your place.
      Such things can happen, this world isn’t perfect.
      We aren’t perfect. In hindsight, it would have been better to have redundancy of our infrastructure at a different datacenter, so that we could have switched to those servers when this outage happened. But doing that looked like overkill and like wasting the money our users donate. Now it looks like a good thing to have.

      • trollie says:

        I’m sorry marja for putting You into this – my sole goal was to add some humour into otherwise unfunny situation. You seem a nice person, undoubtedly committed to Mageia.
        And yeah – some historians say that general Hindsight is the best commander 😉

      • trollie says:

        My opinion about preparedness for a half-month outage is: how about getting prepared for an asteroid strike? Both cases are comparably impeding to Mageia and comparably likely to happen. For a state or an entity that’s listed on the stock exchange, contingency plans are a must, for the others preparedness involves some tradeoff.
        Anyways I was not blaming Mageia for a single moment.

        • filochard says:

          In french we say : “la tartine tombe toujours du côté de la confiture”
          That may be translated into
          Toast always lands jam-side down.
          or
          Toast always lands butter-side down.

          When something wrong is appearing, it may be even worse….

        • trollie says:

          In case Mageia was about to increase preparedness, the expected value might help prioritizing/ranking the concerns (in the layman’s terms it’s a sum of products of benefits or costs of probable outcomes (or just events) by probability of these happening; calculating it just for comprehension of it may help one learn why gamblers lose in a infinitely long run). Taking into consideration the scarcity of resources, the project may set a bar, based on the expected value (nb. the *cost factor* would need to be based on both the expected loss and the expected prevention expense), that would help sifting out the conceivable issues the project can afford preventing.

  21. gukin says:

    How does one fix things when using mock?

    Replicate by doing this:

    mock –arch=i586 –init

    When mock tries to get to the mirrorlist it responds:

    Mageia 9 – i586 76 B/s | 428 B 00:05
    Errors during downloading metadata for repository ‘mageia’:
    – Status code: 503 for https://www.mageia.org/mirrorlist/?release=9&arch=i586&section=core&repo=release (IP: 163.172.148.228)
    Error: Failed to download metadata for repo ‘mageia’: Cannot prepare internal mirrorlist: Status code: 503 for https://www.mageia.org/mirrorlist/?release=9&arch=i586&section=core&repo=release (IP: 163.172.148.228)

    I tried the above sed/dnf command above on both /etc/yum/repos and /etc/mock/mageia*
    Mageia 9 – i586 76 B/s | 428 B 00:05
    Errors during downloading metadata for repository ‘mageia’:
    – Status code: 503 for https://www.mageia.org/mirrorlist/?release=9&arch=i586&section=core&repo=release (IP: 163.172.148.228)
    Error: Failed to download metadata for repo ‘mageia’: Cannot prepare internal mirrorlist: Status code: 503 for https://www.mageia.org/mirrorlist/?release=9&arch=i586&section=core&repo=release (IP: 163.172.148.228)

    • papoteur says:

      Hello,
      You can try the sed trick with optionally replacing mirror with the one of your choice in files /etc/mock/templates/mageia.branched.tpl for Mageia 9 and /etc/mock/templates/mageia.cauldron.tpl for Cauldron.

  22. Frederic Bezies says:

    This is totally unacceptable. Whoever is responsible for this should be fired immediately.

    Who is Mageia infrastructure manager? I want to speak to the manager!

    Please reach out to me ASAP. I am writing an article on the incident. My blog is read by thousands of readers every day.

    • marja says:

      Other than our predecessors Mandrake and Mandriva, we don’t have any employees. We only have volunteers. Apart from that, the broken switch isn’t ours. It belongs to a datacenter that hosts us and other non-profit organizations for free. We’re very thankful for the very many years of flawless hosting they donated to us. This unfortunate event doesn’t change our opinion of them.
      I’ll forward your request to the leader of our Sysadmin Team.

      • Bruno Pasquier says:

        It’s unfortunate that it’s taking so long to fix the issue. But I find the complain misplaced as noted in update 6: Note that they have always hosted us for free.

        They are proving the hosting for free so I can imaging they fix the issue as best effort.

    • Maât says:

      Hello Frederic,

      Marja did inform us about your request. I did send you a mail to get in touch.

      Best regards,

      • I did not post anything here. Somebody take my name and my email address to post such an horrible message.

        Maybe some troll wanting a “war” between me and your distribution.

        Have a good day.

        • Maât says:

          Thanks for your kind clarification Frederic.

          Wishing you the best.

        • Pascal Drabik says:

          Dear Frédéric,

          Thanks a lot really for the clarification, which actually makes sense.

          I really, deeply, apologize for my reaction to, who I thought was you.

          Then, my reaction is even stronger to the miserable coward who impersonated you.

          And thanks a lot again the Mageia Team and Community.

    • Pascal Drabik says:

      Dear Frederic,

      First of all, I do not speak on behalf of the Mageia Team, but only as a user of the Mageia distribution. So, please try to understand that my reply to you has nothing to do with Mageia Team and therefore you may not criticize them.

      What is totally unacceptable is having such a reaction, writing such a comment. Your behavior to my point of view looks like the behavior of a poor and unlucky teenager, desperately willing to get some attention. You should really be ashamed. Human Kind do not need people like you.

      When you will write your article on your blog, I beg you to be honest, which might not be an easy task for an influencer like yourself. But consider it as a challenge on the path of your own education.

      That being said, Mageia is the best distribution I know (and I did try several). I would even dare to claim it is one of the best distribution world-wide, unfortunately not yet sufficiently well known globally. This incident is of course unfortunate. But these kind of things do happen.

      As you have been reminded, this distribution is supported by volunteers, perhaps a concept that you do not know, but that is at the heart of the free software concept (perhaps one more concept that you do not understand yet).

      This means that these people spend their time on the distribution, I guess because they like it, but also for others to use it, privately or professionally, probably with the hope that these users will enjoy it.

      Your suggestion to “fire the responsible” belongs to barbarian time and clearly demonstrate your global ignorance. But I have to acknowledge that “evolution” is a very slow process.

      So let’s stick to some barbarian time, just a little bit. May we know how much do you pay for the right to use the product made by the Mageia Team? Yes, please, say it, and provide evidence for it! And, please, be honest and do not forget to write it explicitly in your “famous” article.

      Enough for the barbarian time Let’s come back now to a little bit more civilized time.

      If you are not happy with the Mageia distribution, you might siimply want to consider to use another product (a coward, but practical approach, since we are not yet in a fully civilized time).

      Perhaps a good alternative, could be to contribute to Mageia by providing some support to the Team. I guess financial support would be appreciated. But you could perhaps also provide ideas to prevent such issues in the future. This, of course, assume that you have ideas different from “firing people”, which is unfortunately not likely the case, considering your reaction.

      Finally, as I mentioned at the beginning of this reply, I did not write it on behalf of the Mageia Team, but only to express my personal revolt to your unacceptable behavior.

      On the other hand, I have to congratulate Marja and Maât who replied to you (but probably all the Mageia Team) for their very professional, productive and civilized reaction.

  23. Dieter says:

    A data center that doesn’t have failover for a switch isn’t a data center. At least it needs backup, if not failover.
    I think after a month of downtime, Mageia will be as good as dead.
    It doesn’t help that this “hosting” is free.
    No security updates are impossible these days.
    And the 9th update speaks volumes. You’re being taken for a ride.

  24. trollie says:

    A relief…

    I hope Mageia will be up-to-date before 14th to ingest crowds of newcomers that, at that time, will be leaving Windows 10 behind.

  25. Gukin says:

    I just saw the happy http://www.mageia.org screen, it was a long wait but Mageia is always worth the wait. I couldn’t get mock going for cauldron so I got to use the online installer which was kinda fun and taught me a new way to get a Cauldron build before the ISOs are ready.

    I’m awfully glad to see http://www.mageia.org back.

    • Maât says:

      We’re happy too… thank all for your support & kindness while waiting. _/\_

      Now comes the time to work on the weaknesses in the architecture and hosting to prevent this kind of incident from happening again in the future. We are already exploring options for complementary sponsored hosting to diversify both hosting locations and network providers. There will probably be a dedicated blog post when we have news to share.

  26. Joe stager says:

    Thank You to all the hard working volunteers and the data center Love Mageia

  27. Palin says:

    I would like to thank all mageia volunteers around the globe. You’re awesome.

  28. Pascal Drabik says:

    Hi,

    I would like to join the supportive people and congratulate the Mageia Team for their incredible work.

    I too was looking every day for “https://www.mageia.org/en/” and was so glad this morning to see again the site being up and running.

    This was indeed an unfortunate incident, but not that dramatic after all. I was always able to use my Mageia systems without any trouble. There exist very large enterprises who could not claim the same.
    I just noticed it because I was investigating a small issue involving the testing of various versions of the Linux kernel.

    Mageia Team, Thanks you very, very much for your work!

  29. Brian R. says:

    Hurray!! Thanks to all who worked on the recovery and to those who maintained the blog explaining the outage.

    Isn’t IT and systems fun! ;0)

    Glad to see the lights on again.

    Thank You,
    Brian

  30. Dmitriy B says:

    Big thanks to the developers and everyone who supports this distribution. Mageia is with us, I’m really hoping for and looking forward to its continued development!

  31. Bernd says:

    Obviously, the problem is still not fixed completely.
    During normal installation from Mageia 9 boot media, the mirrorlist cannot be downloaded due to the following error:
    Die Mageia Webseite kann nicht erreicht werden, um die Liste der verfügbaren Spiegelserver zu erhalten.
    mirror list not found at /usr/lib/libDrakX/mirror.pm line 46

    This has never happened before and yes, network is up and running.

    • marja says:

      Thanks for reporting this, but can you please file a bug report or discuss this in the forums? This is not the right place to dig into this issue and it hasn’t, yet, been reported by anyone else (except for during the outage and for when we were overloaded with requests from bots).