C&CZ Problem Card

Due to a change in how the Microsoft Exchange server validates e-mail, mails sent to an @science.ru.nl address (or similar domains for which we accept e-mail) from an external domain and with a forward to your @ru.nl mail are very likely to be dropped, because the receiving server at Microsoft considers this mail forged, in the sense that the sending client did not address the same domain as the receiving domain. This also applies to our Mail Aliases with @ru.nl addresses in them, when they are addressed from an external mail domain. The forwarded e-mail will be silently dropped, so senders will not notice a problem. ...

After an automatic update, Mattermost no longer starts. The issue has been identified; the solution requires relocating the installation. This may take some time. Mattermost is now running on a different server as gitlab, which was a lot of work to figure out, but all should be well now. For some people it may have been longer before everything worked, due to dns caching of the hostname mattermost.science.ru.nl to the wrong address.

FNWI users have been experiencing delays in email reception since early this morning. We have been working on this issue but have not solved it yet. No emails have been lost, but we suspect certificate issues between the Science mail servers and the central RU mail filter. Update: the exact cause remains unknown. During investigations together with RU-ILS, we noticed a long-standing Certificate Chain issue. However, even after resolving this, the problem remained. ...

In the process of migrating our server networks to new and faster stuff, we are moving the gateways to our (C&CZ) managed firewalls. These firewalls already handle the traffic to our 25Gbit networks, while the older 1Gbit/s connected nodes had the gateway in a temporary router, which came to replace the old DR-HUYG router a few years ago. While we don’t expect too much trouble, a hiccup may occur for up to a minute or so, while the network changes are being applied. ...

We are working to upgrade the mail servers, due to caching in various parts, you may get messages about incorrect certificates or other problems, we expect this to resolve itself once we finish the updates. Update, we apparently still had problems with the certificates, but somehow missed it after last week’s fix. There’s a temporary fix in place, so we can work further on it Tomorrow.

A change in the configuration management code has broken the certificate on the mailservers, we are trying to fix this properly, but until fixed, mail clients will complain about the server certificate. Fixed by restarting sendmail

We picked August 5th to have a planned downtime/reboot of all clusternodes in the Science Cluster. This will involve at least package updates and a reboot, if other maintenance can be included we will try to fit this in as well.

We will be moving servers from the serverroom ak008 (Huygens A-2.008) to other locations. A lot of services and virtual machines will be offline for a few hours while we disconnect, move and reconnect the hardware. Assuming everything goes well, the services should be up after turning the machines back on again. This CPK will be updated with more servers and information about affected services in the weeks before the move on August 7th 2025. ...

There are servers unreachable due to an unknown problem with the 25Gbit network switch in room ak008. We don’t know the root cause yet. Update: rebooting the affected switch has resolved the problem. DOWN amanda22.science.ru.nl DOWN cephgrafana.science.ru.nl DOWN cephgw3.science.ru.nl DOWN cephgw4.science.ru.nl DOWN cephmon2.science.ru.nl DOWN cephosd07.science.ru.nl DOWN cephosd08.science.ru.nl DOWN cephosd09.science.ru.nl DOWN cephosd10.science.ru.nl DOWN cephosd11.science.ru.nl DOWN cephosd12.science.ru.nl DOWN cephosd13.science.ru.nl DOWN cephosd23.science.ru.nl DOWN cephosd26.science.ru.nl DOWN cephosd27.science.ru.nl DOWN chemotionvm.science.ru.nl DOWN containervm02.science.ru.nl DOWN dockervm01.science.ru.nl DOWN dockervm02.science.ru.nl DOWN dockervm03.science.ru.nl DOWN ds3-huyg.net.science.ru.nl DOWN elabrolabvm.science.ru.nl DOWN filemakervm.science.ru.nl DOWN home3.science.ru.nl DOWN jupytervm.science.ru.nl DOWN mariavm02.science.ru.nl DOWN mossvm.vm.science.ru.nl DOWN rootcell.science.ru.nl DOWN vmhost03.science.ru.nl DOWN vmhost04.science.ru.nl DOWN vmhost07.science.ru.nl DOWN zammad3.vm.science.ru.nl

Services on networks that are currently behind our old picos switches will be transferred to be behind our firewalls. This may take a few minutes of downtime due to the changes needed for moving the gateway functionality. If all goes well, it will be one period of a few minutes, if we need to roll back and fix things, there may be a repeat downtime. All went well, there was a few minutes of interruption on the cncz homepage due to ARP caching issues.