After switching the gateway for one of our busiest vlans to our own firewalls, a lot of websites were only available from outside that vlan, it took a while to understand the issue and find a fix for it. We now have a temporary fix in place that should make the websites work again.

Due to an unfortunate manual command, the package for filemaker was removed from the server. Service was restored after the weekend, when the backup was restored.

Due to an unfortunate manual command, the package for DHZ was removed from the servers, causing all attempts to login with MFA to fail on for example roundcube and lilo ssh login. Service was restored about half an hour later when the DHZ package was installed again.

Due to a change in how the Microsoft Exchange server validates e-mail, mails sent to an @science.ru.nl address (or similar domains for which we accept e-mail) from an external domain and with a forward to your @ru.nl mail are very likely to be dropped, because the receiving server at Microsoft considers this mail forged, in the sense that the sending client did not address the same domain as the receiving domain. This also applies to our Mail Aliases with @ru.nl addresses in them, when they are addressed from an external mail domain. The forwarded e-mail will be silently dropped, so senders will not notice a problem. ...

After an automatic update, Mattermost no longer starts. The issue has been identified; the solution requires relocating the installation. This may take some time. Mattermost is now running on a different server as gitlab, which was a lot of work to figure out, but all should be well now. For some people it may have been longer before everything worked, due to dns caching of the hostname mattermost.science.ru.nl to the wrong address.

FNWI users have been experiencing delays in email reception since early this morning. We have been working on this issue but have not solved it yet. No emails have been lost, but we suspect certificate issues between the Science mail servers and the central RU mail filter. Update: the exact cause remains unknown. During investigations together with RU-ILS, we noticed a long-standing Certificate Chain issue. However, even after resolving this, the problem remained. ...

In the process of migrating our server networks to new and faster stuff, we are moving the gateways to our (C&CZ) managed firewalls. These firewalls already handle the traffic to our 25Gbit networks, while the older 1Gbit/s connected nodes had the gateway in a temporary router, which came to replace the old DR-HUYG router a few years ago. While we don’t expect too much trouble, a hiccup may occur for up to a minute or so, while the network changes are being applied. ...

We are working to upgrade the mail servers, due to caching in various parts, you may get messages about incorrect certificates or other problems, we expect this to resolve itself once we finish the updates. Update, we apparently still had problems with the certificates, but somehow missed it after last week’s fix. There’s a temporary fix in place, so we can work further on it Tomorrow.

A change in the configuration management code has broken the certificate on the mailservers, we are trying to fix this properly, but until fixed, mail clients will complain about the server certificate. Fixed by restarting sendmail

We picked August 5th to have a planned downtime/reboot of all clusternodes in the Science Cluster. This will involve at least package updates and a reboot, if other maintenance can be included we will try to fit this in as well.