The fileserver peck crashed this early morning. After a reboot, the machine ran fine again. Users of cluster nodes that use nfs exported volumes from peck might still experience problems as nfs (network file system) does not seem to be recovering well.

Since last Friday (May 8) around 18:30, our SMTP servers have intermittently refused to accept mail for sending. The issue was caused by rspamd refusing to filter messages for unknown reasons. The issue seems to be resolved for now, we will keep an eye out for the problem to re-occur. Update: It looks like the issue was resolved by explicitly setting some default settings to their default value. By doing this we were able to prevent errors that occurred with some e-mails. We think these errors caused the SMTP server stop accepting mail in some cases.

Thunderbird (and probably other mail clients) using authenticated SMTP to send e-mail complained about an invalid certificate. This was due to the replacement certificate being signed with another intermediate certificate from our certificate authority (Harica), which temporarilly broke the upchain. It was fixed around quarter to one o’clock. This is related to the certificate issues previously described in CPK 1396.

A zero day vulnerability in the Linux kernel forced us to reboot a few of the most open to the world servers, other servers will probably reboot as possible with a fixed kernel. We also installed a workaround to mitigate the issue on all servers.

Since about one week ago, the (central) firewall started to apply internal packet inspection to the regular zone transfers of the internal view of ru.nl. The transfer was then blocked by the firewall before completion, causing our copy of the data, which is required for accessing bass.ru.nl and also internal mail delivery to @ru.nl addresses, to expire after 8 days. This period is relatively short, we have 3 weeks for our own zones, and the effect of expiry was underestimated. ...

An automatic GitLab update caused a database migration failure, making the service unavailable. We resolved the issue by restarting the database and reconfiguring the GitLab migration.

A planned reboot of one of the science firewalls at 10:30 should normally have been unnoticable, however, after the firewall came back online, the keepalived service failed to start, which prevented the firewall’s virtual IP from being advertised. As a result, traffic to several internal services appeared to hang. The issue was resolved by manually starting the keepalived daemon at 10:51.

After an upgrade on Feb 3, the DIY service kept an old process running, so self‑service account creation failed. 12 users hit the “account creation failed” error between 12:01 AM and 9:33 AM. The process was restarted, the new version took over and login creation works again. Sorry for the hassle!

We are working on replacing our incoming mailservers and something went wrong. Due to an unexpected block in the mail flow, around 2800 mails which were queued from around 18:00 on Feb 2nd, were lost. The host queuing the mail, was replaced on Feb 3rd, around 8:30 and decommissioned at around 9:00. Unfortunately, the decommissioning also destroyed the disk of the server with the mail queue of around 2800 e-mails. At 18:00 on Feb 2nd, two servers were accepting incoming mail, but only one of the servers was queuing, the other one was running as expected. ...

Elabftw had an expired certificate, because after the move to a new server, this process was left unmonitored. Both the certificate and the monitoring are now fixed.