This morning gitlab.science.ru.nl was not working correctly or at all. Finally a reboot of the host fixed all issues.

Since a while we’ve noticed our public gitlab server being targeted from various places on the internet, it looks like a DDOS or it could be a lot of AI agents scraping data. Whatever the reason, it is harming regular users normal way of working. Therefore we have implemented an agressive blocking strategy, which can block large parts of the ip range of the internet for a limited time. To compensate for the problems this causes, we have also implemented a fixed allowlist to prevent known friendly ranges from being blocked in this way. If you are cooperating with groups of people who experience this blocking mechanism, please inform us of ip ranges you would like to have included in the allowlist. ...

On Whit-Sunday, the certificate for some of our S3 buckets expired. After figuring out that the problem we saw was related to the certificate, we fixed the certificate and rebooted the server.

We became aware of the Kernel vulnerability and implemented the kernel vulnerability mitigation The change was to set kernel.yama.ptrace_scope = 2 (A sysctl setting). For users of MPI jobs, this setting broke the jobs. When we have a fixed kernel and rebooted the nodes, we can change this back to something that will work with MPI. Workaround or fix, use mpirun --mca btl_vader_single_copy_mechanism none etc...

The fileserver peck crashed this early morning. After a reboot, the machine ran fine again. Users of cluster nodes that use nfs exported volumes from peck might still experience problems as nfs (network file system) does not seem to be recovering well.

Since last Friday (May 8) around 18:30, our SMTP servers have intermittently refused to accept mail for sending. The issue was caused by rspamd refusing to filter messages for unknown reasons. The issue seems to be resolved for now, we will keep an eye out for the problem to re-occur. Update: It looks like the issue was resolved by explicitly setting some default settings to their default value. By doing this we were able to prevent errors that occurred with some e-mails. We think these errors caused the SMTP server stop accepting mail in some cases.

Thunderbird (and probably other mail clients) using authenticated SMTP to send e-mail complained about an invalid certificate. This was due to the replacement certificate being signed with another intermediate certificate from our certificate authority (Harica), which temporarilly broke the upchain. It was fixed around quarter to one o’clock. This is related to the certificate issues previously described in CPK 1396.

A zero day vulnerability in the Linux kernel forced us to reboot a few of the most open to the world servers, other servers will probably reboot as possible with a fixed kernel. We also installed a workaround to mitigate the issue on all servers.

Since about one week ago, the (central) firewall started to apply internal packet inspection to the regular zone transfers of the internal view of ru.nl. The transfer was then blocked by the firewall before completion, causing our copy of the data, which is required for accessing bass.ru.nl and also internal mail delivery to @ru.nl addresses, to expire after 8 days. This period is relatively short, we have 3 weeks for our own zones, and the effect of expiry was underestimated. ...

An automatic GitLab update caused a database migration failure, making the service unavailable. We resolved the issue by restarting the database and reconfiguring the GitLab migration.