C&CZ Problem Card

On Whit-Sunday, the certificate for some of our S3 buckets expired. After figuring out that the problem we saw was related to the certificate, we fixed the certificate and rebooted the server.

We became aware of the Kernel vulnerability and implemented the kernel vulnerability mitigation The change was to set kernel.yama.ptrace_scope = 2 (A sysctl setting). For users of MPI jobs, this setting broke the jobs. When we have a fixed kernel and rebooted the nodes, we can change this back to something that will work with MPI. Workaround or fix, use mpirun --mca btl_vader_single_copy_mechanism none etc...

The fileserver peck crashed this early morning. After a reboot, the machine ran fine again. Users of cluster nodes that use nfs exported volumes from peck might still experience problems as nfs (network file system) does not seem to be recovering well.

Since last Friday (May 8) around 18:30, our SMTP servers have intermittently refused to accept mail for sending. The issue was caused by rspamd refusing to filter messages for unknown reasons. The issue seems to be resolved for now, we will keep an eye out for the problem to re-occur. Update: It looks like the issue was resolved by explicitly setting some default settings to their default value. By doing this we were able to prevent errors that occurred with some e-mails. We think these errors caused the SMTP server stop accepting mail in some cases.

Thunderbird (and probably other mail clients) using authenticated SMTP to send e-mail complained about an invalid certificate. This was due to the replacement certificate being signed with another intermediate certificate from our certificate authority (Harica), which temporarilly broke the upchain. It was fixed around quarter to one o’clock. This is related to the certificate issues previously described in CPK 1396.

A zero day vulnerability in the Linux kernel forced us to reboot a few of the most open to the world servers, other servers will probably reboot as possible with a fixed kernel. We also installed a workaround to mitigate the issue on all servers.

Since about one week ago, the (central) firewall started to apply internal packet inspection to the regular zone transfers of the internal view of ru.nl. The transfer was then blocked by the firewall before completion, causing our copy of the data, which is required for accessing bass.ru.nl and also internal mail delivery to @ru.nl addresses, to expire after 8 days. This period is relatively short, we have 3 weeks for our own zones, and the effect of expiry was underestimated. ...

An automatic GitLab update caused a database migration failure, making the service unavailable. We resolved the issue by restarting the database and reconfiguring the GitLab migration.

A planned reboot of one of the science firewalls at 10:30 should normally have been unnoticable, however, after the firewall came back online, the keepalived service failed to start, which prevented the firewall’s virtual IP from being advertised. As a result, traffic to several internal services appeared to hang. The issue was resolved by manually starting the keepalived daemon at 10:51.

After an upgrade on Feb 3, the DIY service kept an old process running, so self‑service account creation failed. 12 users hit the “account creation failed” error between 12:01 AM and 9:33 AM. The process was restarted, the new version took over and login creation works again. Sorry for the hassle!