CPK messages are initially sent to the CPK mailing list, you can (un)subscribe via this link. You can also follow the service interruption messages via RSS using the link in the title under the RSS icon. If the CPK takes more time to resolve, any updates are published on this website.
For RU wide service interruption see meldingen.ru.nl.
Thunderbird (and probably other mail clients) using authenticated SMTP to send e-mail complained about an invalid certificate. This was due to the replacement certificate being signed with another intermediate certificate from our certificate authority (Harica), which temporarilly broke the upchain. It was fixed around quarter to one o’clock. This is related to the certificate issues previously described in CPK 1396.
A zero day vulnerability in the Linux kernel forced us to reboot a few of the most open to the world servers, other servers will probably reboot as possible with a fixed kernel. We also installed a workaround to mitigate the issue on all servers.
Since about one week ago, the (central) firewall started to apply internal packet inspection to the regular zone transfers of the internal view of ru.nl. The transfer was then blocked by the firewall before completion, causing our copy of the data, which is required for accessing bass.ru.nl and also internal mail delivery to @ru.nl addresses, to expire after 8 days. This period is relatively short, we have 3 weeks for our own zones, and the effect of expiry was underestimated. ...
An automatic GitLab update caused a database migration failure, making the service unavailable. We resolved the issue by restarting the database and reconfiguring the GitLab migration.
A planned reboot of one of the science firewalls at 10:30 should normally have been unnoticable, however, after the firewall came back online, the keepalived service failed to start, which prevented the firewall’s virtual IP from being advertised. As a result, traffic to several internal services appeared to hang. The issue was resolved by manually starting the keepalived daemon at 10:51.
After an upgrade on Feb 3, the DIY service kept an old process running, so self‑service account creation failed. 12 users hit the “account creation failed” error between 12:01 AM and 9:33 AM. The process was restarted, the new version took over and login creation works again. Sorry for the hassle!
We are working on replacing our incoming mailservers and something went wrong. Due to an unexpected block in the mail flow, around 2800 mails which were queued from around 18:00 on Feb 2nd, were lost. The host queuing the mail, was replaced on Feb 3rd, around 8:30 and decommissioned at around 9:00. Unfortunately, the decommissioning also destroyed the disk of the server with the mail queue of around 2800 e-mails. At 18:00 on Feb 2nd, two servers were accepting incoming mail, but only one of the servers was queuing, the other one was running as expected. ...
Elabftw had an expired certificate, because after the move to a new server, this process was left unmonitored. Both the certificate and the monitoring are now fixed.
A combination of small failures in configuration and monitoring resulted in Daily backups not being run on peck between September 1st and last Friday. Backups are now running again, but other than the Monthly backups, there are no intermediate Daily backups for the volumes on server peck. Updates in monitoring/testing are in progress to prevent this kind of oversight in the future.
After moving the elabftw site to a new server, it was impossible to send e-mail from the server, like when a user wants to reset their password. The changes that were necessary are in between the host and the docker container running elabftw, while figuring out what was needed also caused some general issues reaching the elabftw webserver.