A planned reboot of one of the science firewalls at 10:30 should normally have been unnoticable, however, after the firewall came back online, the keepalived service failed to start, which prevented the firewall’s virtual IP from being advertised. As a result, traffic to several internal services appeared to hang. The issue was resolved by manually starting the keepalived daemon at 10:51.

After an upgrade on Feb 3, the DIY service kept an old process running, so self‑service account creation failed. 12 users hit the “account creation failed” error between 12:01 AM and 9:33 AM. The process was restarted, the new version took over and login creation works again. Sorry for the hassle!

We are working on replacing our incoming mailservers and something went wrong. Due to an unexpected block in the mail flow, around 2800 mails which were queued from around 18:00 on Feb 2nd, were lost. The host queuing the mail, was replaced on Feb 3rd, around 8:30 and decommissioned at around 9:00. Unfortunately, the decommissioning also destroyed the disk of the server with the mail queue of around 2800 e-mails. At 18:00 on Feb 2nd, two servers were accepting incoming mail, but only one of the servers was queuing, the other one was running as expected. ...

Elabftw had an expired certificate, because after the move to a new server, this process was left unmonitored. Both the certificate and the monitoring are now fixed.

A combination of small failures in configuration and monitoring resulted in Daily backups not being run on peck between September 1st and last Friday. Backups are now running again, but other than the Monthly backups, there are no intermediate Daily backups for the volumes on server peck. Updates in monitoring/testing are in progress to prevent this kind of oversight in the future.

After moving the elabftw site to a new server, it was impossible to send e-mail from the server, like when a user wants to reset their password. The changes that were necessary are in between the host and the docker container running elabftw, while figuring out what was needed also caused some general issues reaching the elabftw webserver.

To make the network less complicated and more easily manageable, we need to make vlans for the subnets 131.174.8.0/24 and 131.174.9.0/24, this will involve some servers getting a new ip address. This affects firewall rules as well. We must coordinate this with ILS networking and the changes will cause disruptions between a few minutes up to a quarter of an hour. List of sites: alice.science.ru.nl alicecontrol.science.ru.nl coursetimetable.science.ru.nl dmarcreport.science.ru.nl edutools.science.ru.nl eduview.science.ru.nl ...

The certificate of the radius server has been replaced this morning, users who use their science account for authenticating to the Eduroam network will not have been able to connect until the certificate was fixed. You may need to ’trust-on-first-use’ or other setting in your device to properly reconnect and trust the certificates, if that still fails.

Every half year we do clusternode maintenance, with at least a package ugprade and a reboot, but sometimes other maintenance can happen, such as changes in filesystems or network configurations. The upcoming date for this maintenance is February 6th, 2026 (Friday) The maintenance is done.

A brief service interruption occurred this morning when one of our LDAP servers refused to start. This affected services that depend on LDAP for user authentication. Services such as Eduroam (using science login for authentication) and GitLab experienced authentication failures during this time. We are investigating the source of the failure to prevent recurrence.