C&CZ Problem Card

In Easter Weekend (March 30th 2024) the new mailserver will replace the old one. The final switch will take a short while, during which mail clients (eg Thunderbird, mail on your phone, etc.) will not be able to connect to the science mailserver. (This is a week later than announced, due to some issues that need to be resolved before final migration) Central mail services (Microsoft @ru.nl) are independent of our mailserver and thus will not be affected at all.

Ceph filesystem failure. Services required for the CephFS filesytem cannot start. Failure of these services wil cause failure to access files on Ceph. We are in contact with our support party 42on to remedy the problem. Duration is currently unknown. Update 2024-01-22 We’re working with 42on on the issue and have a meeting scheduled for 17:00 today. Update 2024-01-23 An initial dentry_recover was successful and according to 42on the Ceph journals are OK. We are hopeful that we can fix the issue with Ceph in the coming days. ...

On December 10, 2023 the main GitLab TLS certificate expired. A new certificate was generated, but it lacked a proper certificate chain. Browsers disregard that chain, but other GitLab components do care, which meant that the registry and the runners were unable to connect to GitLab. On Monday the 11th we renewed all certificates using Let’s Encrypt. Let’s Encrypt will also auto-renew these.

Jitsi (or actually the internal prosody service) had an old certificate, causing it to refuse startup after the normal reboot. The expired certificate was replaced and after a restart of the prosody service it works again.

After maintenance on the central routers, our servers on the 25Gbit network couldn’t send traffic to the internet. The cause or nature of the problem appears to be in the central RU routing. By resetting the interface to our network, RU Connectivity can fix this, when it occurs again. RU Connectivity contacted their maintenance provider to investigate this problem. Final solution: After a few failed attempts, it was possible to define a static route in the central routers to our networks.

Some users were unable to login, the server had become unstable, which made a reboot necessary. Unfortunately, the jupyterhub service still needs a manual startup, which didn’t happen until 13:24 hours. The service is now working again. We are working to fix the reboot problem.

A typo that could propagate to shutdown the DHCP server had the effect that some network devices did not get an IP address when they were switched on and that others, whose address lease expired in this period, lost their IP address and thus their access to the network.. We’ll improve the process to prevent a typo from bringing down the DHCP server in the future.

After a scheduled reboot of the machine running jupyterhub, jupyter failed to start properly. The service started after invoking a manual start command. The measures taken to resolve this recurring problem have not proven to be sufficient.

Due to a misconfiguration of the DNS in the z.science.ru.nl zone, all shares were not available during the outage. Extra tests are added to prevent a future occurrence.

The router for most Science services urgently needs a reboot. This has been scheduled for early morning. In the unlikely case that this reboot fails, ILS Connectivity is on campus to fix it.