C&CZ Problem Card

After the power down of the Huygens building we are experiencing a problem with bringing Ceph file system back online. We currently do not know when the Ceph cluster is operational again. Update 2023-08-01 10:30 Ceph is working again. This CPK is now closed. CPK#1338 is also closed. Update 2023-07-31 12:30 After some more support from 42on, we managed to restart the cephfs, we cannot be sure all files are there, but almost all files are. Please let us know if you are missing crucial data, in theory we can find and restore the data, though we have to process 500 million entries to find it, which will take a long time. ...

The VPNsec service will be moved to a new server. This move will cause downtime and existing VPN connections will be destroyed. Downtime is expected not to exceed several minutes.

Last friday, a change in the mailman configuration has been rolled out which had the inadvertent effect that mails were not delivered to external addresses anymore. However, these mailman posts were sent successfully to internal Science mail addresses. The change has been rolled back for the moment but is a necessity meaning that we’re looking for another solution.

The connecting router (dr-huyg) for all servers in the subnets 131.174.30.0/24, 131.174.31.0/24 and 131.174.16.128/26 will be replaced. It is expected that this will cause an interruption of ca. 10 minutes in the connectivity, but unforeseen circumstances may increase this delay. The reason to do this now is because of the planned power interruption on July 22. The old router hardware has a high probability of failing to survive this.

Friday July 21 from 17:00, we will start shutting down compute clusternodes, in order to prepare for the power outage of the Huygens building Saturday July 22. Other servers will be shut down later. The most important servers (mail, home, file, Ceph, gitlab, loginservers) will be shutdown starting Saturday morning 7:00. We will try to keep basic services (DNS/DHCP, SMTP(mail) and license servers) up during this power outage. RU services are not serviced from the Huygens building, so will not be affected. ...

Due to the expiration of an LDAP certificate, it is temporarily not possible to log in to various services. A new certificate is being installed urgently. Affected services include Eduroam in combination with Science logins, VPN, GitLab and Mattermost.

Due to a failure in a power distribition unit (pdu) the servers felixdisk and bioboost went down. Both servers have been connected to another pdu and are now working again.

The planned routing change, which should not have caused issues for more than a few seconds, didn’t work as planned and caused problems for up to 15 minutes. Update 2023-06-12 - 22:00 The situation has become worse, some problems: DNS resolving, some fileservers and jupyterhub are having problems due to the network change. We will attempt to resolve the issue asap. Update 2023-06-13 - 11:30 After correcting errors (fixed IP addresses) all services are up again. The network configuration is not redundant yet, but user problems should be over. ...

Our smtp mailservers were under attack. In order to prevemt other problems, our configuration limits the number of connections that can be kept open at the same time. We cannot easily distinguish between connections by the attacker(s) and by regular users. When this limit is reached, no new connections can be made. Therefore sending e-mail using our mailservers can take a long time or will not work at all. There’s a good chance that your IP address will be blocked (max. 1 hour), e.g. if you use loginname@science.ru.nl after June 8, 16:25. ...

The datacenter cooling failed around 07:00 this morning. To prevent damage all non-essential systems are being turned off (clusternodes first). Most fileservers have also been turned off. Due to the urgency, some systems that have been turned off may not be in the location with the problem (Huygens HG04.070). Around 07:50 the cooling system came online again, about 30 minutes later the temperature dropped to under 25 degrees Celsius. After ca. 8:35, we started turning systems back on, starting with fileservers and later clusternodes as well. This is now nearly finished. Some hardware cannot easily be managed remotely so that will take a bit longer until someone can turn on the power manually.