C&CZ Problem Card

The VPNsec service will be moved to a new server. This move will cause downtime and existing VPN connections will be destroyed. Downtime is expected not to exceed several minutes.

Last friday, a change in the mailman configuration has been rolled out which had the inadvertent effect that mails were not delivered to external addresses anymore. However, these mailman posts were sent successfully to internal Science mail addresses. The change has been rolled back for the moment but is a necessity meaning that we’re looking for another solution.

The connecting router (dr-huyg) for all servers in the subnets 131.174.30.0/24, 131.174.31.0/24 and 131.174.16.128/26 will be replaced. It is expected that this will cause an interruption of ca. 10 minutes in the connectivity, but unforeseen circumstances may increase this delay. The reason to do this now is because of the planned power interruption on July 22. The old router hardware has a high probability of failing to survive this.

Friday July 21 from 17:00, we will start shutting down compute clusternodes, in order to prepare for the power outage of the Huygens building Saturday July 22. Other servers will be shut down later. The most important servers (mail, home, file, Ceph, gitlab, loginservers) will be shutdown starting Saturday morning 7:00. We will try to keep basic services (DNS/DHCP, SMTP(mail) and license servers) up during this power outage. RU services are not serviced from the Huygens building, so will not be affected....

Due to the expiration of an LDAP certificate, it is temporarily not possible to log in to various services. A new certificate is being installed urgently. Affected services include Eduroam in combination with Science logins, VPN, GitLab and Mattermost.

Due to a failure in a power distribition unit (pdu) the servers felixdisk and bioboost went down. Both servers have been connected to another pdu and are now working again.

The planned routing change, which should not have caused issues for more than a few seconds, didn’t work as planned and caused problems for up to 15 minutes. Update 2023-06-12 - 22:00 The situation has become worse, some problems: DNS resolving, some fileservers and jupyterhub are having problems due to the network change. We will attempt to resolve the issue asap. Update 2023-06-13 - 11:30 After correcting errors (fixed IP addresses) all services are up again....

Our smtp mailservers were under attack. In order to prevemt other problems, our configuration limits the number of connections that can be kept open at the same time. We cannot easily distinguish between connections by the attacker(s) and by regular users. When this limit is reached, no new connections can be made. Therefore sending e-mail using our mailservers can take a long time or will not work at all. There’s a good chance that your IP address will be blocked (max....

The datacenter cooling failed around 07:00 this morning. To prevent damage all non-essential systems are being turned off (clusternodes first). Most fileservers have also been turned off. Due to the urgency, some systems that have been turned off may not be in the location with the problem (Huygens HG04.070). Around 07:50 the cooling system came online again, about 30 minutes later the temperature dropped to under 25 degrees Celsius. After ca....

On Saturday July 22, all power, including emergency power, in the Huygens building will be switched off for planned maintenance from 08:00 until 12:30. If all goes well, power will be available from 12:30. We expect to need at least a few hours to get all Science services up again. During this time, all but a few basic services will be unavailable. Most servers will be turned off the Friday before....