C&CZ Problem Card

Friday July 21 from 17:00, we will start shutting down compute clusternodes, in order to prepare for the power outage of the Huygens building Saturday July 22. Other servers will be shut down later. The most important servers (mail, home, file, Ceph, gitlab, loginservers) will be shutdown starting Saturday morning 7:00. We will try to keep basic services (DNS/DHCP, SMTP(mail) and license servers) up during this power outage. RU services are not serviced from the Huygens building, so will not be affected. ...

Due to the expiration of an LDAP certificate, it is temporarily not possible to log in to various services. A new certificate is being installed urgently. Affected services include Eduroam in combination with Science logins, VPN, GitLab and Mattermost.

Due to a failure in a power distribition unit (pdu) the servers felixdisk and bioboost went down. Both servers have been connected to another pdu and are now working again.

The planned routing change, which should not have caused issues for more than a few seconds, didn’t work as planned and caused problems for up to 15 minutes. Update 2023-06-12 - 22:00 The situation has become worse, some problems: DNS resolving, some fileservers and jupyterhub are having problems due to the network change. We will attempt to resolve the issue asap. Update 2023-06-13 - 11:30 After correcting errors (fixed IP addresses) all services are up again. The network configuration is not redundant yet, but user problems should be over. ...

Our smtp mailservers were under attack. In order to prevemt other problems, our configuration limits the number of connections that can be kept open at the same time. We cannot easily distinguish between connections by the attacker(s) and by regular users. When this limit is reached, no new connections can be made. Therefore sending e-mail using our mailservers can take a long time or will not work at all. There’s a good chance that your IP address will be blocked (max. 1 hour), e.g. if you use loginname@science.ru.nl after June 8, 16:25. ...

The datacenter cooling failed around 07:00 this morning. To prevent damage all non-essential systems are being turned off (clusternodes first). Most fileservers have also been turned off. Due to the urgency, some systems that have been turned off may not be in the location with the problem (Huygens HG04.070). Around 07:50 the cooling system came online again, about 30 minutes later the temperature dropped to under 25 degrees Celsius. After ca. 8:35, we started turning systems back on, starting with fileservers and later clusternodes as well. This is now nearly finished. Some hardware cannot easily be managed remotely so that will take a bit longer until someone can turn on the power manually. ...

On Saturday July 22, all power, including emergency power, in the Huygens building will be switched off for planned maintenance from 08:00 until 12:30. If all goes well, power will be available from 12:30. We expect to need at least a few hours to get all Science services up again. During this time, all but a few basic services will be unavailable. Most servers will be turned off the Friday before. We will attempt to maintain basic DNS/DHCP, SMTP(mail) and license services, but for instance the science mailboxes, home directories, websites, VPN, fileservers, Gitlab, loginservers and compute clusters will not be available during the off period. ...

RU ILS Connectivity announced that Friday April 21 between 18:30 and 20:30 the RU network would be separated from the internet for a period of ca 30 minutes, in order to carry out maintenance.

Friday`April 21 at 10 o’clock C&CZ will commence a GitLab server migration. During the migration, working with Gitlab will not be possible. Because Mattermost is running on the same installation of Gitlab, it will be unavailable during the process as well. We expect things to be up and running again around noon.

RU ILS Connectivity announced that network maintenance will be carried out next Wednesday evening that will cause short disturbances of network connectivity for wired and wireless systems in and around wing 8 Huygens on all floors. For wired network outlets, one can check whether they will be affected by looking at the outlet number (starting with 113- for affected systems) or visiting FNWI ethergids.