C&CZ Problem Card

During the regular change of DNSSEC keys that secure DNS traffic, an incorrect key was introduced in the external DNS of ru.nl for astro.ru.nl. This made astro.ru.nl disappear from the internet. This error was partly corrected 2022-05-02 at ca. 14:00 hours, but the automatic process used an not accepted encryption. It took ILS until 2022-05-22 to correct that by hand after we eventually noticed the error. Because the DNS answer that astro.ru.nl doesn’t exist may be cached for 24 hours, the problem was not completely over until 2022-05-13 13:00. The main problems for users were that mail from an @astro.ru.nl address bounced. ...

The network switch of the Coma cluster seems to be broken, all attached nodes are separated from the rest of the network. We’ll replace the switch a.s.a.p. and (let) analyze the problem after that.

February 14, ILS switched off antique versions of TLS (1.0 and 1.1) for the Eduroam authentication on ILS LDAP servers. From then on, SUSE Linux 15.3 clients can’t authenticate with U- or s-number. They only have TLS1.2 and the ILS servers offer TLS1.3 first, after that an error occurs. By using the Science-account to authenticate, these users succeed in connecting to Eduroam.

Due to human error, the connection between the new datacenter switches and the central router was interrupted.

Vmhost07 was accidentally shut down. Cause: human error. labservanttest neurotech2 printvm msql01 indicoimapp ldap2 eftw jupytervm

As a result of the expansion of the Ceph storage cluster, the cluster had performance and availability issues. The problems were resolved this morning.

Two modules of an important switch in the main C&CZ server room lost power during the preparation of planned maintenance. This disconnected ca. 75% of the servers in the room from the network. Moving the modules to new PDU’s kimited the downtime to ca. 15 minutes.

An error in the management software prevented all license processes from starting correctly at the reboot of the license server. After fixing this error, all licenses were available again.

Course software that had been tested caused an overload of the fileserver when it was used by 100 students. The performance of the fileserver was impaired for all users of network shares of this server.

A broken PDU has offlined a switch, which has caused the VPN server to be unreachable (and several other things, which don’t affect users).