1309: Unplanned reboot of vmhost
After a disk was replaced of a vmhost, a reboot was needed to actually see the disk. Unfortunately, this did not help. A second try, with another disk worked and the machine is rebuilding the RAID.
CPK messages are initially sent to the CPK mailing list, you can (un)subscribe via this link. You can also follow the service interruption messages via RSS using the link in the title under the RSS icon. If the CPK takes more time to resolve, any updates are published on this website.
For RU wide service interruption see meldingen.ru.nl.
After a disk was replaced of a vmhost, a reboot was needed to actually see the disk. Unfortunately, this did not help. A second try, with another disk worked and the machine is rebuilding the RAID.
Recently, some Science accounts were hacked, after users fell for a phishing mail. These accounts were subsequently abused by internet criminals to send mails from our Linux loginservers. Therefore, we have removed the ability to login on these login servers for all Science accounts that didn’t use these servers anyway. Please contact C&CZ if you want to be able to login to the Linux login servers.
To deal with the ongoing abuse of science accounts for spam, we have disabled the possibility to send e-mail from the linux login servers. This means that the lilos cannot be used anymore to run mutt/alpine or as socksproxy for mail clients. For users with cronjobs, note that cronjobs cannot send output via e-mail anymore. Please change your cronjobs to provide feedback in a different way or discuss an alternative solution with postmaster....
After several persons fell for a phishing email, several tens of thousands of emails were sent via Science accounts. This causes our mail servers to be blocked by several mail servers on the internet. We are currently cleaning SPAM mails that are still in the outgoing queues. After that, the mail servers will be made available again. We are considering measures to prevent this form of nuisance.
Jitsi was unavailable after the reboot, because one of the web-interface configurations was still referring to an older certiticate.
The certificate for jitsi.science.ru.nl expired last week and was replaced, but a sub-service was still using a copy of the older certificate. Users were able to open the frontend, but not able to start a meeting. The problem was resolved by updating the copy of the sub-service.
Since midnight there is a problem using Adobe Acrobat Pro DC, you are asked to activate the license. ILS is working on a solution. You can open PDFs using Adobe Reader or some other PDF reader. See: https://meldingen.ru.nl/detail.php?id=1333&lang=en&tg=0&f=0&ii for the original notice.
Just before 23:09 quite a lot of ceph storage nodes became unreachable. This seems to be due to one of the redundant links between two datacenter locations failing for about 4 seconds. This triggered a whole slew of ceph osd processes being killed off and not starting again. A generic configuration change made for all our servers generated an extra interface, which confused some of the osd processes (depending on interface ordering) when starting up....
ILS wifi management notified us that they will upgrade the software of wireless access points, because security flaws have been addressed in the latest release. The maintenance ends on 2022-10-21 01:30. Source: meldingen.ru.nl
Some time ago, some Science users fell for a phishing mail, therefore their account could be misused by spammers to send mail through our mailservers. Therefore our mailservers ended up in blacklists yesterday evening. Some mails could therefore not be delivered to other servers, that mail was returned to the sender. This morning we removed the IP addresses of our servers from several blacklists, we think that the problems are resolved now....