C&CZ Problem Card

Because of a major security update that will be released tonight, GitLab will not be available. We will update GitLab on Tuesday morning. After that, gitlab.science.ru.nl will be available again.

A backup file, dated “Fri Apr 29 05:07:04 CEST 2016” has accidentially been restored on our production GitLab server, where it should have happened on a test server. This caused all repositories, wiki’s, issues etc. to be reset to the state of today 5.07 AM. Commits pushed between 5 am and 2:30 PM should be pushed again. Sorry for the inconvenience.

The server comas1 did not start, probably due to some timing problems between the RAID array and OS. Solution: reboot - hardware reset.

Yesterday the RU outgoing mailserver smtpout3.ru.nl got listed on the SORBS blacklist of confirmed senders of spam and phishing mails. The reason probably is that recently many students fell for phishing mails, which made their username and password known to Internet criminals. These criminals use that to send spam and phishing mails. Because also the Science mailservers use that blacklist, we did not accept mail from the RU mailserver anymore. The senders received an error message: “Your message wasn’t delivered due to a permission or security issue.” We solved this for our Science users by putting the RU mailserver on our own whitelist.

Several services were inaccessible due to power loss of a rack switch caused by a failing PDU (power distribution unit). The PDU has been temporarily replaced. Sevices include: home directories, printing, authentication services, licenses, vpn, websites, databases, mailman and dns resolvers.

Memory modules and or the motherboard of webserver ‘oehoe’ will be replaced. The mentioned web sites will not be available for a while. Affected web services: aerochem,akn,aqua,bgard,boekingen,bomen,cac,cafe,celbi,docenten,exo,filosofie,hfml iconic,ifl,irmacrash,itp2011,iwwr-boookings,kanboard,kndv,labservant,math,mbfys,mbp,mealab, nieuwescheikunde,nlt,pilab,printeger,probiomed,puff,pz,qpl,quantumcs,ra,refbase,rrl, scheikundeolympiade,siam,sigma-lustrum,smartech4health,sws,theorphys,tk,toernooi,tredd, trooster,vsc,w4kangoeroe,wortel,zwaartekrachtsgolven

A defective memory module caused spontaneous reboots. Since two modules where taken out, these reboots have not occurred anymore. Additional down time will be needed to resolve the problem. Affected services: aerochem,akn,aqua,bgard,boekingen,bomen,cac,cafe,celbi,docenten,exo,filosofie,hfml iconic,ifl,irmacrash,itp2011,iwwr-boookings,kanboard,kndv,labservant,math,mbfys,mbp,mealab, nieuwescheikunde,nlt,pilab,printeger,probiomed,puff,pz,qpl,quantumcs,ra,refbase,rrl, scheikundeolympiade,siam,sigma-lustrum,smartech4health,sws,theorphys,tk,toernooi,tredd, trooster,vsc,w4kangoeroe,wortel,zwaartekrachtsgolven

Because of an upgrade of gitlab to version 8.5.7, the GitLab service was offline for a while.

At reboot time some file system (/opt) was not recognized, resulting in a kernel panic. Solutio: Reboot

For yet unknown reasons the private class network interfaces where incorrectly numbered and activated Solution: Reboot