Ceph storage interruption   cpk

Just before 23:09 quite a lot of ceph storage nodes became unreachable. This seems to be due to one of the redundant links between two datacenter locations failing for about 4 seconds. This triggered a whole slew of ceph osd processes being killed off and not starting again. A generic configuration change made for all our servers generated an extra interface, which confused some of the osd processes (depending on interface ordering) when starting up....

October 24, 2022 · updated November 29, 2022 ·  Simon

Data Storage   howto

Data Storage The data storage on C&CZ servers can be used from all kinds of C&CZ servers and personal computers, but also from other PCs or even from home with WinSCP or via VPN. Almost all disks that are managed by C&CZ, are being backed up regularly, in order to be able to restore data in case of small or large calamities. Home directories Every user with a Science login has or is entitled to an amount of disc space of a few Gigabytes on a server....

September 8, 2022 · updated December 7, 2022

Ceph problem   cpk

During a routine upgrade of ceph, a bug in the latest version manifested itself and made the ceph manager unreachable. After aborting the upgrade and with help from the ceph-users mailinglist, everything became available again using a workaround.

March 24, 2021 · updated November 29, 2022 · 

Backup   howto

If accidentally one or more files are lost from a home-directory (U:-disk) or a /vol-directory or some other place on a C&CZ server, probably the files can be restored. To be present on backup, a file must of course have existed during a (nightly) backup. Different independent backup schemes are used : Daily backups to Ceph-storage. Monthly backups to tape. Yearly backups to tape. Daily backup to tape All server disk-partitions except temporary space like /tmp, /var/tmp, /scratch, are backed-up after each working day on Ceph storage....

December 17, 2020 · updated November 29, 2022 ·  Peter

New data storage: Ceph   news

Recently C&CZ presented an extra choice for data storage, next to the traditional RAID storage on individual fileservers: Ceph. The main advantages of Ceph are, that is scales almost without limits and does not have a single point of failure, but instead is “self-healing” and “self-managing”. Within Ceph storage, choices can be made, the more expensive ones have no problem when a complete datacenter goes down. The Ceph storage can also be used as S3-storage....

December 17, 2019 · updated November 29, 2022 ·  Simon