If accidentally one or more files are lost from a home-directory (U:-disk) or a /vol-directory or some other place on a C&CZ server, probably the files can be restored. To be present on backup, a file must of course have existed during a (nightly) backup.
Different independent backup schemes are used :
- Daily backups to Ceph-storage.
- Monthly backups to tape.
- Yearly backups to tape.
All server disk-partitions except temporary space like /tmp, /var/tmp, /scratch, are backed-up after each working day on Ceph storage. Because of the time it takes to do a backup, most of the times this is a so-called incremental dump, which means that only changes w.r.t. the previous backup are written. These backups are started each working day after working hours. These backups are preserved for four weeks. Because these backups are on online Ceph storage, the can be retrieved faster than they would be if they would reside on tapes. The Ceph storage chosen is also available when one datacenter would be unavailable.
Monthly and yearly backup to tape
Every weekend a subset of all server disk partitions are completely written to a couple of LTO6-tapes. Every partition is written to tape roughly once every four weeks. These monthly backup-tapes are preserved for one year. The yearly backups are performed around the Christmas break (end of December and start of January), written on LTO-7 tapes and preserved for at least five years.
If the file to be restored was not on a C&CZ server the last four weeks, it has to be restored from monthly or yearly backups. The most recent monthly tapes are stored off-site for use after calamities.
The backups are made with the Amanda open source network backup software. The backup server, that was renewed in 2013, is a Dell PowerEdge R720xd with 44 TB disc space, that is used to make fast restores from disk possible. The backups are written to LTO-6 tapes in a Dell TL2000 tape library with 2 LTO6 drives and 24 slots. The new service reads and writes LTO-6, but it can also read the LTO-4 tapes that we used before.
The full backup schedule (Daily plus Monthly plus Yearly backups) costs 155 €/TB year for FNWI departments. It breaks down to:
- Daily: 70 €/TB year
- Monthly: 60 €/TB year
- Yearly: 25 €/TB year
The costs of backup are the major part of the cost of C&CZ disc space.
Other backup options
We are currently using some other backup methods as well:
- Restic backups
- ZFS snapshots
- Zrepl replicated snapshots to a second machine
- Cloud backups are in development