Storage on C&CZ News

Data storage

Bram Daams — Thu, 13 Mar 2025 12:36:19 +0000

Introduction

In our Faculty, different types of data storage are offered:

Storage class	Description	Risk for data loss
Home directories	Small (several GB), reliable, backed up storage for individuals	low
Network shares	Larger (1-200 TB), reliable, backed up storage for individuals or groups	low
Local storage	Not backed up storage on desktop computers and cluster nodes, likely to be lost in case of a hardware problem or when the machine gets reinstalled	high

Home directories

Your Science login comes with a home directory of 5GB at no costs. This is a safe place to store you work related documents. Home directories are stored on reliable hardware and backupped automatically. If you need more than 5GB of storage, and you can’t clean up some data, it can be enlarged on request.

Access network shares via other network protocols

Miek Gieben — Wed, 26 Jun 2024 15:33:06 +0200

If you use C&CZ storage, you now have the new option to access these via network protocols other than SMB (for Windows and macOS) and NFS (for Linux), namely:

sftp: a secure variant of FTP
https: a (read-only) view of the share for browsers
webdavs: a (read-write) view of the share for browsers and webdav clients
s3: a read/write view using the Simple Storage Service (S3)

Each of these may be active at the same time, and optionally can have an end date, so you can “open up” a share for writes from another organisation which then automatically “closes” at the end date. For sftp, authentication is done via an SSH key. The other protocols are authenticated with a username and password. These credentials are not tied to any person and are easily revoked or re-generated if the need arises.

ZFS Volumes

Miek Gieben — Fri, 12 Jan 2024 15:12:45 +0000

A new storage option has been developed by C&CZ. This page will show the technical details and properties of this storage, which will hopefully help you make a decision if this is something you want. Any storage we handout we call a “volume”. For these ZFS volumes:

Size: Storage size can be as large as you want, but smaller than the amount we can put in a single machine. Currently (2024+) we can buy machines with roughly 140-200 TB storage.
Snapshots: On this we enable snapshots, so that you can retrieve (deleted) files your self.
Backup: Each volume will be back upped to local S3 storage. This should be thought of as disaster recovery, and restoring from the cloud should (hopefully) be rarely needed.

All of these volumes are visible in DIY in the “Storage” section if you have access to them. How to order such storage can be seen here.

Cleanup disk space

Bram Daams — Fri, 24 Nov 2023 12:17:00 +0000

You might have received an email that your personal or group network disk is almost full. This articles describes some tools to identify large files/directories and cleanup disk space.

Finding large files / directories with WinDirStat (graphical, Windows)

WinDirStat is a disk cleanup tool for Windows that provides a visual representation of disk space usage. It helps you to quickly identify and understand which files and folders are consuming the most space. The program offers detailed information about each file, including type, last modified date, and file path. In addition to visualization, WinDirStat includes built-in cleanup options, enabling you to directly delete or move files within the program.

Disk quota

Ben Polman — Wed, 13 Sep 2023 17:07:00 +0000

Your personal quota only determine the amount of disk space your own data may occupy and the number of files and folders you may have on your Windows U:\ disk or Linux home directory. Quotas may be active on other disks as well, however not all disks have quotas. The mail servers also use quotas to limit the disk space of each user’s mailbox.

Windows

On Windows desktops where the home directory is available as U: or H: disk right click on the disk, then choose Properties to view the amounts of used and available personal disk space in kbytes.

Snapshots with ZFS

Miek Gieben — Wed, 05 Jul 2023 15:12:45 +0000

Network shares with ZFS as file system have the ability (which C&CZ switches on by default) to make snapshots.

A snapshot is a read-only copy of the share, made at a specific point in time. Such snapshots allow you to find previous versions of your files. On most shares, snapshots go back 45 days. After that period they are removed.

Tip

Mounting a network share

Bram Daams — Mon, 16 Jan 2023 15:12:45 +0000

What path to mount?

The storage page describes the paths for your home directory and network shares. Once you know what network share to mount, follow the steps for your operating system below.

Info

A VPN connection is required when accessing these paths from outside of the RU network.

If you have trouble connecting to a network share after following the steps below, do not hessitate to contact us. We’ll be happy to assist you!

Ceph storage interruption

Simon Oosthoek — Mon, 24 Oct 2022 00:00:00 +0200

Just before 23:09 quite a lot of ceph storage nodes became unreachable. This seems to be due to one of the redundant links between two datacenter locations failing for about 4 seconds. This triggered a whole slew of ceph osd processes being killed off and not starting again. A generic configuration change made for all our servers generated an extra interface, which confused some of the osd processes (depending on interface ordering) when starting up. We are reasonably confident we can avoid this from happening in the future.

Ceph data storage expanded

Wim Janssen — Tue, 23 Nov 2021 14:06:00 +0100

Since the end of 2019, C&CZ offers Ceph as a choice for data storage, next to the traditional RAID storage on individual fileservers. Because demand for Ceph storage is rising, the Ceph storage has been upgraded from 1.8 PiB to 2.8 PiB gross. With this upgrade, storage servers have been placed in a central RU datacenter, so the Ceph storage is now distributed over three datacenters. If you want to try whether Ceph suits you, please contact C&CZ.