Introduction
In our Faculty, different types of data storage are offered:
Storage class | Description | Risk for data loss |
---|---|---|
Home directories | Small (several GB), reliable, backed up storage for individuals | low |
Network shares | Larger (< 1TB), reliable, backed up storage for individuals or groups, with ZFS > 1TB is possible | low |
Ceph storage | Large scale, reliable, snapshotted, not backed up storage for groups | medium |
Local storage | Not backed up storage on desktop computers and cluster nodes, likely to be lost in case of a hardware problem or when the machine gets reinstalled | high |
Home directories
Your Science login comes with a home directory of 5GB at no costs. This is a safe place to store you work related documents. Home directories are stored on reliable hardware and backupped automatically. If you need more than 5GB of storage, and you can’t clean it up, it can be made bigger on request.
Home directory paths
On C&CZ managed systems, this is where you can find your home directory:
C&CZ systems | Path |
---|---|
Microsoft Windows | U: a.k.a. the U-drive |
Linux | /home/yourscienceloginname a.k.a ~ |
From other systems like your home PC, you can reach your home directory through the paths below:
Own system | Path | Instructions |
---|---|---|
Microsoft Windows | \\home1.science.ru.nl\yourscienceloginname |
connect network share |
macOS | smb://home1.science.ru.nl/yourscienceloginname |
connect network share |
Linux | smb://home1.science.ru.nl/yourscienceloginname |
connect network share |
Info
A VPN connection is required when accessing these paths from outside of the RU network.
Info
The location of your homedirectory, ie home1
or home2
, can viewed in DIY.
Access rights
By default, home directories are only accessible to the owners themselves. However, it is possible
to change the access rights (ie. chmod o+rx ~
).
Network shares
For larger storage needs and the ability to share files in your group, there are network shares. Network shares are available as both Samba1 and NFS2 share. As mentioned in the storage overview, network shares are backed up and hosted on reliable servers. The costs involved with network shares depend on the share size and the backup options.
Network share costs
Network shares up to 1TB are available with our standard backup plan. For the costs of custom backup options, consult the backup page. Larger shares are possible when using ZFS as a filesystem. These are the yearly costs:
Size | Backed up | Without backup |
---|---|---|
200 GB | € 40 | € 10 |
400 GB | € 80 | € 20 |
1 TB3 | € 200 4 | € 50 |
1 TB (ZFS) | € 1505 | € 50 / TB |
> 1 TB (ZFS) | € 1006 | € 50 / TB |
Shares with ZFS have local and remote snapshots.
Network share paths
On C&CZ managed systems, this is where you can find a network share. Using sharename
as an example:
C&CZ systems | Path |
---|---|
Microsoft Windows | \\sharename-srv.science.ru.nl\sharename |
Linux | /vol/sharename |
For self managed systems, you can access the following network paths:
Own system | Path | Instructions |
---|---|---|
Microsoft Windows | \\sharename-srv.science.ru.nl\sharename |
connect network share |
macOS | smb://sharename-srv.science.ru.nl/sharename |
connect network share |
Linux | smb://sharename-srv.science.ru.nl/sharename |
connect network share |
Info
A VPN connection is required when accessing these paths from outside of the RU network.
Access rights
The ability to access files on these folders can be limited to a group of logins. Groups can be managed by the group owners on DIY.
Special shares
Some services are built on these specific shares:
Share name | Purpose | Documentation |
---|---|---|
temp |
Temporary storage of files | Temp share |
cursus |
Software for courses | T-schijf |
software |
General (Windows) software | S-schijf |
install |
Installer programs for software | Install share |
Ordering
When ordering a network share, please send the following details to Postmaster:
Detail | Description |
---|---|
login name | Your Science account |
share name | How the share can be accessed over the network, see paths |
size | size of the partition, see costs |
charge account | a.k.a. kostenplaats or project code |
unix group (optional) | The group that is granted access to the share. If not specified, a new group will be created with the same name as the share |
Ceph Storage
With our Ceph storage cluster, we can provide much larger storage for the Faculty of Science. The storage cluster consists of multiple disk servers and is spread across three locations on campus. Ceph provides three basic storage modes; Filesystem (FS), Rados Gateway (RGW or S3) and Rados Block Device (RBD). In this section only Ceph FS is meant.
Warning
Ceph volumes are not backed up! The volumes are too large to backup.
Choices in redundancy
Ceph provides different redundancy options for storing data.
By default, Ceph stores data in threefold: the 3copy
pool.
So when one copy is lost, the remaining copies still have redundancy.
Another redundancy option is erasure coding (EC). EC requires much less storage overhead compared to 3copy.
Ceph costs
Costs are dependant of the selected storage pool. Below prices are per TB3 per year:
Pool | Description | without backup |
---|---|---|
EC8+3 |
cheapest, when one datacenter is destroyed, all the data is lost. When one datacenter becomes temporarily unavailable, the data is still safe, but off-line. | € 50 |
EC5+4 |
remains available when a whole datacenter is offline or lost. Data remains safe as long as two datacenters are working well. | € 60 |
3copy |
faster read + write, will remain available when one datacenter becomes unavailable. | € 100 |
Info
The performance properties of Ceph are different from traditional single server storage. Write speed usually exceeds read speed. Lots of small files is killing for throughput (even worse than on traditional storage).
Intended use; storing/archiving large datasets consisting of large files.
Do not use as if it is a local disk
Ceph file systems can be used as a regular network share or accessed by some servers or cluster nodes that have direct Ceph access: Ceph clients. See the next paragraph on Ceph paths.
Object storage (S3 compatible) is also available. Object storage differs fundamentally from a traditional filesystem, data stored in a Ceph filesystem cannot be accessed through S3.
Ceph paths
On C&CZ managed systems, this is where you can find a Ceph based network share. Using sharename
as an example:
C&CZ systems | Path | Instructions |
---|---|---|
Microsoft Windows | \\sharename-srv.science.ru.nl\sharename |
connect network share |
Linux | /vol/sharename |
|
Linux Ceph clients7 | /ceph/sharename |
Info
Accessing Ceph storage from your own managed system works just like any other network share.
Ordering
When ordering Ceph storage, please send the following details to Postmaster:
Detail | Description |
---|---|
share name | How the share can be accessed over the network, see paths |
pool | See Choices in redundancy and Ceph costs |
size | the quota on the ceph volume, see costs |
charge account | a.k.a. kostenplaats or project code |
unix group | The group that is granted access to the volume. |
Local storage
Local storage refers to disk space in or attached to systems like desktop pc’s and cluster nodes. Usually, local storage
consists of single disks. In case of a disk failure, your data is likely te be lost. Local disks are usually called /scratch
to remind you of the temporary nature of the storage. Local storage is not backed up. Advantages of local storage are low costs and fast disk access.
Info
On shared systems, like login servers and cluster nodes, make sure to create a personal directory in /scratch
first. Ie. mkdir /scratch/$USER
. And don’t forget to let your software actually write to that directory instead of the default and much smaller /tmp
.
-
Windows file sharing implementation by the Samba project ↩︎
-
Network file system, https://en.wikipedia.org/wiki/Network_File_System ↩︎
-
Daily backup not possible with lots of daily changing small files. With ZFS this is not an issue ↩︎
-
This is a share with snapshots, local and remote and a monthly backup to tape. ↩︎
-
This is a larger share with local and remote snapshots, but without backup to tape. ↩︎
-
Ceph clients are C&CZ managed Linux servers or cluster nodes that have a direct connection to the Ceph cluster ↩︎