NEMO parallel file system upgrade

NEMO's parallel file system is the work horse for data intensive computing tasks. It was designed to work efficiently on large amounts of transient data, accessible from each compute node through a common name space.

The work space mechanism was introduced to prevent the parallel file system from filling up in an uncontrolled way. It forces users to renew their storage allocations every 100 days.

However, with NEMO's popularity rising and new users arriving, the demand for storage has increased as well.

Therefore, the parallel file system has been upgraded to a net capacity of almost one Petabyte (960 Terabyte to be exact).

Operations on file system meta data (creation, deletion or merely checking for existence of files) have been identified as a possibly bottleneck. To improve performance, two additional services for meta-data have been deployed.

NEMO users should not take this as an invitation to violate the current soft quotas (20 TB, 1 Million files). The parallel file system was almost full before the upgrade. In fact, since no further upgrades are planned, more restrictive quota handling will be needed in the future.