Skip to content

Commit d5087f8

Browse files
committed
Update filesystems/lustre to reflect our new ExaScaler system
1 parent 258219e commit d5087f8

File tree

6 files changed

+144
-109
lines changed

6 files changed

+144
-109
lines changed
337 KB
Loading

docs/filesystems/lfs.md

Lines changed: 16 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -11,36 +11,27 @@ This technique is called [**file striping**](https://en.wikipedia.org/wiki/Data_
1111
The stripes are distributed among the OSTs in a round-robin fashion to ensure load balancing.
1212
It is thus important to know the number of OST on your running system.
1313

14-
As mentioned in the [Lustre implementation section](lustre.md#storage-system-implementation), the ULHPC Lustre infrastructure is composed of 2 MDS servers (2 MDT), 2 OSS servers and **16 OSTs**.
14+
As mentioned in the [Lustre implementation section](lustre.md#storage-system-implementation), the ULHPC Lustre infrastructure is composed of **2 MDTs** and **8 OSTs**.
1515
You can list the MDTs and OSTs with the command `lfs df`:
1616

1717
```bash
1818
$ cds # OR: cd $SCRATCH
1919
$ lfs df -h
2020
UUID bytes Used Available Use% Mounted on
21-
lscratch-MDT0000_UUID 3.2T 15.4G 3.1T 1% /mnt/lscratch[MDT:0]
22-
lscratch-MDT0001_UUID 3.2T 3.8G 3.2T 1% /mnt/lscratch[MDT:1]
23-
lscratch-OST0000_UUID 57.4T 16.7T 40.2T 30% /mnt/lscratch[OST:0]
24-
lscratch-OST0001_UUID 57.4T 18.8T 38.0T 34% /mnt/lscratch[OST:1]
25-
lscratch-OST0002_UUID 57.4T 17.6T 39.3T 31% /mnt/lscratch[OST:2]
26-
lscratch-OST0003_UUID 57.4T 16.6T 40.3T 30% /mnt/lscratch[OST:3]
27-
lscratch-OST0004_UUID 57.4T 16.5T 40.3T 30% /mnt/lscratch[OST:4]
28-
lscratch-OST0005_UUID 57.4T 16.5T 40.3T 30% /mnt/lscratch[OST:5]
29-
lscratch-OST0006_UUID 57.4T 16.3T 40.6T 29% /mnt/lscratch[OST:6]
30-
lscratch-OST0007_UUID 57.4T 17.0T 39.9T 30% /mnt/lscratch[OST:7]
31-
lscratch-OST0008_UUID 57.4T 16.8T 40.0T 30% /mnt/lscratch[OST:8]
32-
lscratch-OST0009_UUID 57.4T 13.2T 43.6T 24% /mnt/lscratch[OST:9]
33-
lscratch-OST000a_UUID 57.4T 13.2T 43.7T 24% /mnt/lscratch[OST:10]
34-
lscratch-OST000b_UUID 57.4T 13.3T 43.6T 24% /mnt/lscratch[OST:11]
35-
lscratch-OST000c_UUID 57.4T 14.0T 42.8T 25% /mnt/lscratch[OST:12]
36-
lscratch-OST000d_UUID 57.4T 13.9T 43.0T 25% /mnt/lscratch[OST:13]
37-
lscratch-OST000e_UUID 57.4T 14.4T 42.5T 26% /mnt/lscratch[OST:14]
38-
lscratch-OST000f_UUID 57.4T 12.9T 43.9T 23% /mnt/lscratch[OST:15]
39-
40-
filesystem_summary: 919.0T 247.8T 662.0T 28% /mnt/lscratch
21+
es7k2-MDT0000_UUID 5.4T 10.0G 5.3T 1% /mnt/scratch[MDT:0]
22+
es7k2-MDT0001_UUID 5.5T 11.4G 5.4T 1% /mnt/scratch[MDT:1]
23+
es7k2-OST0000_UUID 185.0T 52.0T 123.7T 30% /mnt/scratch[OST:0]
24+
es7k2-OST0001_UUID 185.0T 52.0T 123.7T 30% /mnt/scratch[OST:1]
25+
es7k2-OST0002_UUID 185.0T 52.0T 123.7T 30% /mnt/scratch[OST:2]
26+
es7k2-OST0003_UUID 185.0T 51.9T 123.8T 30% /mnt/scratch[OST:3]
27+
es7k2-OST0004_UUID 185.0T 52.0T 123.7T 30% /mnt/scratch[OST:4]
28+
es7k2-OST0005_UUID 185.0T 51.9T 123.8T 30% /mnt/scratch[OST:5]
29+
es7k2-OST0006_UUID 185.0T 52.0T 123.7T 30% /mnt/scratch[OST:6]
30+
es7k2-OST0007_UUID 185.0T 51.9T 123.8T 30% /mnt/scratch[OST:7]
31+
32+
filesystem_summary: 1.4P 415.5T 989.9T 30% /mnt/scratch
4133
```
4234

43-
4435
### File striping
4536

4637
**[File striping](https://en.wikipedia.org/wiki/Data_striping)** permits to increase the throughput of operations by taking advantage of several OSSs and OSTs, by allowing one or more clients to read/write different parts of the same file in parallel. On the other hand, striping small files can decrease the performance.
@@ -63,8 +54,7 @@ _Note_: With regards `stripe_offset` (the index of the OST where the first strip
6354

6455
* Use the `lfs getstripe` command for getting the stripe parameters.
6556
* Use `lfs setstripe` for setting the stripe parameters to get optimal I/O performance. The correct stripe setting depends on your needs and file access patterns.
66-
- Newly created files and directories will inherit these parameters from their parent directory. However, the parameters cannot be changed on an existing file.
67-
57+
* Newly created files and directories will inherit these parameters from their parent directory. However, the parameters cannot be changed on an existing file.
6858

6959
```console
7060
$ lfs getstripe dir|filename
@@ -88,7 +78,8 @@ $ lfs getstripe $SCRATCH
8878
stripe_count: -1 stripe_size: 1048576 pattern: raid0 stripe_offset: -1
8979
```
9080

91-
In this example, we view the current stripe setting of the `$SCRATCH` directory. The stripe count is changed to all OSTs and verified.
81+
In this example, we view the current stripe setting of the `$SCRATCH` directory.
82+
The stripe count is changed to all OSTs and verified.
9283
All files written to this directory will be striped over the maximum number of OSTs (16).
9384
Use `lfs check osts` to see the number and status of active OSTs for each filesystem on the cluster. Learn more by reading the man page:
9485

@@ -138,7 +129,6 @@ stripe_count: 6 stripe_size: 4194304 stripe_offset: -1
138129

139130
Large stripe size allows each client to have exclusive access to its own part of a file. However, it can be counterproductive in some cases if it does not match your I/O pattern. The choice of stripe size has no effect on a single-stripe file.
140131

141-
142132
Note that these are simple examples, the optimal settings defer depending on the application (concurrent threads accessing the same file, size of each write operation, etc).
143133

144134
## Lustre Best practices

0 commit comments

Comments
 (0)