Skip to content

Commit aec35cf

Browse files
AgentKumaraAgentKumara
authored
Update Freezer docs (#789)
2 parents 407e90e + 4a7e8d4 commit aec35cf

File tree

6 files changed

+446
-295
lines changed

6 files changed

+446
-295
lines changed

‎docs/Storage/Long_Term_Storage/.pages.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
11
nav:
22
- Freezer_long_term_storage.md
3+
- Configuring_s3cmd.md
4+
- Freezer_Guide.md
5+
- Other_Useful_Commands.md
6+
- Troubleshooting.md
37
- ...
48
- Release Notes freezer.nesi.org.nz: Release_Notes_freezer-nesi-org-nz

‎docs/Storage/Long_Term_Storage/Configuring_s3cmd.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
created_at: 2025-10-01
3+
description: "Freezer s3cmd configuration"
4+
tags:
5+
- Freezer
6+
- storage
7+
- config
8+
- s3cmd
9+
---
10+
11+
To use Freezer and to configure `s3cmd`, you will need a current Freezer allocation. To apply for an allocation please go to [MyNeSI](https://my.nesi.org.nz/).
12+
13+
Currently Freezer is only available via specific access points, HPC3 and Mahuika. We are currently completing security hardening prior to opening Freezer to wider access.
14+
15+
### Configure s3cmd
16+
17+
You will need to configure the `s3cmd` tool before you use it for the first time. Configuring the `s3cmd` allows for user credentials and default buckets to be remembered. This will only need to be done once.
18+
19+
```sh
20+
s3cmd --configure
21+
```
22+
23+
Enter the following details when prompted in the terminal:
24+
25+
`Access Key`: Your NeSI user ID
26+
27+
`Secret Key`: This is the code from the 1-time link in your Freezer allocation email. Please let us know if you need to <a href="mailto:[email protected]?subject=Reset%20Freezer%20Secret%20Key">reset this key</a>.
28+
29+
Please copy and paste the sections in <span style="color:blue">blue</span>.
30+
31+
<pre><code>Enter new values or accept defaults in brackets with Enter.
32+
Refer to user manual for detailed description of all options.
33+
Access key and Secret key are your identifiers for AWS S3. Leave them empty for using the env variables.
34+
Access Key: <span style="color:green"><b>User ID</b></span>
35+
Secret Key: <span style="color:green"><b>Your Freezer Secret Key</b></span>
36+
Default Region: <span style="color:blue"><b>akl-1</b></span>
37+
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
38+
S3 Endpoint: <span style="color:blue"><b>freezer.nesi.org.nz:7070</b></span>
39+
40+
Use "%(bucket)s.s3.amazonaws.com" to the target AWS S3. "%(bucket)s" and "%(location)s" vars can be used
41+
if the target S3 system supports dns based buckets.
42+
DNS-style bucket+hostname:port template for accessing a bucket: <span style="color:blue"><b>210.7.37.122:7070</b></span>
43+
Encryption password is used to protect your files from reading
44+
by unauthorized persons while in transfer to S3
45+
Encryption password: <span style="color:green"><b>Leave blank, </b>press &lt;Enter&gt;</span>
46+
Path to GPG program [/usr/bin/gpg]: <span style="color:green"><b>Leave blank, </b>press &lt;Enter&gt;</span>
47+
48+
When using secure HTTPS protocol all communication with AWS S3
49+
servers is protected from 3rd party eavesdropping. This method is
50+
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
51+
Use HTTPS protocol: <span style="color:blue"><b>Yes</b></span>
52+
On some networks all internet access must go through a HTTP proxy.
53+
Try setting it here if you can't connect to S3 directly
54+
HTTP Proxy server name: <span style="color:green"><b>Leave blank, </b>press &lt;Enter&gt;</span>
55+
</code></pre>
56+
57+
You will then be presented with a summary.
58+
<pre><code>New settings:
59+
Access Key: User ID
60+
Secret Key: Your Freezer Secret Key
61+
Default Region: akl-1
62+
S3 Endpoint: freezer.nesi.org.nz:7070
63+
DNS-style bucket+hostname:port template for accessing a bucket: 210.7.37.122:7070
64+
Encryption password:
65+
Path to GPG program: /usr/bin/gpg
66+
Use HTTPS protocol: True
67+
HTTP Proxy server name:
68+
HTTP Proxy server port: 0
69+
</code></pre>
70+
71+
Press `Y` to confirm.
72+
73+
<pre><code>Test access with supplied credentials? [Y/n] <span style="color:blue"><b>Y</b></span>
74+
Please wait, attempting to list all buckets...
75+
Success. Your access key and secret key worked fine :-)
76+
77+
Now verifying that encryption works...
78+
Not configured. Never mind.
79+
80+
Save settings? [y/N] <span style="color:blue"><b>y</b></span>
81+
Configuration saved to '/home/&lt;user_id&gt;/.s3cfg'
82+
</code></pre>

‎docs/Storage/Long_Term_Storage/Freezer_Guide.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
---
2+
created_at: 2025-10-01
3+
description: "Freezer Quick Start"
4+
tags:
5+
- Freezer
6+
- storage
7+
---
8+
!!! info s3cmd configuration required
9+
Please ensure you have [configured](Configuring_s3cmd.md) the s3cmd tool.
10+
11+
12+
## Using s3cmd tool to interact with Freezer
13+
14+
Freezer uses the AWS S3 standard as a protocol for temporarily hosting data prior to writing it to tape.
15+
All the data is stored in buckets temporarily before being written to tape - this is similar to a folder on a filesystem, but designed for scalable storage.
16+
17+
Freezer has two types of data storage classes:
18+
19+
- Glacier: data that is on tape
20+
- Standard: data that is in the S3 bucket
21+
22+
Please note that your bucket has the same name as your Freezer allocation. If you have forgotten the name of your bucket, please <a href="mailto:[email protected]?subject=Forgot%20my%20Freezer%20bucket%20name">email us</a> and let us know which project this is for.
23+
24+
25+
26+
## List contents and buckets
27+
28+
### List contents of a bucket
29+
30+
List all objects in a bucket.
31+
32+
```sh
33+
s3cmd ls -r -l -H s3://<freezer_bucket>/
34+
```
35+
36+
This can also be used to list all the objects in path.
37+
38+
!!! warning
39+
The listing only shows the storage class when using the `-l` option. This is important to determine whether the data is available or must be restored from tape first.
40+
41+
### List all buckets
42+
43+
List all objects in all buckets (only for project owners)
44+
45+
```sh
46+
s3cmd la
47+
```
48+
49+
### Storage usage by specific bucket
50+
51+
```sh
52+
s3cmd du -H s3://<freezer_bucket>
53+
7G 1781 objects s3://<freezer_bucket>/
54+
```
55+
56+
`s3cmd du -H` without specifying a bucket is only available for project owners.
57+
58+
!!! warning
59+
60+
If you have a large number files the `s3cmd du` command will fail. If you wish to receive information from `s3cmd du` we advise using a compression command such as `tar` to reduce the total number of files before adding them to Freezer.
61+
62+
## Uploading objects
63+
### Put objects
64+
65+
To transfer files/folders to S3 gateway to be archived. `cd` into where the file/folder is on Mahuika and then use `s3cmd put`.
66+
67+
!!! warning
68+
69+
If you have files larger than 10TB you may need to change chunk size, please see the [Large files and Chunk Size](Other_Useful_Commands.md) section for details
70+
71+
```sh
72+
s3cmd put --verbose your_file s3://<freezer_bucket>/your_directory/your_file
73+
```
74+
75+
``` out
76+
INFO: Cache file not found or empty, creating/populating it.
77+
INFO: Compiling list of local files...
78+
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
79+
INFO: Summary: 1 local files to upload
80+
upload: 'your_file' -> 's3://<freezer_bucket>/your_directory/your_file' [1 of 1]
81+
172202 of 172202 100% in 0s 920.89 KB/s done
82+
```
83+
84+
or folders
85+
86+
```sh
87+
s3cmd put --recursive --verbose yourfolder s3://<freezer_bucket>/your_directory/your_folder/
88+
```
89+
90+
``` out
91+
INFO: Cache file not found or empty, creating/populating it.
92+
INFO: Compiling list of local files...
93+
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
94+
INFO: Summary: 1 local files to upload
95+
upload: 'yourfolder/your_file' -> 's3://<freezer_bucket>/your_directory/your_folder/yourfolder/yourfile' [1 of 1]
96+
172202 of 172202 100% in 0s 1691.71 KB/s done
97+
```
98+
99+
Once the upload is successful, as signalled by the 'done' your files/folders stored as objects will automatically be archived to tape by the Freezer service. No further user action is needed. Do not delete your files from the bucket unless you do not wish for them to be archived to tape. They will remain in the bucket at least until they are copied to tape and likely for some time afterwards until the cache becomes too full and older files are removed.
100+
101+
Partially uploaded files will be deleted automatically.
102+
103+
### Synchronise data
104+
105+
Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options). If you wish to have additional informative output, please use the `--verbose` flag as well.
106+
107+
```sh
108+
s3cmd sync yourfolder s3://<freezer_bucket>/your_directory/your_folder/
109+
```
110+
111+
### Preview or dry-run
112+
113+
Use any of the `s3cmd` options with `-n, --dry-run`to preview the action.
114+
115+
Only shows what should be uploaded or downloaded but doesn't actually do it. May still perform S3 requests to get bucket listings and other information though (only for file transfer commands).
116+
117+
## Restoring objects
118+
### List objects before restore
119+
120+
List contained objects/files/folders:
121+
122+
```sh
123+
s3cmd ls -l -H s3://<freezer_bucket>/your_directory/your_folder/
124+
```
125+
126+
``` out
127+
DIR s3://<freezer_bucket>/your_directory/your_folder/MY_TEST/
128+
2025-06-16 23:13 10G 8add0bf4f023e3dbd36a329d1eae5bbd-684 STANDARD s3://<freezer_bucket>/your_directory/your_folder/10G_test.file
129+
2025-06-16 23:30 10G 8add0bf4f023e3dbd36a329d1eae5bbd-684 STANDARD s3://<freezer_bucket>/your_directory/your_folder/10G_copy.file
130+
2025-06-17 01:26 0 d41d8cd98f00b204e9800998ecf8427e STANDARD s3://<freezer_bucket>/your_directory/your_folder/1test.txt
131+
```
132+
133+
or all objects recursive -r or --recursive
134+
135+
```sh
136+
s3cmd ls -r -l -H s3://<freezer_bucket>/your_directory/your_folder/
137+
```
138+
139+
``` out
140+
2025-06-16 23:13 10G 8add0bf4f023e3dbd36a329d1eae5bbd-684 STANDARD s3://<freezer_bucket>/your_directory/your_folder/10G_test.file
141+
2025-06-16 23:30 10G 8add0bf4f023e3dbd36a329d1eae5bbd-684 STANDARD s3://<freezer_bucket>/your_directory/your_folder/10G_copy.file
142+
2025-06-17 01:31 14 95b28899a460dd8971705dfcd0f5f0d4 STANDARD s3://<freezer_bucket>/your_directory/your_folder/MY_TEST/annotations/3/4/test3.txt
143+
2025-06-17 01:31 14 e76c3a8939fb031bab02a89f6fab520b STANDARD s3://<freezer_bucket>/your_directory/your_folder/MY_TEST/annotations/3/test2.txt
144+
2025-06-17 01:31 14 be2520c884c1be55bab187374a982b12 STANDARD s3://<freezer_bucket>/your_directory/your_folder/MY_TEST/raw_data/test1.txt
145+
2025-06-17 01:26 0 d41d8cd98f00b204e9800998ecf8427e STANDARD s3://<freezer_bucket>/your_directory/your_folder/test/test.txt
146+
```
147+
148+
### Restore from tape
149+
150+
It is necessary to restore data from the tape (Glacier) prior to retrieving it. To restore file from Glacier storage:
151+
152+
```sh
153+
s3cmd restore --recursive --verbose s3://<freezer_bucket>/your_directory/data_folder/
154+
```
155+
156+
``` out
157+
INFO: Retrieving list of remote files for s3://<freezer_bucket>/your_directory/your_folder/ ...
158+
INFO: Summary: Restoring 6 remote files for 1 days at Standard priority
159+
restore: 's3://n<freezer_bucket>/your_directory/your_folder/10G.file'
160+
restore: 's3://<freezer_bucket>/your_directory/your_folder/10G_copy.file'
161+
restore: 's3://<freezer_bucket>/your_directory/your_folder/MY_TEST/annotations/3/4/test3.txt'
162+
restore: 's3://<freezer_bucket>/your_directory/your_folder/MY_TEST/annotations/3/test2.txt'
163+
restore: 's3://<freezer_bucket>/your_directory/your_folder/MY_TEST/raw_data/test1.txt'
164+
restore: 's3://<freezer_bucket>/your_directory/your_folder/test.txt'
165+
```
166+
167+
By default files will remain in the S3 bucket for 1 day. If longer is required, this can be modified at the time of file restoration:
168+
169+
```sh
170+
s3cmd restore --recursive s3://<freezer_bucket>/your_directory/data_folder/ --restore-days=1
171+
```
172+
173+
### Get objects after restore
174+
175+
!!! info
176+
Data needs to be restored (to storage class `STANDARD`) from the tape (storage class `GLACIER`), before it can be retrieved.
177+
178+
Example to get or download the directory `data_folder` and all contained objects/files/folders:
179+
180+
```sh
181+
s3cmd get --recursive s3://<freezer_bucket>/your_directory/data_folder/
182+
```
183+
184+
This will place the all files and subdirectories in the above `data_folder` into your current directory.
185+
186+
187+
## s3cmd reference
188+
189+
[s3cmd tool](https://s3tools.org/usage)

0 commit comments

Comments
 (0)