89 lines
3.1 KiB
Markdown
89 lines
3.1 KiB
Markdown
# How to Download Needed Data from S3
|
|
|
|
## Install AWS CLI v2
|
|
|
|
Install AWS CLI v2 by following the instructions at https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.
|
|
|
|
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
|
unzip awscliv2.zip
|
|
sudo ./aws/install
|
|
|
|
## Configure SSO
|
|
|
|
Run the following command:
|
|
|
|
```console
|
|
aws configure sso
|
|
```
|
|
|
|
Example output:
|
|
```
|
|
(retrieval_scaling) (base) ➜ retrieval_scaling git:(main) ✗ aws configure sso
|
|
SSO session name (Recommended): yichuan
|
|
SSO start URL [None]: https://ucberkeley.awsapps.com/start#/
|
|
SSO region [None]: us-west-2
|
|
SSO registration scopes [sso:account:access]:
|
|
Attempting to automatically open the SSO authorization page in your default browser.
|
|
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:
|
|
|
|
https://oidc.us-west-2.amazonaws.com/authorize?response_type=code&client_id=i3YtHZTRneXEIApSyvdgSHVzLXdlc3QtMg&redirect_uri=http%3A%2F%2F127.0.0.1%3A37899%2Foauth%2Fcallback&state=5f52320e-0929-4e44-83c7-f6bd9b492010&code_challenge_method=S256&scopes=sso%3Aaccount%3Aaccess&code_challenge=HYnZ4Pc-tqI8CdJb6qEAR0LjI1_UjN-zln26lqJKeL8
|
|
The only AWS account available to you is: 976193267581
|
|
Using the account ID 976193267581
|
|
There are 2 roles available to you.
|
|
Using the role name "UCB-FederatedAdmins"
|
|
Default client Region [None]:
|
|
CLI default output format (json if not specified) [None]:
|
|
Profile name [UCB-FederatedAdmins-976193267581]:
|
|
To use this profile, specify the profile name using --profile, as shown:
|
|
|
|
aws sts get-caller-identity --profile UCB-FederatedAdmins-976193267581
|
|
```
|
|
|
|
After configuration, you must include `--profile UCB-FederatedAdmins-976193267581` with each AWS operation to use the SSO credentials.
|
|
|
|
## Refresh the SSO
|
|
|
|
If you encounter the error `Error when retrieving token from sso: Token has expired and refresh failed`, simply run the SSO configuration command again.
|
|
|
|
## Download S3 Data
|
|
|
|
All data is stored in `s3://retrieval-scaling-out`, which includes 4 directories:
|
|
- embeddings/
|
|
- examples/
|
|
- indices/
|
|
- passages/
|
|
|
|
Download the data using AWS CLI:
|
|
|
|
```console
|
|
aws s3 cp s3://retrieval-scaling-out ~/scaling_out --profile UCB-FederatedAdmins-976193267581
|
|
|
|
aws s3 cp s3://retrieval-scaling-out/examples/test_c4.jsonl ~/examples/scaling_out --profile UCB-FederatedAdmins-976193267581
|
|
```
|
|
|
|
### Faster Download Options
|
|
|
|
To accelerate downloads, you can try the following methods:
|
|
|
|
Use multipart downloads:
|
|
```console
|
|
aws s3 cp s3://retrieval-scaling-out ~/scaling_out --profile UCB-FederatedAdmins-976193267581 --recursive --multipart-threshold 128MB --multipart-chunksize 512MB
|
|
```
|
|
|
|
Configure higher concurrency:
|
|
```console
|
|
aws configure set default.s3.max_concurrent_requests 50
|
|
aws configure set default.s3.max_queue_size 10000
|
|
```
|
|
|
|
Utilize S3 Transfer Acceleration:
|
|
```console
|
|
aws s3 cp s3://retrieval-scaling-out ~/scaling_out --profile UCB-FederatedAdmins-976193267581 --recursive --endpoint-url https://s3-accelerate.amazonaws.com
|
|
```
|
|
|
|
Or use alternative tools like `s5cmd`:
|
|
```console
|
|
pip install s5cmd
|
|
s5cmd --profile UCB-FederatedAdmins-976193267581 cp s3://retrieval-scaling-out/* ~/scaling_out/
|
|
``` |