Initial commit

2025-06-30 09:05:05 +00:00
commit 46f6cc100b
1231 changed files with 278432 additions and 0 deletions
--- a/research/utils/s3.md
+++ b/research/utils/s3.md
@@ -0,0 +1,89 @@
+# How to Download Needed Data from S3
+
+## Install AWS CLI v2
+
+Install AWS CLI v2 by following the instructions at https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.
+
+
+curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+unzip awscliv2.zip
+sudo ./aws/install
+
+## Configure SSO
+
+Run the following command:
+
+```console
+aws configure sso
+```
+
+Example output:
+```
+(retrieval_scaling) (base) ➜  retrieval_scaling git:(main) ✗ aws configure sso
+SSO session name (Recommended): yichuan
+SSO start URL [None]:  https://ucberkeley.awsapps.com/start#/
+SSO region [None]: us-west-2
+SSO registration scopes [sso:account:access]:
+Attempting to automatically open the SSO authorization page in your default browser.
+If the browser does not open or you wish to use a different device to authorize this request, open the following URL:
+
+https://oidc.us-west-2.amazonaws.com/authorize?response_type=code&client_id=i3YtHZTRneXEIApSyvdgSHVzLXdlc3QtMg&redirect_uri=http%3A%2F%2F127.0.0.1%3A37899%2Foauth%2Fcallback&state=5f52320e-0929-4e44-83c7-f6bd9b492010&code_challenge_method=S256&scopes=sso%3Aaccount%3Aaccess&code_challenge=HYnZ4Pc-tqI8CdJb6qEAR0LjI1_UjN-zln26lqJKeL8
+The only AWS account available to you is: 976193267581
+Using the account ID 976193267581
+There are 2 roles available to you.
+Using the role name "UCB-FederatedAdmins"
+Default client Region [None]:
+CLI default output format (json if not specified) [None]:
+Profile name [UCB-FederatedAdmins-976193267581]:
+To use this profile, specify the profile name using --profile, as shown:
+
+aws sts get-caller-identity --profile UCB-FederatedAdmins-976193267581
+```
+
+After configuration, you must include `--profile UCB-FederatedAdmins-976193267581` with each AWS operation to use the SSO credentials.
+
+## Refresh the SSO
+
+If you encounter the error `Error when retrieving token from sso: Token has expired and refresh failed`, simply run the SSO configuration command again.
+
+## Download S3 Data
+
+All data is stored in `s3://retrieval-scaling-out`, which includes 4 directories:
+- embeddings/
+- examples/
+- indices/
+- passages/
+
+Download the data using AWS CLI:
+
+```console
+aws s3 cp s3://retrieval-scaling-out ~/scaling_out --profile UCB-FederatedAdmins-976193267581
+
+aws s3 cp s3://retrieval-scaling-out/examples/test_c4.jsonl ~/examples/scaling_out --profile UCB-FederatedAdmins-976193267581
+```
+
+### Faster Download Options
+
+To accelerate downloads, you can try the following methods:
+
+Use multipart downloads:
+```console
+aws s3 cp s3://retrieval-scaling-out ~/scaling_out --profile UCB-FederatedAdmins-976193267581 --recursive --multipart-threshold 128MB --multipart-chunksize 512MB
+```
+
+Configure higher concurrency:
+```console
+aws configure set default.s3.max_concurrent_requests 50
+aws configure set default.s3.max_queue_size 10000
+```
+
+Utilize S3 Transfer Acceleration:
+```console
+aws s3 cp s3://retrieval-scaling-out ~/scaling_out --profile UCB-FederatedAdmins-976193267581 --recursive --endpoint-url https://s3-accelerate.amazonaws.com
+```
+
+Or use alternative tools like `s5cmd`:
+```console
+pip install s5cmd
+s5cmd --profile UCB-FederatedAdmins-976193267581 cp s3://retrieval-scaling-out/* ~/scaling_out/
+```