Leo at 00:05, 25 August 2023

2023-08-25T00:05:09Z

← Older revision		Revision as of 00:05, 25 August 2023
Line 1:		Line 1:
	AWS S3 and storage using a S3-compatible layer can be downloaded from using HTTP.		AWS S3 and storage using a S3-compatible layer can be downloaded from using HTTP.

	== ~~Introduction~~ ==		== The HTTP Authorization header ==

	=== ~~Downloading from a protected S3 bucket~~ ===		=== Generating the Authorization header ===
	To download from a protected S3 bucket, you will need to pass in an Authorization HTTP header. This header can be generated using both the <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> along with the current date and time.		To download from a protected S3 bucket, you will need to pass in an Authorization HTTP header. This header can be generated using both the <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> along with the current date and time.

Line 20:		Line 20:

	=== Scripting it with Bash ===		=== Scripting it with Bash ===
	To download from a S3 bucket, ~~you can~~ use the following Bash script ~~which generates~~ the ~~appropriate headers~~ and ~~passes it~~ to Curl or Aria2. The neat thing with using Aria2 is that you can have it download files in multiple parts to speed up the file retrieval.		To download from a protected S3 bucket using Curl or Aira2, use the following Bash script. The Authorization header is created based on the configurable variables at the top of the script and are then passed to either Curl or Aria2. The neat thing with using Aria2 is that you can have it download files in multiple parts to speed up the file retrieval.
	{{Highlight		{{Highlight
	\| code = # Configurables		\| code = # Configurables

Leo: Created page with "AWS S3 and storage using a S3-compatible layer can be downloaded from using HTTP. == Introduction == === Downloading from a protected S3 bucket === To download from a protected S3 bucket, you will need to pass in an Authorization HTTP header. This header can be generated using both the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` along with the current date and time. When creating the HTTP request, the `Date` and `Authorizati..."`


2023-08-25T00:00:56Z
Created page with "AWS S3 and storage using a S3-compatible layer can be downloaded from using HTTP.   == Introduction ==  === Downloading from a protected S3 bucket === To download from a protected S3 bucket, you will need to pass in an Authorization HTTP header. This header can be generated using both the <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> along with the current date and time.   When creating the HTTP request, the <code>Date</code> and <code>Authorizati..."
New page
AWS S3 and storage using a S3-compatible layer can be downloaded from using HTTP. 



== Introduction ==



=== Downloading from a protected S3 bucket ===

To download from a protected S3 bucket, you will need to pass in an Authorization HTTP header. This header can be generated using both the <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> along with the current date and time. 



When creating the HTTP request, the <code>Date</code> and <code>Authorization</code> HTTP headers must be sent containing the following:



* <code>Date</code> contains the current timestamp in a RFC2616 compliant format. Eg: <code>date -u '+%a, %e %b %Y %H:%M:%S +0000'</code>

* <code>Authorization</code> header should contain 'AWS <code>AWS_Access_Key_Id</code>:<code>Signature</code>', where <code>Signature</code> is a base64 encoded string containing the <nowiki>RFC 2104</nowiki> HMAC-SHA1 of the following items:{{Highlight

| code = METHOD\n

MD5\n

Date\n

PATH

| lang = text

}}



More on this from AWS's documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html#ConstructingTheAuthenticationHeader



=== Scripting it with Bash ===

To download from a S3 bucket, you can use the following Bash script which generates the appropriate headers and passes it to Curl or Aria2. The neat thing with using Aria2 is that you can have it download files in multiple parts to speed up the file retrieval.

{{Highlight

| code = # Configurables

Url="s3://mybucket/example.bin"

Server="s3-backend-url.example.com"

AWS_SECRET_ACCESS_KEY=""

AWS_ACCESS_KEY_ID=""



# Build the headers

Filename=$(basename "$Url")

Path="${Url:4}"

Method="GET"

MD5Sum=""

Timestamp="$(date -u '+%a, %e %b %Y %H:%M:%S +0000')"

printf -v StringToSign "%s\n%s\n\n%s\n%s" "$Method" "$MD5Sum" "$Timestamp" "$Path"

Signature=$(echo -n "$StringToSign" {{!}} openssl sha1 -binary -hmac "${AWS_SECRET_ACCESS_KEY}" {{!}} openssl base64)

Authorization="AWS ${AWS_ACCESS_KEY_ID}:${Signature}"



# Use Curl to download

curl -f -o "$Filename" \

	 -H "Date: $Timestamp" \

	 -H "Authorization: $Authorization" \

	 "http://$Server$Path"



# Use Aria2 to download with 8x concurrency

aria2c -x8 -o "$Filename" \

	--auto-file-renaming=false --allow-overwrite=true \

	--header="Date: $Timestamp" \

	--header="Authorization: $Authorization" \

	 "http://$Server$Path"

| lang = bash

}}

Downloading from S3 - Revision history

Leo at 00:05, 25 August 2023