I am trying to copy a file into an S3 bucket, using Python, like so:
cmd = 'aws s3 %s %s' % (filename, bucketname)
os.system(cmd)
It gives me a sh: 1: aws: not found error.
However, using s3cmd works just fine.
Why would s3cmd work, but not aws?
Also, I did which aws and it returned: /home/username/anaconda/bin/aws.
which s3cmd returns: /home/username/anaconda/bin/s3cmd.
Why does one work, but not the other, despite having the same root?
This is because the AWSCLI is not installed in your system. I had also the same problem and I tried installing it with the pip package manager using pip install --user awscli but it didn't work. So I installed it directly in the system as follows:
curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
And the error was solved.
A quick way to troubleshoot the issue is to try the full path on the OS call to see if it is a PATH problem:
cmd = '/path/to/aws s3 %s %s' % (filename, bucketname)
os.system(cmd)
There could be a few reasons why this is a problem, most likely related to the PATH variable (at first guess). However, it might be better to steer away from os.system as noted in the docs (https://docs.python.org/2/library/os.html#os.system) and use some alternative methods.
Using subprocess:
cmd = ['/path/to/aws', 's3', filename, bucketname]
subprocess.Popen(cmd)
Or just use the python AWS client boto3 package. There are many ways, but one quick example from this SO question (How to save S3 object to a file using boto3):
import boto3
s3_client = boto3.client('s3')
s3_client.upload_file(filename, bucketname, filename)
That one is not testable with moto, which can be annoying. Instead if you want to test, you can do something like this:
import boto3
s3_resource = boto3.resource('s3')
with open(filename, 'rb') as f:
binary = f.read()
s3_resource.Bucket(bucketname).put_object(
Key=filename,
Body=binary
)
Install directly from the zip: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
