AWS CLI command inside bash script can't locate file

Question

I need to move number of files from one server into S3 with some requirements:

Save transfer progress when connection drops
In case of connection drop, next run must continue with non-uploaded files
Run as service -- TODO

So this is what I have, it's not complete, because I'm having an issue with locating a file when running command from within script, and yet command works when running it raw.

How to make script find the $source file from within this script? Also, is there any better ways of doing this?

Command:

aws s3 cp '/var/www/files/folder/file1.mp4' 's3://s3-public/folder/file1.mp4'

Error:

script.sh: line 20: aws s3 cp '/var/www/files/folder/file1.mp4' 's3://s3-public/folder/file1.mp4': No such file or directory

Code:

#!/bin/bash

aws_bucket='s3://s3-public/'
home='/home/user/'
files='/var/www/files/'
input="${home}videos_list.txt"
output="${home}videos_done.txt"

videos_list='()'
videos_done='()'

arrays_hydrate() {
    mapfile -t videos_list < "$input"
    mapfile -t videos_done < "$output"
}

aws_init() {
    for index in "${!videos_list[@]}"
    do
        source="${videos_list[$index]}"
        destination=$( echo "${videos_list[$index]}" | sed -e "s#^$files##" )
        length="${#source}"

        if [ "$length" -ne "0" ]; then
            command="aws s3 cp '${source}' '${aws_bucket}${destination}'"

            echo "$command"
            $( "$command" ) &
        fi
    done
}

init() {
    arrays_hydrate
    aws_init
}

init

exit 0

@ruakh, question about your edit history. The comment in Rev.2 was a mistake or should this post be bit cleaned up? — HelpNeeder
– HelpNeeder, Commented Jun 1, 2017 at 0:50
I rolled back my edit because part of the code dump is necessary in order for the question to make sense. But you should still edit the question per stackoverflow.com/help/mcve; you've posted much more code than is relevant. — ruakh
– ruakh, Commented Jun 1, 2017 at 0:52

ruakh · Accepted Answer · 2017-06-01 00:50:15Z

4

If the command parameter contains aws s3 cp '/var/www/files/folder/file1.mp4' 's3://s3-public/folder/file1.mp4' (as your does), then this:

$( "$command" ) &

is equivalent to this:

$( "aws s3 cp '/var/www/files/folder/file1.mp4' 's3://s3-public/folder/file1.mp4'" ) &

which obviously isn't what you want: you don't have any program named aws s3 cp '/var/www/files/folder/file1.mp4' 's3://s3-public/folder/file1.mp4'.

Instead, you want to call the aws program with the arguments s3, cp, /var/www/files/folder/file1.mp4, and s3://s3-public/folder/file1.mp4:

command=(aws s3 cp "$source" "$aws_bucket$destination")

echo "${command[@]}"
"${command[@]}" &

(where the parameter_name=(value value value value) notation sets the parameter named parameter_name to be an array variable containing the specified values).

answered Jun 1, 2017 at 0:50

ruakh

185k29 gold badges292 silver badges324 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

HelpNeeder Over a year ago

Good note, thank you. I will try it tomorrow morning.

Collectives™ on Stack Overflow

AWS CLI command inside bash script can't locate file

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related