find <> -exec curl ... for ftp file upload having problem with url encoding, for filenames with random characters

Question

Use case: I am trying to copy data/everything from my android mobile to laptop on same wifi, maintaining directory structure. I know there are tons of methods, but I love to do it this way (below) and need help. And then do whatever on the laptop with those files.

File samples/directory structure to upload from mobile to laptop ftp server -

dir1
  |--1.mp3
  |--dir2
       |--download  1.jpg
       |--download (2).jpg

How to reproduce: In mobile I have terminal similar to Ubuntu. In laptop I am running ftp server using python. From mobile I am running below command and it works fine, just that directory structure is not maintained and is dumping every file to the root directory of ftp (and I know why bcz there is no path in url).

find dir1 -type f -exec curl -u ftp:ftp --ftp-create-dirs -T {} ftp://192.168.x.x/ \;

Point is, there is no filename/path in url of curl so there is no issue of url encoding. All files as it is uploaded to root directory.

So to maintain sub directory structure I have to do below url call.

find dir1 -type f -exec curl -u ftp:ftp --ftp-create-dirs -T {} ftp://192.168.x.x/{} \;

Now it will throw error for filenames having spaces, (, ) or anything like that due to url encoding. For example for "download 1.jpg" and "download (2).jpg"

curl: (3) URL using bad/illegal format or missing URL

Stuffs I have tried so far (echo I have used below to show the variables values) -

find dir1 -type f -exec bash -c 'y=$(echo "$1"); fname=$(echo $y | sed "s/ /%20/g"); echo $y;echo $fname;echo ""; curl -u ftp:ftp --ftp-create-dirs -T $y ftp://192.168.x.x/$fname' bash {} \;

Output -

dir1/1.mp3
dir1/1.mp3

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

dir1/dir2/download 1.jpg
dir1/dir2/download%201.jpg

curl: Can't open 'dir1/dir2/download'
curl: try 'curl --help' for more information
curl: (26) Failed to open/read local data from file/application

dir1/dir2/download (2).jpg
dir1/dir2/download%20(2).jpg

curl: (3) URL using bad/illegal format or missing URL

So you can see the problem I am having is with url encoding. Even I have done encoding for space it is not working. And there will be much more random characters.

I thought to use --data-urlencoding of curl but that won't allow -G with -T, or -T with --data. Plus there will be a "?" string added this way if it was going to work which is not my use case.

The usual way to solve this is to write a compressed tar archive and then just transfer that. One file, one transaction, plenty of bandwidth saved by the compression, especially when it can squeeze down information which is repeated across files. — tripleee
– tripleee, Commented Aug 17, 2024 at 8:49
@tripleee - That is one good approach. But 128/256 GB of mobile internal storage compress it and transfer, little big file. First, already phone's internal storage may be full to accommodate that. Second, slight glitch in big compress file upload due to any reason (network hiccup or anything), you dont have anything in hand as partial compress file decompression wont go like normal decompression, do it again. One thing that can be optimized here is, for every single file a new ftp connection is being made which makes it slow. curl already has a feature of -T "{file,file2...}" to use. — iamsmith41
– iamsmith41, Commented Aug 17, 2024 at 9:14
With tar -cJf - dir1 | ssh remote tar -xJf - you don't have a physical file at any point, just a stream; though indeed, if the network is unreliable, that's failure-prone. — tripleee
– tripleee, Commented Aug 17, 2024 at 9:23
but looks like by default mobile terminal is not having ssh client, need to find one with ssh client inbuilt. neither do have nc — iamsmith41
– iamsmith41, Commented Aug 17, 2024 at 10:36

iamsmith41 · Accepted Answer · 2024-08-17 07:09:40Z

Ok I did it. Since it was mobile I wanted to keep it free from any other package/tool and to basics. sed helped me in url encoding. Also this-page of github was helpful in gathering those mappers in one place.

I had to test them and few were causing problem, bcz of my code how it is written. e.g. "s/\'/%27/g" "s/\\/%5c/g" I did not bother to fix them as my job was done without these.

This was causing to put $ at end of every uploaded file - "s/\$/%24/g"

There was one mistake I was doing in curl, I was not putting double quotes here and few errors were bcz of this and I was thinking curl is bad -T "$y"

This is hot it worked.

find dir1 -type f -exec bash -c 'y=$(echo "$1"); fname=$(echo $y | sed -e "s/%/%25/g" -e "s/ /%20/g" -e "s/!/%21/g" -e "s/#/%23/g" -e "s/(/%28/g" -e "s/)/%29/g" -e "s/+/%2b/g" -e "s/,/%2c/g" -e "s/-/%2d/g" -e "s/:/%3a/g" -e "s/;/%3b/g" -e "s/?/%3f/g" -e "s/@/%40/g" -e "s/\&/%26/g" -e "s/\*/%2a/g" -e "s/\./%2e/g" -e "s/\//%2f/g" -e "s/\[/%5b/g" -e "s/\]/%5d/g" -e "s/\^/%5e/g" -e "s/_/%5f/g" -e "s/{/%7b/g" -e "s/|/%7c/g" -e "s/}/%7d/g" -e "s/~/%7e/g" -e "s/\"/%22/g" -e "s/\`/%60/g"); echo $y;echo $fname;echo ""; curl -u ftp:ftp --ftp-create-dirs -T "$y" ftp://192.168.x.x/$fname ; echo ""' bash {} \;

Basically my mobile command is like this - full mobile files backup maintaining directory structure. Suppose your phone screen is broken, it is not booting properly bcz of storage full, it is stuck at boot bcz of multiple reasons, etc. etc. ADB helps.

cd /

find sdcard -type f -exec bash -c 'y=$(echo "$1"); fname=$(echo $y | sed -e "s/%/%25/g" -e "s/ /%20/g" -e "s/!/%21/g" -e "s/#/%23/g" -e "s/(/%28/g" -e "s/)/%29/g" -e "s/+/%2b/g" -e "s/,/%2c/g" -e "s/-/%2d/g" -e "s/:/%3a/g" -e "s/;/%3b/g" -e "s/?/%3f/g" -e "s/@/%40/g" -e "s/\&/%26/g" -e "s/\*/%2a/g" -e "s/\./%2e/g" -e "s/\//%2f/g" -e "s/\[/%5b/g" -e "s/\]/%5d/g" -e "s/\^/%5e/g" -e "s/_/%5f/g" -e "s/{/%7b/g" -e "s/|/%7c/g" -e "s/}/%7d/g" -e "s/~/%7e/g" -e "s/\"/%22/g" -e "s/\`/%60/g"); echo $y;echo $fname;echo ""; curl -u ftp:ftp --ftp-create-dirs -T "$y" ftp://192.168.x.x/$fname ; echo ""' bash {} \;

Collectives™ on Stack Overflow

find <> -exec curl ... for ftp file upload having problem with url encoding, for filenames with random characters

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related