Getting a piped curl command output to a variable in bash

Question

I'm attempting to monitor an output from a proxy server to troubleshoot issues, in order to do that I've created a bash script which uses curl, but I'm having trouble in getting the output in a variable.

I've tried different ways to escape and to use the variables, either using ticks or parentheses.

I was using this first, but wasn't working properly:

response=$(curl -k -x $i:1234 -U $u:$p -w $outputformat -s -o /dev/null $site | cut -d ' ' -f 2)

Thus changed it, and the code now looks like this:

response=$(curl\ -k\ -x\ $i:1234\ -U\ $u:$p\ -w\ \"HTTP\ Response:\ \%\{http\_code\}\ \-\ URL:\ \%\{redirect\_url\}\\n\"\ -s\ -o\ /dev/null\ $site\ \|\ cut\ -d\ \'\ \'\ -f\ 2)
if [ "$response" -eq 200 ]; then
   ((count200++))
   echo $count200 > $counter_200
else
   ((countx0x++))
   echo $count0x0 > $counter_x0x
fi

8/22 - Updated code as requested, I reverted back the code to what it was before and also following WaltDe's recommendation removed the -o option:

echo "Proxy: $i"
echo "Fetching site: $site"
outputformat="\"HTTP Response: %{http_code} - URL: %{redirect_url}\n\"" 
response=$(curl -k -x $i:$port -U $u:$p -w $outputformat -s $site | cut -d ' ' -f 2)
if [ "$response" -eq 200 ]; then
   ((count200++))
   echo $count200 > $counter_200
else
   ((countx0x++))
   echo $count0x0 > $counter_x0x
fi

This is the error message that I'm seeing:

./ptest.sh: line 80: curl -k -x *********:** -U : -w "HTTP Response: %{http_code} - URL: %{redirect_url}\n" -s -o /dev/null https://www.yahoo.com | cut -d ' ' -f 2: No such file or directory

./ptest.sh: line 82: [: : integer expression expected

8/22 Update - This is the issue I'm having now:

Proxy: ************ Fetching site: https://www.yahoo.com html> id="atomic" (...) shortened for readability - html output (...) Response:: integer expression expected ./ptest.sh: line 83: [: Response:: integer expression expected

Could anyone shed some light as to what I'm doing wrong?

Why did you add all those backslashes? The change you made is strictly for the worse (it's looking for not a file like /usr/bin/curl, but rather for one like /usr/bin/curl -k -x ... with the spaces and arguments as part of the program name). — Charles Duffy
– Charles Duffy, Commented Aug 21, 2019 at 15:26
...please provide us the error message you got from the original, closer-to-working code, rather than only saying it "wasn't working properly" with no further details, and giving us a detailed error only for the code that's broken worse than the original was. — Charles Duffy
– Charles Duffy, Commented Aug 21, 2019 at 15:26
Bash rule of thumb #001: Enclose all dollar signs inside double quotes. Always. Without exceptions. Unregulated dollars provoke crashes. Bash rule of thumb #002: The backtick do not exist. It is an alien glyph that you must excise from your brain and keyboard. U+60 is an unattributed code point. There is but one syntax for command substitution, and this syntax is "$(…)". You’ll automatically be a better Bash programmer by applying these rules rigorously. Safety first! Remember! — Maëlan
– Maëlan, Commented Aug 21, 2019 at 21:29
@CharlesDuffy I added it to prevent the issue I'm having now, where curl is not getting the output format, and I'm getting html output, instead of the output I want. I've added the code and the original output I was getting. — Edmar
– Edmar, Commented Aug 22, 2019 at 15:07

WaltDe · Accepted Answer · 2019-08-22 14:59:52Z

2

You need to remove the -o option that writes the output to a file, in your case /dev/null. Here is your updated code that should work.

$ response=$(curl -k -x $i:1234 -U $u:$p -w $outputformat -s $site | cut -d ' ' -f 2)

Another way to check is to remove pipe and send your STDERR to /dev/null. This will show you what is getting passed to your cut command.

You should see something if all is happy.

$ curl -k -x $i:1234 -U $u:$p  -s $site 2>/dev/null

If you don't get any output, then remove the STDERR redirect and echo the return code.

$ curl -k -x $i:1234 -U $u:$p  -s $site
$ echo $?
0

If the exit code is anything other then 0 you have an error. You can find the error codes in the man page or this site has pretty good write up.

No to get the response code.

$ curl  -w "HTTP Response: %{http_code} - URL: %{redirect_url}\n"  -o /dev/null  -s http://yahoo.com/
HTTP Response: 301 - URL: https://yahoo.com/
$  curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/
301
$ response=$(curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/)
$ echo $response
301
$  curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/ | cut -d' ' -f3
301
$ response=$( curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/ | cut -d' ' -f3)
$ echo $response
301

edited Aug 22, 2019 at 14:59

answered Aug 21, 2019 at 21:31

WaltDe

1,88010 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Edmar Over a year ago

This is what's happening, it's not respecting the output format, and outputing html: Iteration: 1 Proxy: ************ Fetching site: yahoo.com html> id="atomic" <head> (...) shortened for readability (...) Response:: integer expression expected ./ptest.sh: line 83: [: Response:: integer expression expected

WaltDe Over a year ago

Now I get what you are trying for. I updated the answer showing I pulled the response code. Note that the response code is field 3 not 2 with cut using your output format.

Edmar Over a year ago

Hi WaltDe, when I do it from the command line it's working, but from the bash script it doesn't (I get html code). Thanks for pointing the typo on cut.

WaltDe Over a year ago

Are you adding quotes around the $outputresponse variable so it looks like "$outputresponse" ?

Edmar Over a year ago

Ok, I feel dumb now! Thanks WaltDe! That was it all along, and I was just making things worse by changing stuff around!

Collectives™ on Stack Overflow

Getting a piped curl command output to a variable in bash

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related