I have a text file containing a list of URLs like this:
http://somesite.com/some-article/
https://anothersite.fr/somepage.php?page=something
What I'd like to do is check which URLs return HTTP code 200 (OK) and put them in a list in another file.
How could I accomplish this using terminal? Curl?
Try doing this using bash :
while read url ; do
ret=$(curl -I -s "$url" -o /dev/null -w "%{http_code}\n")
((ret==200)) && echo "$url" >> new_list_URL
done < list_URL
or POSIX mode :
while read url ; do
ret=$(curl -I -s "$url" -o /dev/null -w "%{http_code}\n")
[ $ret -eq 200 ] && echo "$url" >> new_list_URL
done < list_URL
Check man curl
I use this on-liner
time xargs -n 1 -P 20 \
curl -o /dev/null --silent --head --write-out '"%{url_effective}","%{http_code}"\n' \
< "URLs.txt" \
| tee URLs-status_results.csv
And you'll get a beautiful URLs-status_results.csv and the time it takes.
May not be the best way, but you can use this easy-to-understand shell script too:
while read i
do
wget --spider $i > /dev/null 2>1
if [ $? == 0 ]
then
echo $i >> validlist.txt
fi
done
Run this shell script as ./wget_check.sh < urllist.txt, where wget_check.sh is the script name and urllist.txt is the text file holding the URLs.
This script basically runs a wget --spider against all each of the URLs in the list and will add the URL to another file 'validlist.txt' if the URL returns a valid 200 OK response.
The --spider option will 'crawl' through the URLs and will not download any files.
There will be no output produced since the output will be redirected to /dev/null.
Each wget will return a non-zero return value if it does not get a 200 OK response
