How to use Bash to parallelise my PHP script

Question

I have a PHP script that loops through some urls. The urls are stored in urls.txt, and it needs to call the php script like php urlcheck.php LINE_FROM_urls.txt

When it reaches a set number of processes, I want it to sleep for 5 seconds, like this

COMCHECK=`ps aux | grep -c php`

while [ $COMCHECK -ge 150 ];do
COMCHECK=`ps aux | grep -c php`
echo "Sleeping"
sleep 5

and then I want it to check again. If its -le 150, run the next url etc.

I have no idea how to do this. My Bash is quite limited, and frankly, the language weirds me out.

codaddict · Accepted Answer · 2010-11-18 06:23:55Z

1

You can try something like:

limit=150  

# read lines from the file urls.txt one by one.
while read line; do

        # check number of running php processes.
        while [ $(ps aux | grep -c php) -ge $limit ];do
                # if it's >= limit..sleep.
                echo "Sleeping"       
                sleep 5
        done

        # limit not reached..run another process..in background. 
        php urlcheck.php  $line &

done < urls.txt

answered Nov 18, 2010 at 6:23

codaddict

457k83 gold badges502 silver badges537 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Rob Over a year ago

This did the trick, and quite well I might add. Also, your code is very nice looking <3 Is there a way to stop it from outputting errors etc from the php file, other than removing error reporting on the php file?

codaddict Over a year ago

To remove the errors spit by PHP you can replace php urlcheck.php $line & by php urlcheck.php $line 2> /dev/null &

Rob Over a year ago

Hey, sorry. One more thing. Is there a way I can make it remove each url from the file as it cycles through?

John Kugelman · Accepted Answer · 2010-11-18 06:23:51Z

1

while read URL; do
    while (( $(ps aux | grep -c php) >= 150 )); do
        sleep 5
    done

    php urlcheck.php "$URL" &
done < urls.txt

# Optional: wait for all PHP processes to finish.
wait

answered Nov 18, 2010 at 6:23

John Kugelman

365k70 gold badges555 silver badges600 bronze badges

Collectives™ on Stack Overflow

How to use Bash to parallelise my PHP script

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related