PHP while loop to process faster , maybe with PHP worker implementation [closed]

Question

Closed. This question needs to be more focused. It is not currently accepting answers.

Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened.

Closed 5 years ago.

Improve this question

I would like to get idea on how to process a while loop which consist of more than 3000 rows

while ($row = mysql_fetch_array($result)) {
    if $row["Status"] == 1 {
        //Curl 3PP and Update DB 
    } else {
        //Curl 3PP and Update DB
    }
}

Since the row of the result is huge and the while loop is taking around 1 hour to complete process it. Is there any suggestion to expedite the process?

I was thinking of having another PHP to spawn workers to process the CURL and UpdateDB part. Something like this :

while ($row = mysql_fetch_array($result)) {
    if ($row["Status"] == 1) {
        exec("php curl_update.php?database=success");
    } else {
        exec("php curl_update.php?database=fail");
    }
}

Is it a good idea or is there any other ways to fulfill the following requirement using PHP or any other library?

In the curl command, i'll be sending transactionID to a 3rd party API to get the status whether it's fail or success. Latency is around 1-5 seconds

What you should do is switch over to PDO and learn foreach loops. — Thrallix
– Thrallix, Commented Jul 21, 2020 at 12:00
It is likely the CURL that is taking so much time. Can you give us more info about what you're trying to achieve? Also, = probably should be ==. — waterloomatt
– waterloomatt, Commented Jul 21, 2020 at 12:00
PDO and foreach will not speed things up at all. It is the fact that you're sending out 2000 http requests via CURL. Can you break the CURL request down for us a bit more? What are you sending? Do you really need to send it for each row or can you send a single request? — waterloomatt
– waterloomatt, Commented Jul 21, 2020 at 12:07

waterloomatt · Accepted Answer · 2020-07-21 13:18:12Z

1

I think you have a few options now including @DarkBee's exec('php ...' > /dev/null 2>/dev/null &'); and also curl_multi_init.

Here's a quick example showing the time difference between initiating 100 requests using curl_multi_init Vs. the traditional curl_exec.

<?php
$time_start = microtime(true);

$nodes = array_fill(0, 100, 'https://www.google.ca');

// Total time: 1.8 seconds
multiHandle($nodes);

// Vs.

// Total time: 13.5 seconds
//traditional($nodes);

$time_end = microtime(true);
$execution_time = ($time_end - $time_start);
echo sprintf('<b>Total Execution Time:</b> %s secs', round($execution_time, 2));

/*****************************************************************/

/**
 * Taken from: https://stackoverflow.com/a/9311112/296555
 */
function multiHandle($nodes)
{
    $node_count = count($nodes);

    $curl_arr = array();
    $master = curl_multi_init();

    for ($i = 0; $i < $node_count; $i++) {
        $url = $nodes[$i];
        $curl_arr[$i] = curl_init($url);
        curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
        curl_multi_add_handle($master, $curl_arr[$i]);
    }

    do {
        curl_multi_exec($master, $running);
    } while ($running > 0);

    for ($i = 0; $i < $node_count; $i++) {
        curl_multi_getcontent($curl_arr[$i]);
    }
}

/**
 * Taken from: https://www.php.net/manual/en/curl.examples.php#88055
 */
function traditional($nodes)
{
    foreach ($nodes as $destination) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $destination);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_exec($ch);
        curl_close($ch);
    }
}

answered Jul 21, 2020 at 13:18

waterloomatt

3,7731 gold badge23 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jeeva Suriyaa Over a year ago

Thank you very much @waterloomatt, Will look into it

DarkBee Over a year ago

@JeevaSuriyaa Just wondering for an update

Collectives™ on Stack Overflow

PHP while loop to process faster , maybe with PHP worker implementation [closed]

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related