Have Geek Brain

How can I speed up CURL tasks?

I am using CURL to get some data from user accounts. It is first registered and then redirected to another url where the data is.

My stats showed that it took an average of 14 seconds to get some 5 page data. I would like to speed up the process, my questions are:

Can you see how much each step costs? Do you know how I can speed up / improve CURL?

thanks

+2


source to share


5 answers


you can use parallelCurl by Pete Warden. The source is available here http://github.com/petewarden/ParallelCurl . Module allows you to run multiple sets of CURL URLs in parallel in PHP



+3


source


To make the task feel faster, don't run it as part of a web request, run it in the background as a periodic task (cron job).



Upload the answer to disk or database.

+3


source


You cannot speed up the process of getting the page from the server.

You can make the pages smaller so that they load faster. You can increase the processing power on the servers, or the connection between your server and the server that hosts the pages.

If you use the service, what format is the data in? If it is XML, it might be too verbose and cause, for example, a lot of extra kilobytes.

+1


source


split the task into 3 files.

  • to get the list of pages and as the main script (to install on crontab) (main.php)
  • to parse the actual page. (Parse.php)
  • some shell script to handle your second script.

Then in the first file, do something like this:

<?php
$pagelist = get_page_list();//this will retrieve page list using CURL and save each page to some, let say pagelist.txt and return this absolute path.

exec("/path/to/php /your/3rdscript.sh < $pagelist");
?>

And here is your third file:

#!/bin/bash  

while read line
do
    /path/to/php /path/to/your/2ndscript.php -f $line &
done

Note that in the third script (shell script) I am using and (ampersand). This will force the shell to put this particular process in a background process.

On your second script, you can use something like this:

<?php

$pageurl = $argv[2];
//do your curl process to fetch page $pageurl here

Using the above step, you can speed up fetching multiple pages at once.

0


source


Speed ​​up twisting with this opcion

curl_setopt($curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );

Hello

0


source






More articles:


All Articles