I have a problem getting the Jason data from multiple URLs generated from the database using multiple curl
If I limit the query to 100 to 500 links, but this problem will not happen, but when the link reaches 1000+, im starts to get a random return from curl_multi_getcontent
it happens.
Multi curl function:
function curlMultiExec ($ nodes) {$ node_count = count ($ nodes); $ Ch_arr = array (); $ Master = curl_multi_init (); For ($ i = 0; $ i & lt; $ node_count; $ i ++) {$ url = $ nodes [$ i] ['url']; $ Ch_arr [$ i] = curl_init ($ url); Curl_setopt ($ ch_arr [$ i], CURLOPT_RETURNTRANSFER, TRUE); Curl_setopt ($ ch_arr [$ i], CURLOPT_BINARYTRANSFER, TRUE); Curl_setopt ($ ch_arr [$ i], CURLOPT_FOLLOWLOCATION, TRUE); Curl_setopt ($ ch_arr [$ i], CURLOPT_AUTOREFERER, TRUE); Curl_setopt ($ ch_arr [$ i], CURLOPT_HEADER, FALSE); Curl_multi_add_handle ($ master, $ ch_arr [$ i]); } $ Running = null; Do {curl_multi_exec ($ master, running $); } While ($ running> gt; 0); $ Obj = array (); ($ I = 0; $ i & lt; $ node_count; $ i ++) {$ item = array ('url' = & gt; $ nodes [$ i] ['url'], 'content' = & gt; ; Curl_multi_getcontent ($ Ch_arr [$ i])); Array_push ($ obj, $ item); } Curl_multi_close (Master $); $ Obj return; }
Currently contains 1,912 URLs in the $ node [$ i] ['url']
.
print_r
using Aare ([0] = & gt; array ([url] = & gt; http: // api Output. Worldbank.org/countries/AFG/indicators/NY.GDP.MKTP.CD?per_page=100&date=1960 Month 4014 & format = json [content] = & gt; [[/ / json data * /}]) [1 ] = & Gt; Array ([URL] => http://api.worldbank.org/countries/ALB/indicators/NY.GDP.MKTP.CD?per_page=100&date=1960&&format=json [content] = & Gt; // -> has sample value value here.) - and some random [content] here also contains zero values. [1191] = & gt; Array ([url] = & gt; Http://api.worldbank.org/countries/ZWE/indicators/NY.GDP.MKTP.CD?per_page=100&date=1960 Month 4014 & format = json [content] = & gt; [[/ / json data *
Please ask me Why does it return random random value, or what is the reason for this behavior or what better approach than this?
Update (2014-02-20); I found a solution here:
The problem is that most implementations of curl_multi await each set of requests to complete before processing them. If there are so many requests to process at one time, they are usually broken into groups which are processed at one time at one time.
The solution is to complete the process as soon as possible on each request. This eliminates wasted CPU cycles from busy waiting.
Based on your attitude, I successfully resolve this. This post may help, just when stopping the same problem.
Cheers!
1000 simultaneous connections will allow you to reach the common limit for the maximum number of open files / sockets: 1024. It may be that what you hit here.
Related questions for changing boundaries:
Comments
Post a Comment