Hi all, I have a problem with curl, i am using it to create a sitemap generator, when i test it it works on a few thousand links but the site i want to use the sitemap generator on as many thousands when it gets to about 14000 links it starts to go wrong, its only using basic php curl should i do anything more

 

$ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $link);
  curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
   curl_setopt($ch, CURLOPT_TIMEOUT, 900);

 

Thank you

any error?

maybe it's being timed out ?

 

set_time_limit(0);

Could well be the server holding the target website having a wobbler due to lack of resources especially if it's using a database to dynamically generate those pages. Put a delay in the cURL loop, make it sleep a little and slow things down that may help make it more stable.

Thank you both i will give them both a try

While checking the URL's that were being display i noticed that some contained 2 slashes after the domain name so i added the following

  $link=ltrim($link, "/");
  $link="$request_url/$link";

 

This seems to have stopped the problem