Hi all, I have built a spider to crawl a web page and follow the links and generate an xml sitemap, i would like to have an array of URLS that i would not like to be included in the map eg mailto:

 

I am using the below foreach loop

$badUrls=array("#" ,"mailto", "javascript:document", "basket");
foreach ($badUrls as $badUrl => $value)
   {

    if (strpos($test, $value) == false)
 {
    $alllinks[] = $test;
			    echo "$test<br />";
 }
   }

 

if i replace $value with "basket" then it works, i just can not make it work with the array

 

any ideas

 

Thank you

replace your foreach by this

foreach ($badUrls as $value)

replace your foreach by this

foreach ($badUrls as $value)

 

There's no need to get rid of the $badUrl variable as the loop functions perfectly with or without it.

 

The following is the code I tested and it works. (I copied the same exact code above but added a $test variable so I could experiment with the loop)

 

$test = "Hey, I have a basket";
$badUrls = array("#" ,"mailto", "javascript:document", "basket");
	 foreach ($badUrls as $badUrl => $value)
	 {
		 if (strpos($test, $value) == false)
	 {
			 $alllinks[] = $test;
				 echo "$test<br />";
	 }
	 }

 

The loop is functioning properly, however when looking at this code, I immediately notice some bugs and these could be responsible for your trouble.

 

The first thing you should be aware of is the fact that in a loosely typed language like PHP, the value of 0 can also mean false. In other words, "if(0 == false)" will evaluate to true and the code in the block will execute. The strpos() function either returns an integer representing the position of the substring found or the boolean false. If strpos() returns 0 (meaning that it found "$value" at the beginning of the string).... then there's your first bug. To fix this, use the identity operator (===) instead of the equality operator (==). For example...

 

if (strpos($test, $value) === false)
	 {
		 $alllinks[] = $test;
			 echo "$test<br />";
	 }

 

The above code is better and if I had to guess, it would be the solution to your issue because "#" and "mailto:" would be at the beginning of a "badlink" string, yea?

 

Second, instead of using the strpos() function, consider using the stripos() function which does the exact same thing and has the same exact syntax, but it's case insensitive. Your list of strings in the array may not always be in lowercase in which case your code will fail. A simple switch to stripos() will fix that.

 

Lastly, if you're trying to add $test to the $alllinks array, use the array_push() function which will append an element to the end of the array. For example:

 

array_push($alllinks, $test);

 

I apologize if I misunderstood what your issue is, but hopefully I helped.

Edited October 2, 201213 yr by Pete Prosper

There's no need to get rid of the $badUrl variable as the loop functions perfectly with or without it.

 

um actually theres no reason to keep the $badUrl variable as u can see its not even being used for anything

 

so id say just do this

 

  foreach ($badUrls as $value)
	 {
		 if (strpos($test, $value) == false)
	 {
			 $alllinks[] = $test;
				 echo "$test<br />";
	 }
	 }

 

the method u were doing with the BadUrl variable would be more suitable for something like this i believe

 

 $badUrls = array(
"key1" => "value1",
"key2" => "value2"
);
foreach ($badUrls as $badUrl => $value)
	 {
		 if (strpos($test, $value) == false)
	 {
			 $alllinks[] = $test;
				 echo "$test<br />";
	 }
	 }

um actually theres no reason to keep the $badUrl variable as u can see its not even being used for anything

 

True, but you could also see that the code the OP posted is incomplete. I didn't modify it because doing so doesn't make a single difference in regard to the OP's problem and I considered the possibility that perhaps $badUrl still serves some obscure purpose for ELITE. Also, I wrote what you quoted under the impression that altering foreach was the proposed solution to the issue.

Edited October 2, 201213 yr by Pete Prosper

Thanks everyone I realsed the my problem was that the foreach loop is already inside a foreach loop, I have changed things around so it now looks like this if you can help with this

foreach ($alllinks as $link)
 {
 foreach($badUrls as $value)
{
	 if (strpos($link, $value)=== false)
			 {
				 $alllinks[] = $link;
				 echo "$link<br />";
			 }
}
}

 

Thanks

Edited October 2, 201213 yr by ELITE

$alllinks[] = $link;

 

What are you trying to do here?

I have it working thank

foreach ($alllinks as $link)
{
    $badUrls=array("basket", "mailto");// array of urls we dont want to include
   $is_accepted = true;
    foreach ($badUrls as $badurl)
    {
	    if (strpos($link, $badurl) !== false)
	    {
			    $is_accepted = false;
	    }
    }
   if ($is_accepted)
	  {
		 $alllinks[] = $link;
		  echo " $link<br />";
    }
}

$alllinks[] = $link;

 

What are you trying to do here?

I am putting all the links into an array so that a loop in the next stage will follow each links i hope to have a drop down option to how deep you wish to crawl a website, so if it finds more links it will follow them, i will post the code later

 

Thanks Everyone learned something today a few +1 are needed

Glad you figured it out.

 

Guess I learned something new too. All this time working with php and I didn't know there was a simpler way of appending elements to an array ("$myArray[] = $newElement") instead of using array_push()

 

Mindblown...

Edited October 2, 201213 yr by Pete Prosper

you guys are making this so much more difficult than it is

 

foreach ($array as $data) with array('val1','val2','val2')

foreach ($array as $k=>$v) with array ('k1'=>'v1','k2'=>'v2','k3'=>'v3')

Edited October 2, 201213 yr by Nullified

you guys are making this so much more difficult than it is

 

foreach ($array as $data) with array('val1','val2','val2')

foreach ($array as $k=>$v) with array ('k1'=>'v1','k2'=>'v2','k3'=>'v3')

 

lol thats the samething i said?

lol thats the samething i said?

I just wanted to reiterate the difference between the two.

Thanks everyone here is the PHP it goes 2 links deep at the moment, i am sure it can be improved but its a start

 

<!DOCTYPE html>
<html>
<head>
<title>Curl Test</title>
</head>

<body>
<?php									   
   $request_url ="http://www.yourdomain.com";//put your url here
   $url=str_replace("http://", "", $request_url); // i added this because to check if wwwdomain.com is in the url
   $alllinks=array();// create an array for all the links
   $badUrls=array("basket", "#" ,"mailto", "javascript:document", "reviews", "review");// array of urls we dont want to include
   $ch = curl_init(); //search for links
   curl_setopt($ch, CURLOPT_URL, $request_url); 
   curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
   $result = curl_exec($ch);
   $regex='|<a.*?href="(.*?)"|';
   preg_match_all($regex,$result,$parts);
   $links=$parts[1];
   echo "<h2>Home</h2>";//display links on the home page

foreach($links as $link){
   $check = substr($link, 0, 4);
  if($check=="http")
    {
	  //do nothing
	 }
 else {
 $link="$request_url/$link";
 }

if($link =="$request_url/#" || $link == "$request_url//") {	
	    //do nothng
		  }
		  elseif  (strpos($link,'#') == true  && (strpos($link,'mailto') == false)) {
		    //do nothing
				    }
				    else {
						  if(!in_array($link,$alllinks))//so we dont duplicate links
								 {
								   if(strpos($link, $url) == true)//make sure links are displayed if only on checked domain
									  {
									   $alllinks[] = $link;
									   echo "$link <br>";

							    }
				    }
		  }
	    }
	    //start level 2
	    echo "<h2>Level 2</h2>";
foreach ($alllinks as $link)//start second level
{
   $ch = curl_init();
	    curl_setopt($ch, CURLOPT_URL, $link);
	    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
	    $result = curl_exec($ch);
	    $regex='|<a.*?href="(.*?)"|';
	    preg_match_all($regex,$result,$parts);
	    $links=$parts[1];
	    foreach($links as $test){

  $check = substr($test, 0, 4);

  if($check==="http")
 {
	 //do nothing
   }

  else {
	   $test="$request_url/$test";
	   }
	    if($test=="$request_url/#" || $test == "$request_url//")  {
	    //do nothing
	    }
	    else {
			  if(!in_array($test,$alllinks))//so we dont duplicate links
				   {
					   if (strpos($test,'#') == false && (strpos($test,'mailto') == false)&& (strpos($test,'javascript:document') == false)) {
					   if(strpos($test, $url) == true)//make sure links are displayed if only on checked domain
										    {
					   $alllinks2[] = $test;
				 }
			   }
		    }		 
	    }
	 }
   }
	 $alllinks2=array_unique($alllinks2);
    foreach ($alllinks2 as $link)
	 {
		 $is_accepted = true;
		    foreach ($badUrls as $badurl)
			   {
				  if (strpos($link, $badurl) !== false)
				    {
					  $is_accepted = false;
				    }
			  }
		   if ($is_accepted)
			  {
				 $alllinks3[] = $link;
				 echo " $link<br />";
		    }
		  }						
		  //end row 2	  
	 curl_close($ch);
?>
</body>
</html>

in_array anyone?

 

yep in array works, i put it all in a loop so that it follows all links maximum 10 deep

 

thanks