Hi,

 

I'm looking to create a simple PHP script that:

 

• Has one input box to enter a URL
  
• PHP then crawls all the internal links it can find within the site
  
• Then list all the internal links
  

 

I would appreciate any tips/direction on this!

 

Note: My PHP skill is medium and I know RegEX very well. I presume I will have to use the function a number of times file_get_contents

Could use CURL to get the webpage into a variable:

 

$url = 'example.com';
$c = curl_init();
curl_setopt($c, CURLOPT_URL, $url);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)');
$webpage= curl_exec($c);
curl_close($c);

//alternatively: $webpage = file_get_contents($url) or die("Could not access: $url"); 

 

Then find links (this some old code I had laying around from ages ago):

 

$rxp = "<a\s[^>]*href=(\"??)([^\" >]*?)\\1[^>]*>(.*)<\/a>";   
if(preg_match_all("/$rxp/siU", $webpage, $matches)) { 
   //$matches[2] = array of link addresses 
   //$matches[3] = array of link text - including HTML code 
 }

 

Thanks! Very useful.

You shouldn't really use regex to parse HTML (it's very hacky and was never designed for such a complex task.) Take a look into php.net/dom

Don't mean to burst anyones bubble here but a Open source php crawler is available here.

 

I have used this before and it is a very straight-forward script, maybe you could take from it and enhance it!, Good luck my friend!