February 16, 201214 yr I'm trying to run a check to see if a URL that a user enters into my site is real, or if it still exists. I've looked at this tutorial, http://css-tricks.com/snippets/php/check-if-website-is-available/ which is pretty good but the problem I'm having is that when I do some testing of sites that don't exist like http://www.afahkjdhfkajdlfalkdjflafj.com, on css-tricks it correctly returns that the site doesn't exist, also on my local host it correctly determines between sites that do exist and the ones that don't, but when I upload the script to my site, it doesn't... Is there any reason for this that anyone can think of, it's a bit of a weird one...
February 16, 201214 yr Which technique from CSS-Tricks are you using? Is cURL available on your server (check PHPInfo)?
February 16, 201214 yr Author Which technique from CSS-Tricks are you using? Is cURL available on your server (check PHPInfo)? I'm using techinque 1. How do I check if cURL it is available?
February 16, 201214 yr Author it says cURL Support - enabled cURL Information - libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 Edited February 16, 201214 yr by adamsmith
February 18, 201214 yr Author Any help on this would be great please, I'm at a bit of a loss on what my options are and what I should do... My live site has the cURL settings of... cURL Support - enabled cURL Information - libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 My localhost has the settings of... cURL support enabled cURL Information 7.20.1 Age 3 Features AsynchDNS No Debug No GSS-Negotiate No IDN No IPv6 Yes Largefile Yes NTLM Yes SPNEGO No SSL Yes SSPI No krb4 No libz Yes CharConv No Protocols dict, file, ftp, ftps, http, https, imap, imaps, pop3, pop3s, rtsp, smtp, smtps, telnet, tftp Host i386-apple-darwin9.8.0 SSL Version OpenSSL/0.9.8o ZLib Version 1.2.5 Is this a problem with my hosting? I'm only with them at the moment because they are free while I'm developing... ###### Is there a way for me to change it? Should my localhost phpinfo be the same as my live host?
February 18, 201214 yr They may have locked something down, yeah. Why not get a full-featured package somewhere else? Plenty of cheap hosts around, it doesn't need to be top quality or even in a UK datacentre because it's not business-critical.
February 18, 201214 yr Author They may have locked something down, yeah. Why not get a full-featured package somewhere else? Plenty of cheap hosts around, it doesn't need to be top quality or even in a UK datacentre because it's not business-critical. You're right I do want to start looking at better hosting, I was planning on doing it in May when I finish Uni. Is there a way I can do it in .htaccess at all?
February 18, 201214 yr Have you tried grabbing something that does exist with cURL and examining the response?
February 18, 201214 yr Author Have you tried grabbing something that does exist with cURL and examining the response? It's only know that I'm expanding into cURL so I don't know how to do this... what would this show me?
February 18, 201214 yr Try this: function http_response($url) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_HEADER, TRUE); curl_setopt($ch, CURLOPT_NOBODY, TRUE); // remove body curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); $head = curl_exec($ch); $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE); curl_close($ch); return $httpCode; } ...and see what it returns for real and imaginary URLs. This page, for example - if it returns 200 you'll know cURL works. Edited February 18, 201214 yr by Renaissance-Design
February 18, 201214 yr Author adfasdfasd.com returns 307 both asdfj returns 0 on local and 404 on live www.google.com returns 302 on local and 200 on live www.apple.com returns 200 on both http://www.webdesignerforum.co.uk/topic/58951-check-if-url-is-realexists/page__pid__376055#entry376055 returns 200 on both what do you make of that?
February 18, 201214 yr adfasdfasd.com returns 307 both asdfj returns 0 on local and 404 on live www.google.com returns 302 on local and 200 on live www.apple.com returns 200 on both http://www.webdesignerforum.co.uk/topic/58951-check-if-url-is-realexists/page__pid__376055#entry376055 returns 200 on both what do you make of that? cURL works. I'm guessing your host's servers are located in the US - from there you're getting 200 on google.com, from your own machine you're getting 302 (essentially: "See Other") - I bet google.co.uk would return 200 from your machine. asdfj - it's failing because the function didn't take malformed hostnames into account. Not sure what's happening with your live host there, unless it's trying to serve yourdomain.com/asdfj adfasdfasd.com is actually registered, but redirects to freeforums.org; I imagine it was used for spam or something.
February 18, 201214 yr Author You're right, www.google.co.uk returned 200 What does this show us about the function for testing websites that I'm using? Should I use something else or is there a reason it isn't working?
February 18, 201214 yr Technique 1 on CSS-Tricks is geared to seeing if the domain exists. Are you looking to verify a domain or any URL? Chris Coyier's function returns true (in theory) if any HTTP response is received - could you post examples of what works/doesn't work with his function? Or did you test using the same URLs as mine?
February 18, 201214 yr Author Technique 1 on CSS-Tricks is geared to seeing if the domain exists. Are you looking to verify a domain or any URL? Chris Coyier's function returns true (in theory) if any HTTP response is received - could you post examples of what works/doesn't work with his function? Or did you test using the same URLs as mine? I'm verifying any URL to see if it exists and it seems that anything that passes for a valid URL is returning true whether it is real or not.
February 18, 201214 yr Only real difference I can see is the use of filter_var() - but that still shouldn't make the function return true if not available. What version of PHP are you running?
February 18, 201214 yr Ah, I give up. Maybe I'm hard of thinking today or something, but I just don't see what would cause false positives.
February 18, 201214 yr Author Ah, I give up. Maybe I'm hard of thinking today or something, but I just don't see what would cause false positives. Me neither, but thanks for you're help lol Guess I'll just keep trying ha
February 18, 201214 yr Author could it be the last bit... if ($response) return true; return false; should it not be... if ($response) { return true; } else { return false; } But I suppose that wouldn't explain why it works correctly on my local host
February 18, 201214 yr For ease of reading, yeah - but that should work fine. "Return" exits execution of the function, so if true the last false isn't reached.
February 18, 201214 yr Author Well the only other thing I can think of is that it is failing at one of the checks, either; $curlInit = curl_init($domain); curl_setopt($curlInit,CURLOPT_CONNECTTIMEOUT,10); curl_setopt($curlInit,CURLOPT_HEADER,true); curl_setopt($curlInit,CURLOPT_NOBODY,true); curl_setopt($curlInit,CURLOPT_RETURNTRANSFER,true); Can I take any of them out individually to see if that makes a difference or will it break the function?
February 18, 201214 yr I've just put the two together and come up with this which works for me. Give it a shot... function domainExists($domain) { $invalid_http_codes = array(0, 404); if(!filter_var($domain, FILTER_VALIDATE_URL)) return FALSE; $curlInit = curl_init($domain); curl_setopt($curlInit,CURLOPT_CONNECTTIMEOUT,10); curl_setopt($curlInit,CURLOPT_HEADER,true); curl_setopt($curlInit,CURLOPT_NOBODY,true); curl_setopt($curlInit,CURLOPT_RETURNTRANSFER,true); curl_exec($curlInit); $httpCode = curl_getinfo($curlInit, CURLINFO_HTTP_CODE); curl_close($curlInit); return !in_array($httpCode, $invalid_http_codes); }
February 18, 201214 yr Author I've just put the two together and come up with this which works for me. Give it a shot... function domainExists($domain) { $invalid_http_codes = array(0, 404); if(!filter_var($domain, FILTER_VALIDATE_URL)) return FALSE; $curlInit = curl_init($domain); curl_setopt($curlInit,CURLOPT_CONNECTTIMEOUT,10); curl_setopt($curlInit,CURLOPT_HEADER,true); curl_setopt($curlInit,CURLOPT_NOBODY,true); curl_setopt($curlInit,CURLOPT_RETURNTRANSFER,true); curl_exec($curlInit); $httpCode = curl_getinfo($curlInit, CURLINFO_HTTP_CODE); curl_close($curlInit); return !in_array($httpCode, $invalid_http_codes); } after initial tests, that seems to work perfectly, could you please tell me what you've done differently...
February 19, 201214 yr It just fetches the HTTP code returned from the curl request. It then checks if that code is either 0 or 404 - ie. the page (or domain) was not found. If it is either of those, then it returns false, otherwise it will return true for all other HTTP codes. You can add other invalid codes in the array at the top. So, for example, adding 302 to the invalid_http_codes array then testing 'google.com' from a UK location will cause it to return false. I wouldn't recommend adding 302, that's a temporary redirect so it's likely the domain is valid. Edited February 19, 201214 yr by andyl
February 19, 201214 yr Author do you know why that code is working compared to the other that didn't? Also, do you know if it is as reliable as the first code I had?
February 19, 201214 yr Your original code would return true for 404's. It should be, it's exactly the same with slightly tighter rules.
February 19, 201214 yr Author Is there anyway that a malicious website could spoof it's HTTP code or a legit website not have a HTTP code for some reason? I'm just trying to figure out if there is anyway around this function.
February 19, 201214 yr Yes and no. But what does it matter if it is spoofed - even if the website is malicious, it's valid. Spoofing a 404 would be like shooting yourself in the foot, or face. There probably is ways round it, but I don't see why it needs to be MI5-secure-worthy?
February 19, 201214 yr I've been pondering why Google 302s from .com to .co.uk. Only thing I can think of is cache busting for user agents that may be used in different locales (eg laptops, mobiles that travel).
February 19, 201214 yr Author Yes and no. But what does it matter if it is spoofed - even if the website is malicious, it's valid. Spoofing a 404 would be like shooting yourself in the foot, or face. There probably is ways round it, but I don't see why it needs to be MI5-secure-worthy? lol, you're probably right. The reason I ask is because those websites are being displayed in my site via an iFrame, I don't know if there are security implications there with a malicious website being able to access any of my sites information through the iFrame.
February 19, 201214 yr Author I've been pondering why Google 302s from .com to .co.uk. Only thing I can think of is cache busting for user agents that may be used in different locales (eg laptops, mobiles that travel). I've no idea, that's beyond my scope of understanding at the moment, ha
Create an account or sign in to comment