How To Extract All URLs From A Page Using PHP
Recently I needed a crawler script that would create a list of all pages on a single domain. As a part of that I wrote some functions that could download a page, extract all URLs from the HTML and turn them into absolute URLs (so that they themselves can be crawled later). Here’s the PHP code.
Extracting All Links From A Page
Here’s a function that will download the specified URL and extract all links from the HTML. It also translates relative URLs to absolute URLs, tries to remove repeated links and is overall a fine piece of code 
Depending on your goal you may want to comment out some lines (e.g. the part that strips ‘#something’ (in-page links) from URLs).
function crawl_page($page_url, $domain) { /* $page_url - page to extract links from, $domain - crawl only this domain (and subdomains) Returns an array of absolute URLs or false on failure. */ /* I'm using cURL to retrieve the page */ $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $page_url); curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); /* Spoof the User-Agent header value; just to be safe */ curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)'); /* I set timeout values for the connection and download because I don't want my script to get stuck downloading huge files or trying to connect to a nonresponsive server. These are optional. */ curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); curl_setopt($ch, CURLOPT_TIMEOUT, 15); /* This ensures 404 Not Found (and similar) will be treated as errors */ curl_setopt($ch, CURLOPT_FAILONERROR, true); /* This might/should help against accidentally downloading mp3 files and such, but it doesn't really work :/ */ $header[] = "Accept: text/html, text/*"; curl_setopt($ch, CURLOPT_HTTPHEADER, $header); /* Download the page */ $html = curl_exec($ch); curl_close($ch); if(!$html) return false; /* Extract the BASE tag (if present) for relative-to-absolute URL conversions later */ if(preg_match('/<base[\s]+href=\s*[\"\']?([^\'\" >]+)[\'\" >]/i',$html, $matches)){ $base_url=$matches[1]; } else { $base_url=$page_url; } $links=array(); $html = str_replace("\n", ' ', $html); preg_match_all('/<a[\s]+[^>]*href\s*=\s*([\"\']+)([^>]+?)(\1|>)/i', $html, $m); /* this regexp is a combination of numerous versions I saw online; should be good. */ foreach($m[2] as $url) { $url=trim($url); /* get rid of PHPSESSID, #linkname, & and javascript: */ $url=preg_replace( array('/([\?&]PHPSESSID=\w+)$/i','/(#[^\/]*)$/i', '/&/','/^(javascript:.*)/i'), array('','','&',''), $url); /* turn relative URLs into absolute URLs. relative2absolute() is defined further down below on this page. */ $url = relative2absolute($base_url, $url); // check if in the same (sub-)$domain if(preg_match("/^http[s]?:\/\/[^\/]*".str_replace('.', '\.', $domain)."/i", $url)) { //save the URL if(!in_array($url, $links)) $links[]=$url; } } return $links; }
How To Translate a Relative URL to an Absolute URL
This script is based on a function I found on the web with some small but significant changes.
function relative2absolute($absolute, $relative) { $p = @parse_url($relative); if(!$p) { //$relative is a seriously malformed URL return false; } if(isset($p["scheme"])) return $relative; $parts=(parse_url($absolute)); if(substr($relative,0,1)=='/') { $cparts = (explode("/", $relative)); array_shift($cparts); } else { if(isset($parts['path'])){ $aparts=explode('/',$parts['path']); array_pop($aparts); $aparts=array_filter($aparts); } else { $aparts=array(); } $rparts = (explode("/", $relative)); $cparts = array_merge($aparts, $rparts); foreach($cparts as $i => $part) { if($part == '.') { unset($cparts[$i]); } else if($part == '..') { unset($cparts[$i]); unset($cparts[$i-1]); } } } $path = implode("/", $cparts); $url = ''; if($parts['scheme']) { $url = "$parts[scheme]://"; } if(isset($parts['user'])) { $url .= $parts['user']; if(isset($parts['pass'])) { $url .= ":".$parts['pass']; } $url .= "@"; } if(isset($parts['host'])) { $url .= $parts['host']."/"; } $url .= $path; return $url; }
Sure, you can use these functions in your dissertation
Though I doubt they’ll be much faster than the other version – the bottleneck is probably your connection speed, and using cURL won’t change that.
ps- I’ve got it working now =)
Also – If I have permission to use it I’d also need to modify it a bit. For instance, to make it not include sub-domains, and also so that it doesn’t consider “www.ilovephp.net” to be part of the php.net domain =) (it seems to have that slight flaw at the moment, from my extremely brief testing). This is how I would modify the last bit (in case anyone’s interested):
$temp_current = preg_replace(“#http://(www\.)?#i”, ”, $url);
if(preg_match(“#^$domain#i”,$temp_current)) {
if(!in_array($url, $links)) $links[]=$url;
}
-Elin
Wow – fast response!! =D Thanks very much! You may be right about the bottleneck, but I like your function better than the other one anyway – it’s clearer, much less convoluted and I feel better equipped to work with it =)
Turns out it’s only semi-fixed, it works for my local test site but not external ones – I will continue tweaking! =)
- Elin
Allright, you can modify it, too.
I suspect your modification wouldn’t handle (extremely rare) situations where the external domain looks like this : “www.php.netasdfasdf.com”. Perhaps modify it like this :
Edit : fixed quotes.
True- thanks ^_^
Don’t suppose you have any idea how I could get around the bottleneck issue do you? =(
Elin
Well, you could probably use multiple download threads to get a little speed boost, but doing multi-threading in PHP can be a nightmare (in addition to all the other horrors of multi-threading). Alternatively, you could host the finished script on an actual web server somewhere; servers typically have good bandwidth.
Ah good- I should be able to host it on a server in the department. Even if I don’t I can just explain that in my dissertation and it should be fine.
Thanks very much indeed!! =) ^_^ You rule!
- Elin
Hi,
Nice work.
I do not know if I am doing something stupid but the following regular expression on my PC appears to be neither able to cope with URL’s which include spaces or single quotes.
preg_match_all(‘/]*href\s*=\s*[\"\']?([^\'\" >]+)[\'\" >]/i’, $html, $m);
David.
Hi,
Sorry the above regular expression does not render properly so please refer to the regular expression shown in your article instead.
David.
You’re right, the regexp couldn’t handle those characters. I’ve now modified it so that it can parse URLs with spaces and single/double quotes.
The downside is that the new regular expression will no longer detect links where the
hrefattribute isn’t quoted, i.e.will work, but
will not.
That’s great.
The new regexp now parses my URLs fine.
Thank you for your very prompt response.
David.
pls send me a code regarding url fetch and store in mysql db
@ Praveen : Not a chance. I’m not falling for that “plz send me the codes” nonsense.
This link extraction is working fine.
Thanks.
I want a tree searching code.
can you help me ?
What do you mean by “tree searching mode”?
What if I don’t want it to translate back to absolute URL? I cant seem to find which portion should be removed.
Remove or comment out this line :
I did that and the weird thing is the array outputs nothing. Just blank when using:
[code]
print_r(crawl_page("xxx.com/index.php", "xxx.com"));
[/code]
You could put some echo/print_r statements in the crawl_page function to see where and why it fails. For example, I’d put one after curl_exec to display the loaded HTML and a
after the preg_match_all() call.
cud u plz help me by teling how to use these functions in my php page.
i’m naive to all this.
but i need it for my project.. plz tell
I’m trying to work out how to do this for the whole site… Currently I have the script working with 2 simple variables, but I need this function to work within a preg-replace. most of what I want is working, but finding the absolute path is vital to make sure the output of CSS layouts works properly.
I’m currently doing this:
$page = preg_replace($pattern, $replacement, $page);
How do I add your function in there?
I don’t think there is an easy way to do that. The function isn’t really suited for such use; you’d probably have to figure out how it works and write a custom function for your specific situation.
Are you, by any chance, trying to download an entire site and rewrite the link paths so that it displays properly, etc? If so, I’m pretty sure there are already existing applications that can do that. Maybe it would be easier to use one of those instead of writing your own.
I managed to work this much out:
0) {
$url = preg_replace(“#(/[a-z0-9-]+){{$num_of_them}}$#iD”, ”, $url);
$path = $url . ‘/’. str_replace(‘../’, ”, $path);
}
else{
$path=str_replace($path,($url.’/’.$path),$path);
}
return $path;
}
echo ”;
?>
Now I just need to work out how to make this pseudo code work:
$pattern=’~src=”(.?*)”~’;
$new_url=’~src=”get_src($1,$path)”~’;
$page = preg_replace($pattern, $new_url, $page);
Well, I still don’t get what you’re trying to do. However, look into preg_replace_callback, it lets you use a function for replacements.
Sorry, yes… I’m making an application which can take a website code and render it in the browser, with some changes made to it (kinda like a spell-check).
I managed to get most of it working, but have an issue with putting a variable into the get_src function at the bottom. I want to replace the hard-coded URL for google with the pre-defined one at the top of the code.
So far I have this:
<?php
$url='http://www.google.com/one_level_up/2_levels_up/3levels_up/';
$path = '../../intl/en/images/logo.gif'; #this should display
$path2 = '../../../intl/en/images/logo.gif'; # this should not
$page=' Don\’t Display This: ‘;
#————————- Function Below ——————————–
function get_src($tmp_url,$path){
$tmp_url = rtrim($tmp_url, ‘/’);
$path = ltrim($path, ‘\\’);
if(($num_of_them = substr_count($path, ‘../’)) > 0) {
$tmp_url = preg_replace(“#(/[a-z0-9-]+){{$num_of_them}}$#iD”, ”, $tmp_url);
$path = $tmp_url . ‘/’. str_replace(‘../’, ”, $path);
}
else{
$path=str_replace($path,($tmp_url.’/’.$path),$path);
}
return $path;
}
?>Display This:
I’ve been searching for something like this for ages, so hopefully someone may find it useful.
This shout box seems to cut off my code. Here’s the last part:
function real_links($matches){
return ’src=”‘.get_src(‘http://www.google.com/one/two/’,$matches[1]).’”‘;
}
$page=preg_replace_callback(‘~src=”(.*?)”~’,'real_links’,$page);
echo $page;
You can do that by defining the $url variable as global in the function, like this :
Hi….
You did a great work….
but plz will you help me in some problem….
I am using net whose traffic go through proxy server:
HTTP proxy: 172.16.0.9
Port:8080
Above script run fines when i use direct connection with no proxy,but doesnt works when i use it in my college where proxy is needed…
plz help me how to tunnel traffic of your code through this proxy…….
Will be greatly thankful….
You are really gr8,i am using wamp5…
To make the script use a proxy, add
and
after the curl_init() call in the crawl_page() function. More information can be found in the curl_setopt documentation.