Comments on: How To Extract All URLs From A Page Using PHP

By: sadr1an1

sadr1an1 — Sat, 04 Feb 2012 19:30:47 +0000

10x, it workd like a charm

By: Mathieu

Mathieu — Fri, 13 May 2011 19:24:06 +0000

Her is another working one:
http://web-o-blog.blogspot.com/2011/04/function-absoluteurlbaseurl-relativeurl.html

By: White Shadow

White Shadow — Sun, 10 Jan 2010 17:53:25 +0000

To make the script use a proxy, add

curl_setopt($ch, CURLOPT_PROXY, proxyip);

and

curl_setopt($ch, CURLOPT_PROXYPORT, portnumber)

after the curl_init() call in the crawl_page() function. More information can be found in the curl_setopt documentation.

By: Your fan

Your fan — Sun, 10 Jan 2010 17:25:07 +0000

Hi….
You did a great work….
but plz will you help me in some problem….
I am using net whose traffic go through proxy server:

HTTP proxy: 172.16.0.9
Port:8080

Above script run fines when i use direct connection with no proxy,but doesnt works when i use it in my college where proxy is needed…
plz help me how to tunnel traffic of your code through this proxy…….
Will be greatly thankful….
You are really gr8,i am using wamp5…

By: White Shadow

White Shadow — Tue, 22 Dec 2009 17:01:25 +0000

You can do that by defining the $url variable as global in the function, like this :

function real_links($matches){
    global $url;
    return 'src="'.get_src($url, $matches[1]).'"';
}

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 22:48:13 +0000

function real_links($matches){
return ‘src=”‘.get_src(‘http://www.google.com/one/two/’,$matches[1]).’”‘;
}
$page=preg_replace_callback(‘~src=”(.*?)”~’,'real_links’,$page);
echo $page;

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 22:47:11 +0000

This shout box seems to cut off my code. Here’s the last part:

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 22:45:27 +0000

Sorry, yes… I’m making an application which can take a website code and render it in the browser, with some changes made to it (kinda like a spell-check).

I managed to get most of it working, but have an issue with putting a variable into the get_src function at the bottom. I want to replace the hard-coded URL for google with the pre-defined one at the top of the code.
So far I have this:

$url='http://www.google.com/one_level_up/2_levels_up/3levels_up/';
$path = '../../intl/en/images/logo.gif'; #this should display
$path2 = '../../../intl/en/images/logo.gif'; # this should not
$page=' Don\’t Display This: ‘;
#————————- Function Below ——————————–
function get_src($tmp_url,$path){
$tmp_url = rtrim($tmp_url, ‘/’);
$path = ltrim($path, ‘\\’);
if(($num_of_them = substr_count($path, ‘../’)) > 0) {
$tmp_url = preg_replace(“#(/[a-z0-9-]+){{$num_of_them}}$#iD”, ”, $tmp_url);
$path = $tmp_url . ‘/’. str_replace(‘../’, ”, $path);
}
else{
$path=str_replace($path,($tmp_url.’/’.$path),$path);
}
return $path;
}
?>Display This:

I’ve been searching for something like this for ages, so hopefully someone may find it useful.

By: White Shadow

White Shadow — Sat, 19 Dec 2009 21:04:24 +0000

Well, I still don’t get what you’re trying to do. However, look into preg_replace_callback, it lets you use a function for replacements.

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 20:59:29 +0000

I managed to work this much out:

0) {
$url = preg_replace(“#(/[a-z0-9-]+){{$num_of_them}}$#iD”, ”, $url);
$path = $url . ‘/’. str_replace(‘../’, ”, $path);
}
else{
$path=str_replace($path,($url.’/’.$path),$path);
}
return $path;
}

echo ”;
?>

Now I just need to work out how to make this pseudo code work:
$pattern=’~src=”(.?*)”~’;
$new_url=’~src=”get_src($1,$path)”~’;
$page = preg_replace($pattern, $new_url, $page);