Comments on: How To Extract All URLs From A Page Using PHP

By: White Shadow

White Shadow — Sun, 10 Jan 2010 17:53:25 +0000

To make the script use a proxy, add

curl_setopt($ch, CURLOPT_PROXY, proxyip);

and

curl_setopt($ch, CURLOPT_PROXYPORT, portnumber)

after the curl_init() call in the crawl_page() function. More information can be found in the curl_setopt documentation.

By: Your fan

Your fan — Sun, 10 Jan 2010 17:25:07 +0000

Hi….
You did a great work….
but plz will you help me in some problem….
I am using net whose traffic go through proxy server:

HTTP proxy: 172.16.0.9
Port:8080

Above script run fines when i use direct connection with no proxy,but doesnt works when i use it in my college where proxy is needed…
plz help me how to tunnel traffic of your code through this proxy…….
Will be greatly thankful….
You are really gr8,i am using wamp5…

By: White Shadow

White Shadow — Tue, 22 Dec 2009 17:01:25 +0000

You can do that by defining the $url variable as global in the function, like this :

function real_links($matches){
    global $url;
    return 'src="'.get_src($url, $matches[1]).'"';
}

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 22:48:13 +0000

function real_links($matches){
return ’src=”‘.get_src(‘http://www.google.com/one/two/’,$matches[1]).’”‘;
}
$page=preg_replace_callback(‘~src=”(.*?)”~’,'real_links’,$page);
echo $page;

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 22:47:11 +0000

This shout box seems to cut off my code. Here’s the last part:

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 22:45:27 +0000

Sorry, yes… I’m making an application which can take a website code and render it in the browser, with some changes made to it (kinda like a spell-check).

I managed to get most of it working, but have an issue with putting a variable into the get_src function at the bottom. I want to replace the hard-coded URL for google with the pre-defined one at the top of the code.
So far I have this:

$url='http://www.google.com/one_level_up/2_levels_up/3levels_up/';
$path = '../../intl/en/images/logo.gif'; #this should display
$path2 = '../../../intl/en/images/logo.gif'; # this should not
$page=' Don\’t Display This: ‘;
#————————- Function Below ——————————–
function get_src($tmp_url,$path){
$tmp_url = rtrim($tmp_url, ‘/’);
$path = ltrim($path, ‘\\’);
if(($num_of_them = substr_count($path, ‘../’)) > 0) {
$tmp_url = preg_replace(“#(/[a-z0-9-]+){{$num_of_them}}$#iD”, ”, $tmp_url);
$path = $tmp_url . ‘/’. str_replace(‘../’, ”, $path);
}
else{
$path=str_replace($path,($tmp_url.’/’.$path),$path);
}
return $path;
}
?>Display This:

I’ve been searching for something like this for ages, so hopefully someone may find it useful.

By: White Shadow

White Shadow — Sat, 19 Dec 2009 21:04:24 +0000

Well, I still don’t get what you’re trying to do. However, look into preg_replace_callback, it lets you use a function for replacements.

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 20:59:29 +0000

I managed to work this much out:

0) {
$url = preg_replace(“#(/[a-z0-9-]+){{$num_of_them}}$#iD”, ”, $url);
$path = $url . ‘/’. str_replace(‘../’, ”, $path);
}
else{
$path=str_replace($path,($url.’/’.$path),$path);
}
return $path;
}

echo ”;
?>

Now I just need to work out how to make this pseudo code work:
$pattern=’~src=”(.?*)”~’;
$new_url=’~src=”get_src($1,$path)”~’;
$page = preg_replace($pattern, $new_url, $page);

By: White Shadow

White Shadow — Sat, 19 Dec 2009 20:55:55 +0000

I don’t think there is an easy way to do that. The function isn’t really suited for such use; you’d probably have to figure out how it works and write a custom function for your specific situation.

Are you, by any chance, trying to download an entire site and rewrite the link paths so that it displays properly, etc? If so, I’m pretty sure there are already existing applications that can do that. Maybe it would be easier to use one of those instead of writing your own.

By: Neil Trigger

Neil Trigger — Sat, 19 Dec 2009 14:54:09 +0000

I’m trying to work out how to do this for the whole site… Currently I have the script working with 2 simple variables, but I need this function to work within a preg-replace. most of what I want is working, but finding the absolute path is vital to make sure the output of CSS layouts works properly.

I’m currently doing this:
$page = preg_replace($pattern, $replacement, $page);

How do I add your function in there?