Logging In With CURL and PHP
This is an example of how you can use CURL to “log in” and retrieve some protected info. I’ve showed the somewhat extreme case where you need to maintain cookies, spoof HTTP referer and use SSL. The example page used here is ClickBank – one of the largest digital product retailers. I’ve tried to comment the source as much as possible to make it easier to understand. See the end of this post for more information on CURL. You can also leave a comment if you have further questions.
/*********************************
**Set up your variables**
**********************************/
$cookiefile = tempnam("/tmp", "cookies");
/* Create a temporary file to store cookies.
This should work on most systems and is more
flexible than specifying path explicitly */
$login_url='https://www.clickbank.com/login.htm';
/* The page that displays the login form. */
$login_post_url='https://www.clickbank.com/account/login';
/* The "action" value of the login form. This is not always
equal to $login_url. */
$username = "username";
$password = "passw0rd";
$agent="Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
/*********************************
**Load the "login" page and get some cookies**
**********************************/
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$login_url);
/* The URL of the page to retrieve */
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
/* Disguise self as a browser app. Some servers
might need a different value here. Some servers
might try to check if the page is visited by a
real human being using this value. */
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
/* Don't output the results -
return them as a string instead */
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
/* Follow redirects.
This isn't actually necessary here :P */
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
/* Read cookies from this file */
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
/* Save cookies to the same file too */
/* SSL stuff - remove if not needed */
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
/* Check the existence of a common name and also
verify that it matches the hostname provided. Not
strictly necessary in most cases. Use 0 to disable. */
/* SSL stuff - remove if not needed */
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
/* Turn off SSL peer certificate verification. This prevents
the "SSL certificate problem, verify that the CA cert is OK."
error. If you really need this set to "true",
see this link for a solution -
http://lu.php.net/manual/en/ref.curl.php#71326
*/
$result = curl_exec ($ch);
/* Perform the query, retrieve the page. */
curl_close ($ch);
/*************************************
Actually log in with the proper referer and cookies
**************************************/
/* The fields of the login form. These will probably be
different for every particular page. */
$postfields = array(
'nick' => $username,
'pass' => $password,
//'rememberMe' => 'false',
'j_username' => $username,
'j_password' => $password,
);
$reffer = $login_url;
/* If the server checks the referer we need to spoof it */
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$login_post_url);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
http_build_query($postfields));
/* http_build_query() will properly escape the fields and
build a query string. */
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
/* Follow redirects. This is probably necessary here. */
curl_setopt($ch, CURLOPT_REFERER, $reffer);
/* spoof the HTTP referer */
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
/* Note that this is the same file as before */
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$result = curl_exec ($ch);
/* Now we've got the contents of the page you see after
logging in saved in $result */
curl_close ($ch);
/*****************************************
**If you need to get another page....**
This is similar to the above examples, just use the same
cookie file and maybe spoof the referer if needed
******************************************/
$data_url='https://www.clickbank.com/account/showTransactions.htm';
$reffer = $login_post_url;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$data_url);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $reffer);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$result = curl_exec ($ch);
curl_close ($ch);
echo $result;
/******************************************
**All done. Kill the cookie file once it's not needed anymore**
*******************************************/
unlink($cookiefile);
More information
PHP Manual – http://lu.php.net/curl
cUrl/PHP Examples : http://curl.haxx.se/libcurl/php/examples/
Subscribe via RSS
Get Updates by Email
Hello,
Is there some special setting in cURL that needs to be set to download one of the CSV reports?
What’s the best way to have a script download a specific date range for a CSV report?
Thanks!
-Jeffery
No, you don’t need to use any special settings as far as I know.
The URL to download the CSV looks something like this (with “sales” and “refunds” boxes checked) –
https://www.clickbank.com/account/showTransactions.htm?d-8275208-e=1&startDate=2007-06-01&form=dates&type=SALE_ARRAY&type=RFND_ARRAY&6578706f7274=1&endDate=2007-07-09&s=1
I’d guess you can just plug in your own startDate and endDate to to get a CSV file for a specific period.
By the way, do you have anything specific in mind? I thought I’d make a script that sends e-mail notifications when a sale is made. I also have some other ideas… would love to hear yours 😉
Hey!
Not sure why my post didn’t take. Here it is again.
Here’s what I added to your test script to try and download the CSV
$ch = curl_init();
$reffer = “https://www.clickbank.com/account/showTransactions.htm?form=dates&s=1&type=SALE_ARRAY&startDate=2007-06-16&endDate=2007-06-30”;
$thisFilename = “./test-a-cb-2007-06-30.csv”;
$data_url = “https://www.clickbank.com/account/showTransactions.htm?d-8275208-e=1&startDate=2007-06-16&startDate=2007-06-16&type=SALE_ARRAY&form=dates&6578706f7274=1&endDate=2007-06-30&endDate=2007-06-30&s=1”;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$data_url);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $reffer);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$result = curl_exec($ch);
$bytes = file_put_contents($thisFilename, $result);
curl_close ($ch);
But for some reason it was only giving me the value from the $reffer URL instead of the expected content from the CSV file from $data_url.
I played around with the URL. Noting I had the & as & and switched them to the correct format (ClickBank’s HTML source had the & in it).
That worked perfectly!
I’m thinking this would be a great function to use for an app that would automatically help you build up your list (with double optins of course 😉
Since people who buy from you are on your “hot list” it’s also useful information to have.
cURL is pretty cool. How did you learn to use it? I found most of the docs I ran across weren’t that great. I’ve already forwarded your link to several friends.
Thanks again.
-Jeffery
You mean “&” and “& amp ;”, right? (my browser displays them both as “&” so I can’t tell whcih one you used). I haven’t actually tried to download the CSV with cURL yet.
Yes, it could be useful for building a list. However there might be a problem with that – someone claims that contacting a buyer in this fashion is against CB TOS. Unfortunately I can’t find any mention of that in the FAQs.
I haven’t used cURL that much. I needed to download the marketplace feed for my CBTool site somehow so I looked around and found cURL. The “official” documentation is a bit scarce, but it is generally possible to find an explanation of most features/bugs on the Web – if you have a few hours to waste on that 😛
Yes, that’s correct. The amp ; part didn’t show up when I posted the message 😉
I agree you must be careful about what you do with the emails. Never spam, and always double-optin.
What I was referring to however is not to just add people to your list automatically from the CB sales reports. That would be bad, regardless of whether CB says it’s OK or not. I’m sure you know that, but I just wanted to make it clear that that is not my intention.
What I think would be great though is having an opt-in list that you market to, then when someone on your list buys something you can use the custom report to cross-match their email and flag them in your database. This would tell you they’re real buyers, not just tire-kickers. This would allow you to perhaps extend special offers to them by sending a special link or newsletter for those special folks…
I’m not a marketer, but I see potential for data-mining at least.
Just thinking out loud…
Thanks
-Jeffery
Quote : “What I think would be great though is having an opt-in list that you market to, then when someone on your list buys something you can use the custom report to cross-match their email and flag them in your database. This would tell you they’re real buyers, not just tire-kickers. (…)”
Ahh, that really is a good idea! I think I read somewhere about dividing your list in “tiers” and sending them different info/offers depending on what type of a subscriber they are. Should be more efficient – quality over quantity et cetera.
Anyway, I don’t have a list, so I’m theorizing here 😉
I don’t have a list yet either. I’m a develper, not a marketer.
But you have to make a conscious decision to really care for what you intend to do with a list you build yourself… I think people are finding more and more that a list isn’t much more than the person behind it. It might do well at first, but then most marketers seem to just run out of steam, lose interest, or whatever and then the list goes dead.
One way I found that’s good for building a list like with your free tool is to have an activation link tied to the signup process. It allows you to make the process of signing up for your mailing list and creating an account pretty painless and seamless…
http://www.getresponse.com/ModStatic/Section3_6_3
Just in case you might be looking for an example.
-J
I’m not looking to build a list at this time. I probably could get a few subscribers, but I wouldn’t know what to do with them.
Sure, there is a lot of info available and it’s all good in theory, but I’d need to be sure I can make that commitment and really provide value to people on my hypothetical list.
[…] in” to the target website before you can upload anything, so you might want to read my how to log in with PHP and CURL post. Share and Enjoy: These icons link to social bookmarking sites where readers can share and […]
Thank you. I have just spent hours combing through documentation, talking to people, and analyzing snippets of code to get a similar thing running, but to no avail. Several google pages down, I came across this, and it finally works. Well written, good documentation on your code.
[…] Logging In With CURL and PHP SHARETHIS.addEntry({ title: “Get Your Clickbank Sales Stats Programmatically”, url: “http://www.digitalproductsreview.net/blog/get-your-clickbank-sales-stats-programmatically/” }); Tags: clickbank sales stats, return on investment […]
Thanks a lot! I used it in my application practically without any modifications.
Hi, nice stuff there. Thanx. I have a ques: for you. I developed a curl script to loging rapidshare using my premium account. But it didn’t work. They are using cgi for logon script. How can I do it? Help me. Here is a part if my code…
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_URL, ‘https://ssl.rapidshare.com/cgi-bin/premiumzone.cgi’);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, ‘login=’ . samy . ‘&password=’ . samy);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
return (curl_exec($ch));
curl_close($ch);
Hi, I’ve been trying to do this for UPS.com – use curl to login and create a shipping label from my shopping cart software , but no success yet. I tried everything and It seems like UPS has blocked it somehow. Login does not work. But even if I login manually and just try to fill in the shipping labels using curl, it does not let me do it unless i have the current cookies and i cant get those from the script. If i retrieve the cookies manually (looking at the header), it works. I would be grateful for any help. Thanks!
@vp – Use a network sniffer and compare what happens when you do it with a browser vs. CURL. Then modify the script to act exactly as the browser.
It’s also possible that UPS.com has some kind of JavaScript validation; e.g. a JS script that fills out a hidden “checksum” field before the form is submitted. CURL doesn’t run JS, so ups.com can detect that something is wrong. Examining the traffic with a sniffer should reveal tricks like this, too.
Funciona a la perfeccion !!
Muy agradecido
Could this method be used to login to digg.com, I have been trying all day and no avail.
Probably not, at least not easily. I’m sure Digg.com has some kind of measures set up specifically to prevent this kind of thing.
Great tutorial. Worked like a charm against my test server.
i want to make some php script to auto post to diigo.
i’ll try this code to make it work with my idea 😀
thanks