Jump to content

using curl on Username & Password Protected pages


confused and dazed

Recommended Posts

Hello internet,

 

Recently I have been working with curl sessions and scrapping data from webpages. I have been fairly successful until I tried to access data from pages that are username and password protected. I have the username and password so that's not an issue - but I am not able to get the data on the page. Any thoughts?

 

$username='usr1';
$password='pswd1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'somelink');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_USERPWD, $username.":".$password);
$out = curl_exec($ch);
if(curl_exec($ch) === false)
{
echo 'Curl error: ' . curl_error($ch);
}
echo $out;
curl_close($ch);
Link to comment
Share on other sites

There is no error message. Maybe it has something to do with what I expect to see. From the echo $out; I expect to see the logged in page embedded in the current page. All I see is the login page embedded in the current page. I expected to see the user info page instead of the login page as well as have access to the source data from the user page so I can scrape the source code.

Edited by confused and dazed
Link to comment
Share on other sites

The login page is what's being served. If you want something else to be served, you will need to send a POST request with a proper username and password and get a cookie in return.

 

Since curl probably won't save cookies for you, you'll have to manually read the Set-Cookie header and manually send a Cookie header when requesting a new page.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...