Jump to content
confused and dazed

using curl on Username & Password Protected pages

Recommended Posts

Hello internet,

 

Recently I have been working with curl sessions and scrapping data from webpages. I have been fairly successful until I tried to access data from pages that are username and password protected. I have the username and password so that's not an issue - but I am not able to get the data on the page. Any thoughts?

 

$username='usr1';
$password='pswd1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'somelink');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_USERPWD, $username.":".$password);
$out = curl_exec($ch);
if(curl_exec($ch) === false)
{
echo 'Curl error: ' . curl_error($ch);
}
echo $out;
curl_close($ch);

Share this post


Link to post
Share on other sites

what's the error??

Share this post


Link to post
Share on other sites

There is no error message. Maybe it has something to do with what I expect to see. From the echo $out; I expect to see the logged in page embedded in the current page. All I see is the login page embedded in the current page. I expected to see the user info page instead of the login page as well as have access to the source data from the user page so I can scrape the source code.

Edited by confused and dazed

Share this post


Link to post
Share on other sites

The login page is what's being served. If you want something else to be served, you will need to send a POST request with a proper username and password and get a cookie in return.

 

Since curl probably won't save cookies for you, you'll have to manually read the Set-Cookie header and manually send a Cookie header when requesting a new page.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...