python authentication with requests library via POST

Question

I read several similar topic.. I try to follow the others examples but I'm still stuck in the middle of nowhere.. I have basic skills of python programming and little knowledge about http protocol, my two goals are: -succesfull authentication to a website via requests library -fetch data from the website after the login while the session is active

This is the code:

import requests

targetws = 'https://secure.advfn.com/login/secure'

s = requests.session()

payload_data = {'login_username': 'xxx', 'login_password': 'yyy'}

response = s.post(targetws, data=payload_data)


url='https://it.advfn.com/mercati/BIT/generali-G/ordini'

result = s.get(url) 

print result.content

But I always get no success with log in.. Maybe I miss some value in post data like submit action or else? Any help will be appreciated, best regards!

Here the html code from the page:

form action="https://secure.advfn.com/login/secure" id="login_form" name="login_form" method="POST" target="">

        <input type="hidden" value="aHR0cDovL2l0LmFkdmZuLmNvbQ==" name="redirect_url" id="redirect_url">
        <input type="hidden" value="it" name="site" id="site">

        <div class="fields"><label for="login_username">Username</label> 
            <input type="text" tabindex="1" class="text ui-widget-content" value =""
            id="login_username" name="login_username" maxlength="64">
        </div>

        <div class="fields"><label for="login_password">Password</label> 
            <input tabindex="2" type="password" class="text ui-widget-content" value="" id="login_password" name="login_password" maxlength="16">
        </div>
                <div class="fields">
                    <strong><a href="/common/account/password/request">Se ti sei dimenticato la tua password</a></strong> &nbsp;
                    <input  class="button"  tabindex="3" type="submit"   value="Accedi" id="login_submit">
                </div>
    </form

What is the response you get back when the login attempt fails? Check the status_code and text attributes. — J. Owens
– J. Owens, Commented Jun 14, 2016 at 15:47
Thanks for reply, status code always 200 and text: 200 Traceback (most recent call last): File "deepbook.py", line 21, in <module> print result.text UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 909: ordinal not in range(128) — egariM
– egariM, Commented Jun 14, 2016 at 15:51
your connection type expecting header of form-url encoded and you are sending json object. data = 'username=' + user + '&password=' + password header = { "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8" } — Devendran Nehru
– Devendran Nehru, Commented Jun 14, 2016 at 15:53

Padraic Cunningham · Accepted Answer · 2016-06-14 20:47:13Z

2

If you look at what gets posted:

You see you need the redirect_url and site which you can parse from the input in the source with bs4:

import requests
from bs4 import BeautifulSoup

data = {"redirect_url": "",
        "site": "uk",
        "login_username": "foo",
        "login_password": "bar"}

with requests.Session() as s:
    log = "https://secure.advfn.com/login/secure"
    r = s.get("http://uk.advfn.com/")
    soup = BeautifulSoup(r.content)
    redirect_url = soup.select_one("#redirect_url")["value"]
    site =  soup.select_one("#site")["value"]
    data["redirect_url"] = redirect_url
    p = s.post(log, data=data)
    print(p.content)
    print(s.get('https://it.advfn.com/mercati/BIT/generali-G/ordini').content)

Once you do that you will be successfully logged in.

edited Jun 14, 2016 at 20:47

answered Jun 14, 2016 at 20:41

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

egariM Over a year ago

The code that you write works! A big thanks for this help! Just few question to understand my mistake.. 1. When I use the s.post() the data are submitted over the header? 2. You said that when I call secure.advfn.com/login/secure I was redirected to uk.advfn.com but how you get that site? It's coded base64 in redirect_url? Should I check that variable (redirect_url) in the response message every time to be sure? Thanks a lot for help!

egariM Over a year ago

4. What kind of software do u use to catch the headers? cause with "live http header" extension for chrome I think that datas are sends via cookies after the first login.. Thanks!

Padraic Cunningham Over a year ago

INo prob, if you look at the source code, you can see under input tags the redirect_url and site, it is just a matter of parsing the values from the tags to get what you need to post. If you open chrome developer tools or firebug you can see exactly what is happening under the network tab

egariM Over a year ago

Ok I alredy use firebug but I dindn't check the network tab. Just last question :) The page that s.get(page).content execute a javascript.. Could I execute it inside that requests library object or have to use something else like webkit?

Collectives™ on Stack Overflow

python authentication with requests library via POST

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related