Scrape web with login session (using python scrapy)

Question

I am trying to extra some data under a login session using y. Usually when I intercept data I can view the format data in the network tab. In this login page, when the log on is submitted, I can't seems to find the format data tab.

https://lumen.ni.com/nicif/login.xhtml

Can anyone advice ?

My current implementation:

import y
from y import FormReuest

class LoginSpider(y.Spider):
name = 'example.com'


def start_requests(self):
    return [y.FormRequest("https://lumen.ni.com/nicif/login.xhtml",
                               formdata={'LoginForm:email': 'email', 'LoginForm:password': 'password'},
                               callback=self.logged_in)]

def logged_in(self, response):
    if("Please use correct email & password combination") in str(response.body):
        print("fail")
    else:
        data = {'p_part_number':'779494-01','p_duration':'','p_duration_uom':''}
        yield FormRequest(url="http://sine.ni.com/apps/utf8/nipp.purchase_form_ajax", formdata=data)

Umair Ayub · Accepted Answer · 2018-03-15 08:01:39Z

1

There are lots of other parameters they are using along with form, they are hidden in the login form, you need to send them too along with your request.

answered Mar 15, 2018 at 8:01

Umair Ayub

21.7k14 gold badges82 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user2386301 Over a year ago

wow, thanks. I didn't noticed that there is a login.xhtml, I always checked xhr tab so didn't realize that's there. May I ask how to I get the information of the javax.faces.ViewStates to send along with form ?

Umair Ayub Over a year ago

As I said its in the login page, look for hidden input fields

Mushirih Over a year ago

Is it possible to do scraping past a /login while the /login is disallowed in the robots.txt?

Umair Ayub Over a year ago

@Mushirih yes its possible, set ROBOTSTXT_OBEY = False in settings.py

Mushirih Over a year ago

@Umair I have only one file which has all my code, tell me how to implement this flag

Collectives™ on Stack Overflow

Scrape web with login session (using python scrapy)

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related