Running a scrapy program from another python script

Question

This question has been kind of answered before but answers are years old.

In my "project" I have 4 spiders and each one of them deals with different kinds of products I encounter (scraping amazon ATM). Each product has a category, for example, if I want to scrape "laptops" I use one scraper but if the objective is to scrape clothes, I have another one.

So, is it there a way to run a python script that, depending on the product I have to scrape (products are read from a txt file) a different spider is called?

Code would look like this

#Imports

def scrapyProject():

    #Get the products I want to scrape
    if productIsClothes:

        runClothesSpider

    else productIsGeneric:

        runGenericSpider

I know the previous code is rough, It's kind of a sketch for the final code.

It would also help knowing which imports I need for the program to work

Sorry, for answering late, had some issues with my internet. I'm not really using any patterns, each spider is coded separately. What I'm trying to get is a way of running a script from another python code, for example, running (talking about scrapy) 'scrapy crawl spider' from the python script — Manuel
– Manuel, Commented Jan 22, 2019 at 2:07

Granitosaurus · Accepted Answer · 2019-01-22 07:53:30Z

4

You could just set spider class with an if statement:

import sys

import scrapy
from scrapy.crawler import CrawlerProcess

from project.spiders import Spider1, Spider2

def main():
    process = CrawlerProcess({})

    if sys.argv[1] == '1':
        spider_cls = Spider1
    elif sys.argv[1] == '2':
        spider_cls = Spider2
    else:
        print('1st argument must be either 1 or 2')
        return
    process.crawl(spider_cls)
    process.start() # the script will block here until the crawling is finished

if __name__ == '__main__':
    main()

answered Jan 22, 2019 at 7:53

Granitosaurus

21.6k6 gold badges64 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Running a scrapy program from another python script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related