Scrapy: Converting Javascript array to Json on Python

Question

I have been struggling with a site I am scrapping using scrappy. This site, returns a series of Javascript variables (array) with the products data. Example:

datos[0] = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
datos[1] = ["12346","3M GREEN CAT5E CABLE","7.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
...

So on...

Fetching the array into a string with scrapy was easy, since the site response prints the variables. The problem is I want to transform it into Json so I can process it and store it in a database table.

Normally I would use Javascript's function Json.stringify to convert it to Json and post it in PHP.

However when using Python's json.loads and even StringIO I am unable to load the array into json.

Probably is a format error, but I am unable to identify it, since I am not expert in Json nor Python.

EDIT: I just realize since scrapy is unable to execute Javascript probably the main issue is that the data is just a string. I should format it into a Json format.

Any help is more than welcome.

Thank you.

Could you check if the below answer works for you? I'll appreciate it if you mark it as accepted if it works. Thanks! — gutsytechster
– gutsytechster, Commented Jun 29, 2022 at 9:35

Max · Accepted Answer · 2022-06-17 23:07:17Z

2

If you wanted to take an array and create a json object, you could do something like this.

values = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1]
keys = [x for x in range(len(values))]
d = dict(zip(keys, values))
x = json.dumps(d)

answered Jun 17, 2022 at 23:07

Max

836 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

gutsytechster · Accepted Answer · 2022-06-18 09:15:21Z

1

There is a section in the scrapy doc to find various ways to parse the JavaScript code. For your case, if you just need to have it in an array, you can use the regex to get the data.

Since the website you are scraping is not present in the question, I am assuming this would be a more straightforward way to get it, but you could use whichever way seems suitable.

answered Jun 18, 2022 at 9:15

gutsytechster

1316 bronze badges

Collectives™ on Stack Overflow

Scrapy: Converting Javascript array to Json on Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related