Scrape a websites's javascript variables using NodeJS

Question

I am trying to scrape the real time values of variables of a website using my NodeJS application.

I have tried "cheerio" but it didn't worked. It only returns me the HTML of the target in a string not the runtime values of the variables.

For example the value of "google.authuser" of "www.google.com"

screenshot

Please suggest me a simple solution. Thanks.

You are looking for a headless browser (that runs the code in the website) then. — Bergi
– Bergi, Commented Jan 28, 2019 at 17:44
+1 @Bergi, a headless browser will do the job, you can then do something like .executeScript(...) and get the value back. — Yassine Addi
– Yassine Addi, Commented Jan 28, 2019 at 17:46

Dürrani · Accepted Answer · 2019-01-28 18:02:53Z

3

Thanks for the recommendations of the Headless browser. Headless Chrome NodeJS API Puppeteer worked for me.

async function crawl() {
const puppeteer = require('puppeteer');
const browser = await puppeteer.launch();

const page = await browser.newPage();
await page.goto('http://www.google.com');

const dataPromise = await page.evaluate(() => {
    return Promise.resolve({
        number: google.authuser
    });
});

browser.close();
return dataPromise;}

answered Jan 28, 2019 at 18:02

Dürrani

1,2251 gold badge17 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

vsemozhebuty Over a year ago

From page.evaluate() doc: "If the function passed to the page.evaluate returns a Promise, then page.evaluate would wait for the promise to resolve and return its value." So your code can be simplified a bit: const data = await page.evaluate(() => google.authuser);

Collectives™ on Stack Overflow

Scrape a websites's javascript variables using NodeJS

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related