Scraping javascript data within a grid of a webpage

Question

I am really new to web scraping and I am working on a project, where I need to scrape data from a grid that loads and needs to be scrolled in order to fetch all the values.
The webpage is (https://applipedia.paloaltonetworks.com/).

I need all the data within the grid - (data containing NAME , CATEGORY, SUBCATEGORY, RISK, TECHNOLOGY).

Can anyone please guide me through the way I should tackle this problem. I have researched and found out that selenium with js or phantomjs might be a good solution but not really sure about it. The programming part I will be using Python.

Yes. Using a headless browser is the way to go. Beyond that; this question is far too broad in scope per guidleines in the help center — charlietfl
– charlietfl, Commented Aug 6, 2018 at 21:14

cruisepandey · Accepted Answer · 2018-08-07 04:30:07Z

1

You can use this code to scrape everything from required web site:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 

driver   = webdriver.Chrome(executable_path = r'C:/Users/abhishep/Downloads/chromedriver_win32/chromedriver.exe')
driver.maximize_window()

driver.get("https://applipedia.paloaltonetworks.com/") 

wait = WebDriverWait(driver,30)

table = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'tbody#bodyScrollingTable tr')))

for tab in table:
  print(tab.text)

answered Aug 7, 2018 at 4:30

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Swordsman Over a year ago

Thank you so much cruisepandey. Can you please share some links on best learning selenium or such tasks

cruisepandey Over a year ago

you can follow this basic but official docs selenium-python.readthedocs.io

Swordsman Over a year ago

How do I get data from the names which only has subdomains. For example: in line number 5: 2ch has 2 subdomains |_2ch-base and 2ch-posting. Like this I only want to get the list of apps having subdomains

cruisepandey Over a year ago

@Swordsman : It would be nice , if you just could raise a new ticket/concerns. SO users will be happy to help you.

Swordsman Over a year ago

sure. I ll do the same

|

Collectives™ on Stack Overflow

Scraping javascript data within a grid of a webpage

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related