2

I would like to download automatically the table from this link: https://www.barchart.com/options/iv-rank-percentile/stocks

In order to do this, with the help of some tutorials, I wrote this piece of code:

# Import libraries
from urllib.request import Request, urlopen
import requests
from bs4 import BeautifulSoup as soup

# Set the URL you want to webscrape from
url = 'https://www.barchart.com/options/iv-rank-percentile/stocks?viewName=main'

# Connect to the URL
response = requests.get(url)
print(response)

req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
print(req)

# Parse HTML and save to BeautifulSoup object¶
page_soup = soup(webpage, "html.parser")
#print(page_soup)

containers = page_soup.findAll("a", "toolbar-button download")

for container in containers:
    print(container)
    url = container.get('href')
    print(url)

The results I printed are the followings:

<Response [403]>
<urllib.request.Request object at 0x030766F0>
<a class="toolbar-button download" data-bc-download-button="  Stocks IV Rank and IV Percentile  "> <i class="bc-glyph-download"></i> <span>download</span></a>
None

It seems that I'm not able to find the 'href'

At this point, I'm struggling a bit with the next steps, because I don't really know how I can download the file (since the 'href' can't be found).

Maybe someone can help / or propose another solution?

Thanks a lot in advance,

Marketwizard

Improve this question

2 Answers 2

Reset to default
4

The data is loaded dynamically via Javascript from different URL. You can use this example how to load the data:

import json
import requests
from urllib.parse import unquote


headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0"
}

url = "https://www.barchart.com/proxies/core-api/v1/quotes/get?list=options.mostActive.us&fields=symbol,symbolName,lastPrice,priceChange,percentChange,optionsTotalVolume,optionsWeightedImpliedVolatility,optionsImpliedVolatilityRank1y,optionsImpliedVolatilityPercentile1y,optionsWeightedImpliedVolatilityHigh1y,tradeTime,symbolCode,symbolType,hasOptions&between(lastPrice,.10,)=&between(tradeTime,2021-03-22,2021-03-23)=&orderBy=optionsTotalVolume&orderDir=desc&meta=field.shortName,field.type,field.description&hasOptions=true&page=1&limit=100&raw=1"

with requests.Session() as s:
    # get all cookies
    s.get(
        "https://www.barchart.com/options/iv-rank-percentile/stocks",
        headers=headers,
    )
    # use one cookie as HTTP header
    headers["X-XSRF-TOKEN"] = unquote(s.cookies["XSRF-TOKEN"])
    data = s.get(url, headers=headers).json()


# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for d in data["data"]:
    print("{:<8}{:<50}{}".format(d["symbol"], d["symbolName"], d["lastPrice"]))

Prints:

AAPL    Apple Inc                                         123.39
TSLA    Tesla Inc                                         670.00
FB      Facebook Inc                                      293.54
AMC     Amc Entertainment Holdings Inc                    12.49
PLTR    Palantir Technologies Inc Cl A                    24.22
NIO     Nio Inc                                           42.94
AMD     Adv Micro Devices                                 80.30
F       Ford Motor Company                                12.85
SNDL    Sundial Growers Inc                               1.3000
BAC     Bank of America Corp                              37.66
MSFT    Microsoft Corp                                    235.99
BABA    Alibaba Group Holding                             237.12
BA      Boeing Company                                    251.23
GE      General Electric Company                          13.13
AAL     American Airlines Gp                              23.83
DKNG    Draftkings Inc                                    71.72
WFC     Wells Fargo & Company                             38.97
AMZN    Amazon.com Inc                                    3,110.87
GM      General Motors Company                            58.10
INTC    Intel Corp                                        65.63
GME     Gamestop Corp                                     194.49
SNAP    Snap Inc                                          58.16
SOS     Sos Ltd                                           6.90
PFE     Pfizer Inc                                        36.00
NOK     Nokia Corp                                        4.06
T       AT&T Inc                                          29.99
CCL     Carnival Corp                                     27.48
NVDA    Nvidia Corp                                       527.11
MARA    Marathon Digital Hldgs Inc                        39.97
FTCH    Farfetch Ltd Cl A                                 62.00
UBER    Uber Technologies Inc                             55.69
TLRY    Tilray Inc                                        23.90
DIS     Walt Disney Company                               192.86
FCEL    Fuelcell Energy Inc                               15.04
QS      Quantumscape Corp                                 64.29
SQ      Square                                            226.13
CCIV    Churchill Capital IV Cl A                         26.15
V       Visa Inc                                          208.00
CSCO    Cisco Systems Inc                                 50.30
XOM     Exxon Mobil Corp                                  55.91
FCX     Freeport-Mcmoran Inc                              35.01
JPM     JP Morgan Chase & Company                         150.97
PLUG    Plug Power Inc                                    38.91
NFLX    Netflix Inc                                       523.11
VALE    Vale S.A.                                         17.01
TEVA    Teva Pharmaceutical Industries Ltd                11.93
CLF     Cleveland-Cliffs Inc                              15.86
MU      Micron Technology                                 91.27
BOX     Box Inc                                           23.65
TSM     Taiwan Semiconductor Manufacturing                117.18
RIOT    Riot Blockchain Inc                               56.01
BLNK    Blink Charging Company                            40.66
VZ      Verizon Communications Inc                        56.59
UAL     United Airlines Holdings Inc                      58.33
QCOM    Qualcomm Inc                                      134.09
CLVS    Clovis Oncology Inc                               7.47
RLX     Rlx Technology Inc ADR                            10.15
LUMN    Centurylink                                       14.37
WMT     Wal-Mart Stores                                   132.37
TWTR    Twitter Inc                                       65.21
NCLH    Norwegian Cruise Ord                              28.65
GOOGL   Alphabet Cl A                                     2,030.69
C       Citigroup Inc                                     71.96
JD      Jd.com Inc Ads                                    84.97
BB      Blackberry Ltd                                    10.71
X       United States Steel Corp                          21.79
RKT     Rocket Companies Inc Cl A                         22.99
PDD     Pinduoduo Inc ADR                                 137.15
NLY     Annaly Capital Management Inc                     8.92
FUBO    Fubotv Inc                                        31.53
MO      Altria Group                                      51.64
DASH    Doordash Inc Cl A                                 135.91
UWMC    Uwm Hldg Corp                                     8.78
KSS     Kohl's Corp                                       58.74
DAL     Delta Air Lines Inc                               47.97
NKLA    Nikola Corp                                       15.55
LYFT    Lyft Inc Cl A                                     64.13
WKHS    Workhorse Grp                                     15.63
PENN    Penn Natl Gaming Inc                              113.16
CRM     Salesforce.com Inc                                215.17
XPEV    Xpeng Inc ADR                                     37.88
BCRX    Biocryst Pharma Inc                               11.80
ET      Energy Transfer LP                                8.10
PTON    Peloton Interactive Inc                           109.54
BIDU    Baidu Inc                                         266.13
NKE     Nike Inc                                          138.27
PSTH    Pershing Square Tontine Hldgs Cl A                25.89
ACB     Aurora Cannabis Inc                               9.70
PYPL    Paypal Holdings                                   244.38
TME     Tencent Music Entertainment Group ADR             30.87
CAN     Canaan Inc ADR                                    22.97
GOLD    Barrick Gold Corp                                 20.62
SPCE    Virgin Galactic Holdings Inc                      32.24
ZM      Zoom Video Communications Cl A                    328.50
NNDM    Nano Dimension Ads                                9.83
CVX     Chevron Corp                                      102.54
SPRT    Support.com Inc                                   7.10
OXY     Occidental Petroleum Corp                         27.46
COST    Costco Wholesale                                  334.49
USAT    USA Technologies Inc                              12.45
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

From the comment over at Scraping pre-market table on Barchart with Python, all that was required was to add more headers to the request.

You can find your own headers using Chrome > Developer Tools; and then just find the API request for the table and add in a few of the headers associated with that request.

Right-click/ Inspect / Network (then just find the API request for the table and add in a few of the headers associated with that request).

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.