Find a Javascript variable definition with BeautifulSoup

Question

I am trying to find a specific variable definition inside a site's HTML (in a tag) I have the following code:

logResponse = scrape.post(url, params=logindata, headers=UA)
soup = bs(logResponse.text, 'html.parser')
x = soup.find_all('var my_post_key')
print(x)

The variable I am looking for is "my_post_key", but the soup.find_all function returns an empty list ([]). I suspect I am using it wrong, but am wondering how one would do this properly. This is how the variable is stored in page's HTML:

<script type="xxxxxxxxxxxxxxxxx-text/javascript">
var my_post_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
</script>

To recap, I am just trying to fetch the "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" value. Any help is appreciated.

find_all() looks for tags, and "var my_post_key" isn't a tag. Use find_all('script') instead, then search through those results. — John Gordon
– John Gordon, Commented Apr 29, 2020 at 21:34
@JohnGordon thank you, but then is there a way to "search through the results" as you said? — Mave
– Mave, Commented Apr 29, 2020 at 21:40
@Samuurai Pay attention that find and select_one will return only the first found tag, while findAll and select will return a ResultSet, and then you can parse the element using .text or .string, so you can split then according to your desired, something like split('"')[1] or using regex — αԋɱҽԃ αмєяιcαη
– αԋɱҽԃ αмєяιcαη, Commented Apr 29, 2020 at 21:44

αԋɱҽԃ αмєяιcαη · Accepted Answer · 2020-04-29 21:45:07Z

2

from bs4 import BeautifulSoup
import re


html = """
<script type="xxxxxxxxxxxxxxxxx-text/javascript">
var my_post_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
</script>
"""

soup = BeautifulSoup(html, 'html.parser')
target = soup.select_one("script").string


print(target.split('"')[1])

#Or

match = re.search(r'key = \"(.+?)\"', target).group(1)

print(match)

Output:

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

answered Apr 29, 2020 at 21:45

αԋɱҽԃ αмєяιcαη

11.6k3 gold badges23 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Find a Javascript variable definition with BeautifulSoup

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related