ruby nokogiri restclient to scrape javascript variable

Question

I'm using restclient and nokogiri to parse some html which works great, but there is one piece of information stored in a js (jquery) variable which I need to return and I'm not sure how to parse it. I can use Nokogiri to parse the javascript block, but I need one subset of it which is probably simple but I'm not sure how to do it. I could probably regex it but I'm assuming there's an easier way to just ask for it using JS.

@resource = RestClient.get 'http://example.com'

doc = Nokogiri::HTML(@resource)

doc.css('script').each do |script|
    puts script.content
end

What I'm trying to get:

        <script type="text/javascript">
            $(function(){
                //this is it
                $.Somenamespace.theCurrency = 'EUR';
                //a lot more stuff

Vlad Khomich · Accepted Answer · 2011-04-18 10:08:33Z

3

not sure if that fits, but you could retrieve it as follows:

irb(main):017:0>

string
=> "<script type=\"text/javascript\">    $(function(){$.Somenamespace.theCurrency = \"EUR\"}); "

irb(main):018:0>

string.scan(/\$\.Somenamespace\.(.*)}\);/)
=> [["theCurrency = \"EUR\""]]

answered Apr 18, 2011 at 10:08

Vlad Khomich

5,9001 gold badge30 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

the Tin Man · Accepted Answer · 2011-04-18 19:19:13Z

Nokogiri is an XML and HTML parser. It doesn't parse the CDATA or text content of nodes, but it can give you the content, letting you use string parsing or regex to get at the data you want.

In the case of Javascript, if it's embedded in the page then you can get the text of the parent node. Often that is simple:

js = doc.at('script').text

if there is the usual <script> tag in the <head> block of the page. If there are multiple script tags you have to extend the accessor to retrieve the right node, then process away.

It gets more exciting when the scripts are loaded dynamically, but you can still get the data by parsing the URL from the script's src parameter, then retrieving it, and processing away again.

Sometimes Javascript is embedded in the links of other tags, but it's just another spin on the previous two methods to get the script and process it.

Collectives™ on Stack Overflow

ruby nokogiri restclient to scrape javascript variable

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related