What is the most efficient way in JavaScript to parse huge amounts of data from a file

I would recommend a SAX based parser for these kind of JavaScript or a stream parser.

DOM parsing would load the whole thing in memory and this is not the way to go by for large files like you mentioned.

For Javascript based SAX Parsing (in XML) you might refer to https://code.google.com/p/jssaxparser/

and

for JSON you might write your own, the following link demonstrates how to write a basic SAX based parser in Javascript http://ajaxian.com/archives/javascript-sax-based-parser

Have you tried encoding it to a binary and transferring it as a blob?

https://developer.mozilla.org/en-US/docs/DOM/XMLHttpRequest/Sending_and_Receiving_Binary_Data

http://www.htmlgoodies.com/html5/tutorials/working-with-binary-files-using-the-javascript-filereader-.html#fbid=LLhCrL0KEb6

There isn't a really good way of doing that, because the whole file is going to be loaded into memory and we all know that all of them have big memory leaks. Can you not instead add some paging for viewing the contents of that file?

Check if there are any plugins that allow you to read the file as a stream, that will improve this greatly.

UPDATE

http://www.html5rocks.com/en/tutorials/file/dndfiles/

You might want to read about the new HTML5 API's to read local files. You will have the issue with downloading 250mb of data still tho.

I can think of 1 solution and 1 hack

SOLUTION: Extending the split the data in chunks: it boils down to http protocol. REST parts on the notion that http has enough "language" for most client-server scenarios.

You can setup on the client a request header Content-len to establish how much data you need per request

Then on the backend have some options http://httpstatus.es

• Reply a 413 if the server is simply unable to get that much data from the db
• 417 if the server is able to reply but not under the requested header (Content-len)
• 206 with the provided chunk, letting know the client "there is more from where that came from"

HACK: Use Websocket and get the binary file. Then use the html5 FileAPI to load it into memory. This is likely to fail though because its not the download causing the problem, but the parsing of an almost-endless JS object

You're out of luck on the browser. Not only do you have to download the file, but you'll have to parse the json regardless. Parse it on the server, break it into smaller chunks, store that data into the db, and query for what you need.