Python interactive plotting for large data sets

Question

Suppose I have a dataset with 100k rows (1000 different times, 100 different series, an observation for each, and auxilliary information). I'd like to create something like the following: (1) first panel of plot has time on x axis, and average of the different series (and standard error) on y axis. (2) based off the time slice (vertical line) we hover over in panel 1, display a (potentially down sampled) scatter plot of auxilliary information versus the series value at that time slice.

I've looked into a few options for this: (1) matplotlib + ipywidgets doesn't seem to handle it unless you explicitly select points via a slider. This also doesn't translate well to html exporting. This is not ideal, but is potentially workable. (2) altair - this library is pretty sleek, but from my understanding, I need to give it the whole dataset for it to handle the interactions, but it also can't handle more than 5kish data points. This would preclude my use case, correct?

Any suggestions as to how to proceed? Is what I'm asking impossible in the current state of things?

take a look at Plotly.

peter
– peter

2021-04-05 20:03:34 +00:00
Commented Apr 5, 2021 at 20:03 — peter
– peter, Commented Apr 5, 2021 at 20:03

joelostblom · Accepted Answer · 2021-04-06 15:18:59Z

1

You can work with datasets larger than 5k rows in Altair, as specified in this section of the docs.

One of the most convenient solutions in my opinion is to install altair_data_server and then add alt.data_transformers.enable('data_server') on the top of your notebooks and scripts. This server will provide the data to Altair as long as your Python process is running so there is no need to include all the data as part of the created chart specification, which means that the 5k error will be avoided. The main drawback is that it wont work if you export to a standalone HTML because you rely on being in an environment where the server Python process is running.

answered Apr 6, 2021 at 15:18

joelostblom

49.8k20 gold badges166 silver badges179 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python interactive plotting for large data sets

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related