matplotlib-users Mailing List for matplotlib (Page 1453)

Perry:
Currently using connected line plots, but do not want to limit myself in 
any way when it comes to presenting data. I am certain that at one 
point, I will use every plot available in the matplotlib arsenal. On a 
3.2Ghz P4 with 2GB RAM get ~90 seconds for a 100,000 data set, ~50 
seconds for 50,000 and ~9 seconds for a 10,000 (sort of linear). This is 
way to long for my purposes. I was hoping more for ~5 seconds for 
100,000 points.

John:

I routinely plot data sets this large.  500,000 data points is a

>I routinely plot data sets this large.  500,000 data points is a
>typical 10 seconds of EEG, which is the application that led me to
>  
>
That sounds good!

>If your xdata are sorted, ie like time, the following
>
>  l = plot(blah, blah)
>  set(l, 'lod', True)
>
>could be a big win.
>
>Whether this is appropriate or not depends on the data set of course,
>whether it is continuous, and so on.  Can you describe your dataset in
>more detail, because I would like to add whatever optimizations are
>appropriate -- if others can pipe in here too that would help.
>  
>
Will mostly be plotting time Vs value(time) but in certain cases will 
need plots of other data, and therefore have to look at the worst case 
scenario.
Not exactly sure what you mean by "continuous" since all are descrete 
data points. The data may not be smooth (could have misbehaving sensors 
giving garbage) and jump all over the place.

>econdly, the standard gdmodule will iterate over the x, y values in a
>python loop in gd.py.  This is slow for lines with lots of points.  I
>have a patched gdmodule that I can send you (provide platform info)
>that moves this step to the extension module.  Potentially a very big
>win.
>  
>
Yes, that would be great!
System info:

OS: RedHat9 ( kernel 2.4.20)

gcc version from running 'gcc -v':
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
--infodir=/usr/share/info --enable-shared --enable-threads=posix 
--disable-checking --with-system-zlib --enable-__cxa_atexit 
--host=i386-redhat-linux
Thread model: posix
gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)

Python: Python 2.2.2 (#1, Feb 24 2003, 19:13:11)

matplotlig: matplotlib-0.50e

gdpython: 0.51 (with modified _gdmodule.c)

gd: gd-2.0.21

>Another possibility: change backends.  The GTK backend is
>significantly faster than GD.  If you want to work off line (ie, draw
>to image only and not display to screen ) and are on a linux box, you
>can do this with GTK and Xvfb.  I'll give you instructions if
>interested.  In the next release of matplotlib, there will be a libart
>paint backend (cross platform) that may be faster than GD.  I'm
>working on an Agg backend that should be considerably faster than all
>the other backends since it does everything in extension code -- we'll
>see 
>
Yes I am only planning to work offline. Want to be able to pipe the 
output images to stdout. I am looking for the fastest solution possible.

Thanks again.
Peter

John Hunter writes:
> 
> could be a big win.  LOD is "Level of Detail" and if true subsamples
> the data according to the pixel width of the output, as you described.
> Whether this is appropriate or not depends on the data set of course,
> whether it is continuous, and so on.  Can you describe your dataset in
> more detail, because I would like to add whatever optimizations are
> appropriate -- if others can pipe in here too that would help.
> 
>
What I was alluding to was that if a backend primitive was added that
allowed plotting a symbol (patch?) or point for an array of points.
The base implementation would just do a python loop over the single
point case so there is no requirement for a backend to overload this
call. But it could do so if it wanted to loop over all points in C. 
How flexible to make this is open to discussion (e.g., allowing
x, and y scaling factors, as arrays, for the symbol to be plotted, and
other attributes that may vary with point such as color)

Perry 

>>>>> "Peter" == Peter Groszkowski <pgr...@ge...> writes:

    Peter> Hello: We will be dealing with large (> 100,000 but in some
    Peter> instances as big as 500,000 points) data sets. They are to
    Peter> be plotted, and I would like to use matplotlib.

Are you working with plot/loglog/etc (line data) or
pcolor/hist/scatter/bar (patch data)?

I routinely plot data sets this large.  500,000 data points is a
typical 10 seconds of EEG, which is the application that led me to
write matplotlib.  EEG is fairly special: the x axis time is
monotonically increasing and the y axis is smooth.  This lets me take
advantage of level of detail subsampling.

If your xdata are sorted, ie like time, the following

  l = plot(blah, blah)
  set(l, 'lod', True)

could be a big win.  LOD is "Level of Detail" and if true subsamples
the data according to the pixel width of the output, as you described.
Whether this is appropriate or not depends on the data set of course,
whether it is continuous, and so on.  Can you describe your dataset in
more detail, because I would like to add whatever optimizations are
appropriate -- if others can pipe in here too that would help.

Secondly, the standard gdmodule will iterate over the x, y values in a
python loop in gd.py.  This is slow for lines with lots of points.  I
have a patched gdmodule that I can send you (provide platform info)
that moves this step to the extension module.  Potentially a very big
win.

Another possibility: change backends.  The GTK backend is
significantly faster than GD.  If you want to work off line (ie, draw
to image only and not display to screen ) and are on a linux box, you
can do this with GTK and Xvfb.  I'll give you instructions if
interested.  In the next release of matplotlib, there will be a libart
paint backend (cross platform) that may be faster than GD.  I'm
working on an Agg backend that should be considerably faster than all
the other backends since it does everything in extension code -- we'll
see :-).

JDH

How are you plotting the data? As a scatter plot (e.g., symbols
or points) or as a connected line plot. The former can be quite
a bit slower and we have some thoughts about speeding that up
(which we haven't broached with JDH yet). How long is it taking
and how much faster do you need it?

Perry Greenfield

> -----Original Message-----
> From: mat...@li...
> [mailto:mat...@li...]On Behalf Of Peter
> Groszkowski
> Sent: Wednesday, February 11, 2004 2:14 PM
> To: mat...@li...
> Subject: [Matplotlib-users] large data sets and performance
> 
> 
> Hello:
> 
> We will be dealing with large (> 100,000 but in some instances as big as 
> 500,000 points) data sets. They are to be plotted, and I would like to 
> use matplotlib. I did a few preliminary tests, and it seems like 
> plotting that many pairs is a little too much for the system to handle. 
> Currently we are using (as a backend to some other software) gnuplot for 
> doing this plotting. It seems to be "lighting-fast" but I suspect (may 
> be wrong!) that it reduces this data before the plotting takes place, 
> and only selects every nth point. I have to go through the code that 
> calls it to be certain. I would imagine that it is not necessary to get 
> evrey point in 100,000 to produce a page-size plot, but I'm not sure if 
> simply grabbing every nth point and reducing the data like that is the 
> best way about this. So my question is to anyone else out there who is 
> also dealing with these large (and very large) data sets? What do you 
> do? Any library routines that you use before plotting to massage that 
> data? Are there any ways (ie flags to set) to optimize this in 
> matplotlib? Any other software you use? I should note that I use the GD 
> backend and pipe the output to stdout for a cgi scrpit to pick up.
> 
> Thanks.
> 
> -- 
> Peter Groszkowski         Gemini Observatory
> Tel: +1 808 974-2509      670 N. A'ohoku Place
> Fax: +1 808 935-9235      Hilo, Hawai'i 96720, USA
> 
> 
> 
> 
> -------------------------------------------------------
> SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> Build and deploy apps & Web services for Linux with
> a free DVD software kit from IBM. Click Now!
> http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> _______________________________________________
> Matplotlib-users mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-users
> 

Hello:

We will be dealing with large (> 100,000 but in some instances as big as 
500,000 points) data sets. They are to be plotted, and I would like to 
use matplotlib. I did a few preliminary tests, and it seems like 
plotting that many pairs is a little too much for the system to handle. 
Currently we are using (as a backend to some other software) gnuplot for 
doing this plotting. It seems to be "lighting-fast" but I suspect (may 
be wrong!) that it reduces this data before the plotting takes place, 
and only selects every nth point. I have to go through the code that 
calls it to be certain. I would imagine that it is not necessary to get 
evrey point in 100,000 to produce a page-size plot, but I'm not sure if 
simply grabbing every nth point and reducing the data like that is the 
best way about this. So my question is to anyone else out there who is 
also dealing with these large (and very large) data sets? What do you 
do? Any library routines that you use before plotting to massage that 
data? Are there any ways (ie flags to set) to optimize this in 
matplotlib? Any other software you use? I should note that I use the GD 
backend and pipe the output to stdout for a cgi scrpit to pick up.

Thanks.

-- 
Peter Groszkowski         Gemini Observatory
Tel: +1 808 974-2509      670 N. A'ohoku Place
Fax: +1 808 935-9235      Hilo, Hawai'i 96720, USA

>>>>> "Jean-Baptiste" =3D=3D Jean-Baptiste Cazier <Jean-Baptiste.cazier@d=
ecode.is> writes:

    Jean-Baptiste> S=E6ll !  Thanks for the info and update. I upgaded
    Jean-Baptiste> my library and my program.  The dramatic change of
    Jean-Baptiste> API was a bit painful, but the new syntax is more
    Jean-Baptiste> clear.  Do you have any idea when you will hav
    Jean-Baptiste> reached a stable version in general and in term of
    Jean-Baptiste> version ?

It has been a bit painful, my apologies; I still have one application
to port myself.  But it was necessary.  matplotlib is undergoing
active development.  The basic idea is to write a backend for a
powerful image renderer, http://antigrain.com or libart , and use that
backend to draw to the GUIs.  Rather than each GUI implementing it's
own drawing, I'm moving to one high quality image renderer that will
draw to the GUI.

Why? Four major benefits

  * Easier maintenance: when I want to add a feature, I add it to the
    image backend and all the GUIs automatically benefit.

  * Enhanced drawing capabilities - the GUIs don't support a lot of
    the more sophisticated drawing capabilities, eg, paths from PS and
    SVG, or alpha blending, or gouraud shading.  The agg backend
    supports all of these, and therefore by extension, so will the
    GUIs.

  * Font standardization across backends.  With a common image
    renderer that supports freetype, all the GUIs can have freetype
    support with common font names.
=20
  * Ease of plain old image integration with figures - 2D image plots,
    like pcolor, will become very fast and very pretty.

Any solution along these lines will be performance competitive with
native GUI and work on all major platforms (win32, osx, linux, unix)
or we won't do it.

In order to do this right, I needed the Figure class (and all it's
children) to be totally backend independent.  So a WX figure can
render to a postscript renderer, or an agg renderer, or a gd renderer,
or a libart renderer.  In earlier releases of matplotlib, the AxisText
and Figure classes were tied up in the backend.

I have made it a policy not to predict a stable API because I don't
want to go back on my word.  That said, I think the existing design is
clean and I am happy with it.  I would not have said that a month ago.
I don't anticipate any major changes to the Figure API or the
FigureCanvas API.

Changes I do expect to see in the near future are

 * the addition of GTK/AGG, WX/AGG and Tkinter/AGG backends.  These
   will be optional so for those of you who want to stay with the
   classic GUI backends you can.  But I would encourage you to move
   over at that time since you'll get better drawing, more
   sophisticated plots, etc...  I will probably add a matplotlibrc
   file in which you can select your default backend.  This will be an
   almost trivial API port, if you choose to.  This will probably be
   around matplotlib-0.6 (1-2 months).

 * Addition of path drawing methods in the renderer API.  This will
   allow for the more sophisticated that state dependent paths
   support.  If you don't work directly with the renderer, it won't
   affect you.  It's mainly for backend implementers.

As for API stability in general, I have 2 additional thoughts.

 * for the matlab interface, if it's matlab compatible it's 99% likely
   to be stable.  We've made minor departures from this when there was
   clearly a better way to do it, but it's a good rule of thumb.

 * As with many projects, I think the 1.0 release should guarantee
   some API stability.  I predict you'll see a pretty stable API until
   then.

JDH

>>>>> "LUK" == LUK ShunTim <shu...@po...> writes:

    >> OK, now we at least know where the problem is.  I don't get
    >> such an error message on my system (rhl9, pygtk-2.0.0).  What
    >> platform are you on, and what versions of GTK and pygtk are you
    >> running?  JDH

    LUK> W2K, Enthought python 2.3, pygtk 2.0, gtk 2.0

Tracked this one down.  Apparently the latest version of GTK for
windows does different font aliasing.  This is controlled by the file

c:\gtk\etc\pango\pango.aliases

Try adding the line:

times = "times new roman,angsana new,mingliu,simsun,gulimche,ms gothic,latha,mangal,code2000"

If you get other font messages, adding more font aliases to this file
may help.

I'm making some changes in the gtk backend - mapping times -> serif,
so in the next release of matplotlib this should be fixed
automagically.

JDH

>>>>> "derek" == derek bandler <d_b...@ho...> writes:

    derek>  A small problem - I downloaded the tar/zipped file to my
    derek> pc, but winzip errors on onloading it.  I downloaded
    derek> something called power zip and have the same problem.
    derek> There's an error msg that the archive may be corrupt.

Oh, I assumed you were on a UNIX platform because of the Solaris
reference.  Here's a link to a windows installer


  http://nitace.bsd.uchicago.edu:8080/files/share/matplotlib-0.50j.win32.exe

    derek>    As far as the patch, would that require re-building the
    derek> software?

Yep, and this is far from trivial on win32.  Hopefully, the pygtk
maintainers will include the patch in the next release, but there has
been grumbling on the pygtk list about patches collecting dust.  In
any case, I cache most of the relevant stuff now, so repeated draws
should not expose the memory leak as before.

Give it a try.

JDH

Hi

You might know that octave is a free software clone of matlab. I used it 
heavily for a previous project. One thing I liked about it's plot command 
is an extension to the fmt string which allowed you to specify the legend 
title for each line between semicolons. Like this:

plot(x, sin(x), 'r;sine;')
plot(x, cos(x), 'g;cosine;')
plot(x, tan(x), 'b;tangent;')

It seems like a handy way to be able to do things.

I couldn't see this mentioned in the matplotlib docs, and when I tried it 
I got a dialog "Unrecognized character in format string". May I suggest it 
as a useful addition to matplotlib? It seems like legend() might need 
adjustment to allow for it too.

BTW, using the GTK backend the "Unrecognized character in format string"  
dialog has locked up my python session after I clicked OK. It appeared
straight after the plot command -- not when I ran show(). After I clicked 
OK, the window stayed visible but unresponsive to the close button etc.

m.

>>>>> "Paul" == Paul Barrett <ba...@st...> writes:

    Paul> I occasionally use the scatter() function to make light
    Paul> curves of astronomical targets.  The scale of the x-axis is
    Paul> often given in days with a range of 0.54 - 0.68 days.  The
    Paul> y-axis is in counts.  This type of data produces unusual
    Paul> scatter plots with the blue symbols being very elongated
    Paul> ovals (in the x-direction).

    Paul> Are there plans to change this behavior - maybe by drawing
    Paul> the symbols in device space instead of data or user space?

There are a couple of ways to do this.

The line style provides markers, the sizes of which are in points.  If
you want fixed size markers independent of data coords, you can use
plot with the 'o' marker symbol (or any other marker symbol), and then
set the marker size, facecolor, and edgecolor, as you desire

  markers = plot(x, y, 'o')
  set(markers, 'markersize', 20)
  set(markers, 'markeredgecolor', 'k')
  set(markers, 'markerfacecolor', 'b')

The set function takes a sequence of markers and applies the arguments
to all of them.  To control the marker properties separately, you can
do (for example)

  rgbs = [ ... ]   # a len x list of rgb tuples
  sizes = [ ... ]  # a len x list of marker sizes in points
  for m, rgb, size in zip(markers, rgbs, sizes):
      m.set_markerfacecolor(rgb)
      m.set_markersize(size)
   
See the documentation for matplotlib.lines for more info.

Patches (which is what scatter and hist create) on the other hand, are
in data coordinates by default.  This too can be changed by setting
the transformation functions.  You can, for example, specify the
patches in relative (0, 1) axes coords with a little extra work.  All
the figure "Artists" can be placed with arbitrary transforms -- see
matplotlib.transforms for more info.  Many of these features could use
some better documentation....

My guess is that you will be happy with the line markers -- let me
know.

JDH

I occasionally use the scatter() function to make light curves of 
astronomical targets.  The scale of the x-axis is often given in days 
with a range of 0.54 - 0.68 days.  The y-axis is in counts.  This type 
of data produces unusual scatter plots with the blue symbols being very 
elongated ovals (in the x-direction).

Are there plans to change this behavior - maybe by drawing the symbols 
in device space instead of data or user space?

  -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218

>>>>> "John" == John Gill <jn...@eu...> writes:

    John> I have some plots I'd like to print out and they would make
    John> better use of the paper if they were done landscape.

    John> Can the postscript backend do this?

Hi John,

I haven't had time to take a close look at this.  My initial
suggestions is to experiment with the paper size

import matplotlib
matplotlib.use('PS')
import matplotlib.backends.backend_ps as backend_ps
backend_ps.defaultPaperSize = 11,8.5  # default is 8.5, 11

You may also have to specify a landscape portrait at print time, or
rotate it.

I'll take a closer look later.

JDH

>>>>> "derek" == derek bandler <d_b...@ho...> writes:

    derek>    I am interested in using Matplotlib for real-time
    derek> updating plot applications.  My concern is with system load
    derek> issues (they're pretty high).  Using the code below, which
    derek> came off of an earlier posting to this list, on a pc (2.4
    derek> gig processor) memory usage ranges from 20 - 50+ megs
    derek> (there's a cycling pattern) and 30-40% of the cpu is being
    derek> used for this basic app.  If left to run for 20 minutes or
    derek> so, it seems to crash (entire screen turns grey and reboot
    derek> is required to get machine back to working order).

I answered this problem on the support request you submitted to
sourceforge a couple of days ago.  I'll paste in my response from
the sourceforge site

Date: 2004-02-06 14:22
Sender: jdh2358
Logged In: YES 
user_id=395152

Thanks for the bug report!

I have spent a lot of time optimizing matplotlib for
pltotting and interacting with large data sets, but not for
frequently redrawing small data sets.

Fortunately, I was able to replicate the bug on my system. 
I identified a memory leak in pygtk in a function I use for
rotating tick labels.

It turns out almost all the CPU usuage (and memory) was for
repeatedly computing layout information, rotations, and so
on for text onthe tick labels.  Since these rarely change
(eg, in system demo) I cache all the relevant information. 
I also cache the result that is causing the memory leak in
pygtk, so repeated calls will not drain memory.

The result is a system demo with CPU usage reduced from
30-40% to 4-8% and memory consumption reduced from a growing
memory leak to a stable 1.2%.

Not bad.  Please let me know if you continue to experience
performance issues.  I I have updated CVS.  I you have
trouble accessing CVS, email me at
jdh...@ac... and I'll send you a snapshot.

I have no idea about the solaris redraw problem currently

JDH

Date: 2004-02-06 15:51
Sender: jdh2358
Logged In: YES 
user_id=395152

I tracked down the memory leak in pygtk.  I you are
intersting in patching your pygtk, the bug report and patch
are here

http://bugzilla.gnome.org/show_bug.cgi?id=133681

<html><div style='background-color:'><DIV class=RTE>
<P>I am interested in using Matplotlib for real-time updating plot applications.&nbsp; My concern is with system load issues (they're pretty high).&nbsp; Using the code below, which came off of an earlier posting to this list, on a pc (2.4 gig processor) memory usage ranges from 20 - 50+ megs (there's a cycling pattern) and 30-40% of the cpu is being used for this&nbsp;basic app.&nbsp; If left to run for 20 minutes or so, it seems to crash (entire screen turns grey and reboot is required to get machine back to working order).</P>
<P>On a solaris machine, the performance numbers are similar, though the memory didn't appear to exceed 30 megs (or cycle).&nbsp; One other solaris&nbsp;issue is that the&nbsp;plot doesn't update, unless it is forced to redraw (by moving another window over the plot window).&nbsp; I've tinkered with the redraw rate, but it didn't seem to help.</P>
<P>Short of the performance issues, this software looks great.&nbsp; Is there a way of using the software that will minimize system impact, or better yet,&nbsp;can this become&nbsp;a development priority?&nbsp; It's not clear if the performance issue is driven by matplotlib, pygtk or gtk.&nbsp; Is it possible that different backends lead to non-trivial differences in&nbsp;performance?&nbsp; </P>
<P>Also, the crashing on a pc and non-updating on solaris are outstanding concerns.</P>
<P>Derek Bandler</P>
<P>####</P>
<P>import pygtk<BR>pygtk.require('2.0')<BR>import gtk<BR>import time<BR>from matplotlib.matlab import *<BR>&nbsp;<BR>def get_memory():<BR>&nbsp;&nbsp;&nbsp; "Simulate a function that returns system memory"<BR>&nbsp;&nbsp;&nbsp; return 100*(1+sin(0.5*pi*time.time()))<BR>&nbsp;<BR>def get_cpu():<BR>&nbsp;&nbsp;&nbsp; "Simulate a function that returns cpu usage"<BR>&nbsp;&nbsp;&nbsp; return 100*(1+sin(0.2*pi*(time.time()-0.25)))<BR>&nbsp;<BR>def get_net():<BR>&nbsp;&nbsp;&nbsp; "Simulate a function that returns network bandwidth"<BR>&nbsp;&nbsp;&nbsp; return 100*(1+sin(0.7*pi*(time.time()-0.1)))<BR>&nbsp;<BR>def get_stats():<BR>&nbsp;&nbsp;&nbsp; return get_memory(), get_cpu(), get_net()<BR>&nbsp;<BR>fig = figure(1)<BR>ax = subplot(111)<BR>ind = arange(1,4)<BR>pm, pc, pn = bar(ind, get_stats())<BR>centers = ind + 0.5*pm.get_width()<BR>pm.set_facecolor('r')<BR>pc.set_facecolor('g')<BR>pn.set_facecolor('b')</P>
<P>ax.set_xlim([0.5,4])<BR>ax.set_xticks(centers)<BR>ax.set_ylim([0,100])<BR>ax.set_xticklabels(['Memory', 'CPU', 'Bandwidth'])<BR>ax.set_ylabel('Percent usage')<BR>ax.set_title('System Monitor')<BR>&nbsp;<BR>def updatefig(*args):<BR>&nbsp;&nbsp;&nbsp; m,c,n = get_stats()<BR>&nbsp;&nbsp;&nbsp; pm.set_height(m)<BR>&nbsp;&nbsp;&nbsp; pc.set_height(c)<BR>&nbsp;&nbsp;&nbsp; pn.set_height(n)<BR>&nbsp;&nbsp;&nbsp; ax.set_ylim([0,100])</P>
<P>&nbsp;&nbsp;&nbsp; fig.draw()<BR>&nbsp;&nbsp;&nbsp; <BR>&nbsp;&nbsp;&nbsp; return gtk.TRUE</P>
<P>gtk.timeout_add(250, updatefig)<BR>show()</P></DIV></div><br clear=all><hr> <a href="http://g.msn.com/8HMBENUS/2752??PS=">Create your own personal Web page with the info you use most, at My MSN.</a> </html>

I have some plots I'd like to print out and they would make better use
of the paper if they were done landscape.

Can the postscript backend do this?

John

>>>>> "LUK" == LUK ShunTim <shu...@po...> writes:

    >> OK, now we at least know where the problem is.  I don't get
    >> such an error message on my system (rhl9, pygtk-2.0.0).  What
    >> platform are you on, and what versions of GTK and pygtk are you
    >> running?  JDH
    >> 

    LUK> W2K, Enthought python 2.3, pygtk 2.0, gtk 2.0

Hi Luk,

I get the same error message on my WinXP platform.  I recently
decoupled the text code from the various backends (gtk, ps, etc).  In
the past, each backend specified their own default font, and this is
no longer possible since text is now backend independent.  On the
upside, this enables you to save PS or EPS from a GTK or WX GUI
window.  On the downside, this exposes the problem that font choosing
is not backend independent.  We've talked a number of times about the
importance of this on matplotlib-devel, but haven't done the dirty
work of actually getting it implemented.

So in a nutshell, there is nothing wrong with your setup.  We on the
development side just need to spend some time getting fonts
standardized across backends.  Ie, we need to agree on a bare minimum
number of fonts with standard names that we can provide on all the
backends, so scripts using these names will work on all the backends.
This is complicated by the fact that most fonts are proprietary, so we
can't ship them or guarantee their existence on a given system.

It's on the list of things to do.  If this proves to be a major
impediment for you in the mean time, don't hesitate to bring it up
again.  Sometimes those who complain the loudest get their bugs fixed
the soonest.

JDH

Thanks, John!  Funny thing, I, too, recently installed Glade!  That's =
it!
Best,
Dave

>>>>> "Nordquest," == Nordquest, David A <NOR...@ga...> writes:

    Nordquest> BTW, if I do the pygtk test ( >>>import pygtk >>>
    Nordquest> pygtk.require('2.0')
    >>>> import gtk ) once, I get an error message.  If I do it again
    >>>> without changing anything, I get no error message.

This reimport situation you describe is expected.  If you import a
module a second time, python simply ignores it.  So if it failed the
first time, it will fail silently the next times.  You should exist
python and start over.  The problem you are experiencing is definitely
with gtk and not matplotlib.

In these situations, the best thing to do is go into DOS or a command
shell (Start->Run->command ENTER).  Write a little script test.py that
contains only

import pygtk
pygtk.require('2.0')
import gtk

Then go into the shell, an cd into the dir containing test.py.

Manually set your PATH, something like (depends on which windows
platform you are on)

  c:> set PATH=c:\GTK\bin;C:\GTK\lib;C:\Python23;C:\windows\command
  c:> python test.py

I know you've checked your path ad nauseum, but there is still a
decent change is the cause of your problem, 9 times out of 10.

Hey, didn't you solve this once before :-) ?  Is this a new platform
for you?  Did you reinstall GTK, if so to where?  What does

  c:> dir c:\GTK\bin

reveal?  I assume you've reread
http://www.async.com.br/faq/pygtk/index.py?req=show&file=faq21.012.htp
100 times.

There is a long thread on the pygtk mailing list where Cousing Stanley
got his gtk corrupted by installing glade, which writes some older
libgtk versions into the windows system dir.  Do a file search for
libgtk and make sure nothing shows up outside of your GTK install
tree.  Read this thread
http://www.mail-archive.com/py...@da.../msg07324.html, which is
filled with good advice.

Good luck!
JDH

>>>>> "Peter" == Peter Groszkowski <pgr...@ge...> writes:

    Peter> plot(date_in_some_format, data)

Are the dates strings, python datetime instances, mx datetime
instances, or what?

Here is a thread on python-list you may be interested in

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=mailman.766.1068941179.702.python-list%40python.org&rnum=1&prev=/groups%3Fq%3Dmatplotlib%2Bdate%26ie%3DUTF-8%26oe%3DUTF-8%26hl%3Den%26btnG%3DGoogle%2BSearch

    Peter> I suppose an alternative would be to convert the times to
    Peter> say seconds, create a plot and manually add the actual
    Peter> date/time label (text). This would be tedious but doable.

Currently this is the best way.  It would not be too much work to
write a function called plotdate which took, for example, python2.3
datetime instances as the x arg, arbitrary y values, and a datetime
format string as an optional arg, and did all the conversions for you,
set the tick labels, etc...  If you want to do it, I think it would be
a nice addition to matplotlib.  

JDH

Hi everyone:

I am interested in plotting data in the form:

2004/01/29 15:29:17.161621101   5
2004/01/29 15:29:18.161621061   6
2004/01/29 15:29:19.161621021   7
2004/01/29 15:29:20.161620981   8
2004/01/29 15:29:21.161620941   9
2004/01/29 15:29:22.161620901   0
2004/01/29 15:29:23.161620861   1
...

In some cases I will have data every second, other times every minute or 
hour.. and so on..

Ideally I would like the x-axis of my plots to show the time/dates as 
the units. Is this currently possible with matplotlib? So in other words 
could I call on:

plot(date_in_some_format, data)

I suppose an alternative would be to convert the times to say seconds, 
create a plot and manually add the actual date/time label (text). This 
would be tedious but doable.

Thanks.

-- 
Peter Groszkowski         Gemini Observatory
Tel: +1 808 974-2509      670 N. A'ohoku Place
Fax: +1 808 935-9235      Hilo, Hawai'i 96720, USA

I've checked my path several times and all seems to be as it should be.  =
However when I try to use pygtk and the latest gtk runtime and =
matplotlib under Win98, I get the following error:

>>> import pygtk
>>> pygtk.require('2.0')
>>> import gtk
>>> from matplotlib.matlab import *

Traceback (most recent call last):
  File "<pyshell#6>", line 1, in -toplevel-
    from matplotlib.matlab import *
  File "D:\PROGRA~1\PYTHON23\Lib\site-packages\matplotlib\matlab.py", =
line 124, in -toplevel-
    from backends import new_figure_manager, error_msg, \
  File =
"D:\PROGRA~1\PYTHON23\Lib\site-packages\matplotlib\backends\__init__.py",=
 line 12, in -toplevel-
    from backend_gtk import  \
  File =
"D:\PROGRA~1\PYTHON23\Lib\site-packages\matplotlib\backends\backend_gtk.p=
y", line 15, in -toplevel-
    from gtk import gdk
ImportError: cannot import name gdk

I'm probably overlooking something obvious but would be grateful for =
suggestions.

BTW, if I do the pygtk test (  >>>import pygtk >>> pygtk.require('2.0')
  >>> import gtk  ) once, I get an error message.  If I do it again =
without changing anything, I get no error message.

Dave

>>>>> "matthew" == matthew arnison <ma...@ca...> writes:

    matthew> I haven't been using the pan and zoom stuff very much,
    matthew> the issues I described this week are all from initial
    matthew> plots. If I had been doing more zooming then I would have
    matthew> noticed your very good point about jumping ticks being
    matthew> distracting. It's fiddly stuff, but getting it right
    matthew> helps a lot when interpreting results from plots!

OK, I think my approach will be to optimize the number of ticks, tick
locations and view limits on the initial plot and then fix num ticks
for interactive mode (pan/zoom).  Ie, the initial guess should be
good, but with interaction, you're on your own.  I'll send you some
code when I get this figured out.

    >> Ie, ignore the end point is the default behavior of python.

    matthew> I guessed as much. But I think in this case the python
    matthew> behaviour needs to be over-ridden. Python range logic is
    matthew> generally about integers, arange stretches it, and using
    matthew> this [) style range for plots over-stretches the
    matthew> principle beyond usefulness. There are also a couple of
    matthew> contradictions in matplotlib's behaviour:

Yes, but I can defend myself!  From the first line of the homepage 

  matplotlib is a pure python 2D plotting library with a Matlab
  syntax

Ie, matplotlib does contain inherent contradictions because it is both
matlab-like and python-like.  matlab has FORTRAN style indexing
(starts with 1) and python has C style indexing (starts with 0).  So
the first matplotlib figure starts with figure(1).  When using the
matlab interface matplotlib.matlab, I strive for matlab compatibility.
Thus, when you set the axis limits with

  axis([0, 2, -1, -1])

or 

  set(gca(), 'xlim', [0, 2])

I do it like matlab does, ie, endpoints inclusive.  

However, I plead innocence in the case of

  t = arange(0.0, 2.0, 0.1)
  s = sin(2*pi*t)
  plot(t, s)

If the arrays t and s passed to the plot function do not have the
point at t=2.0 defined, "plot" can't guess them.  I plot the points
you give me.  Note that matplotlib *does* provide the matlab function
"linspace", which returns an evenly sampled array, endpoints included.
So if you want matlab-like behavior you should use matlab-like array
functions (linspace) rather Numeric python array functions (arange) to
define your arrays.

  t = linspace(0.0, 2.0, 20)
  s = sin(2*pi*t)
  plot(t, s)

In a nutshell, with the matlab interface I try to be consistent with
matlab, but there are inconsistencies which arise by virtue of the
fact that it's natural to use python functions (thank god!, that's why
we're all here).

I definitely appreciate the criticism, so feel free to keep at it.

JDH

On Tue, 3 Feb 2004, John Hunter wrote:

> So enough excuses!  I agree that the current implementation is
> suboptimal and will give it some more thought.  Out of curiosity: do
> the undersirable tick locs appear more frequently for you on an
> initial plot or after interacting with the plot.

I haven't been using the pan and zoom stuff very much, the issues I 
described this week are all from initial plots. If I had been doing more 
zooming then I would have noticed your very good point about jumping 
ticks being distracting. It's fiddly stuff, but getting it right helps a 
lot when interpreting results from plots!

> This is a consequence of the way python and Numeric do ranges, and
> doesn't really have anything to do with matplotlib.  eg, the Numeric
> function arange
> 
>   >>> from Numeric import *
>   >>> arange(0.0, 1.0, 0.2)
>   array([ 0. ,  0.2,  0.4,  0.6,  0.8])
> 
>   >>> range(5)
>   [0, 1, 2, 3, 4]
> 
> Ie, ignore the end point is the default behavior of python.

I guessed as much. But I think in this case the python behaviour needs to 
be over-ridden. Python range logic is generally about integers, arange 
stretches it, and using this [) style range for plots over-stretches the 
principle beyond usefulness. There are also a couple of contradictions in 
matplotlib's behaviour:

* the axis and tick ranges are inclusive, but the data point range is 
exclusive of the higher end point only

* the auto data range (axis()) is inclusive, but setting it manually is 
exclusive of the higher end point only

Some of this may be alleviated by chooing better tick points. But I think 
it would also be helpful to make whichever behaviour is chosen more 
consistent.

Cheers,
Matthew.

>>>>> "Jean-Baptiste" =3D=3D Jean-Baptiste Cazier <Jean-Baptiste.cazier@d=
ecode.is> writes:

    Jean-Baptiste> S=E6ll !  I am trying to plot very small number for
    Jean-Baptiste> the Y-axis on semilogy but they do not appear at
    Jean-Baptiste> all unless one of the value is higher Moreover the
    Jean-Baptiste> labels on the Y axis become 0 below 0.001

    >>> semilogy([1.0, 2.3, 3.3],[9.4e-05, 9.4e-05, 9.4e-05]) <-- does
    >>> not work
    Jean-Baptiste> [<matplotlib.lines.Line2D instance at 0x935255c>]
    >>> semilogy([1.0, 2.3, 3.3],[9.4e-04, 9.4e-05, 9.4e-05]) <---
    >>> work
    Jean-Baptiste> [<matplotlib.lines.Line2D instance at 0x940e964>]

S=E6ll Jean!

Thanks for this example.  The relevant code which handles autoscaling
is in matplotlib.axis.autoscale_view.  I wasn't handling the special
case where min=3Dmax for log scaling (though I do handle it for linear
scaling).

Try this replacement code for axis.py: and the functions decade_down
and decade_up and replace the autoscale_view function.

def decade_down(x):
    'floor x to the nearest lower decade'
    lx =3D math.floor(math.log10(x))
    return 10**lx

def decade_up(x):
    'ceil x to the nearest higher decade'
    lx =3D math.ceil(math.log10(x))
    return 10**lx

class Axis(Artist):

    def autoscale_view(self):
        'Try to choose the view limits intelligently'

        vmin, vmax =3D self.datalim.bounds()
        if self._scale=3D=3D'linear':

            if vmin=3D=3Dvmax:
                vmin-=3D1
                vmax+=3D1
            try:
                (exponent, remainder) =3D divmod(math.log10(vmax - vmin),=
 1)
            except OverflowError:
                print >>sys.stderr, 'Overflow error in autoscale', vmin, =
vmax
                return
            if remainder < 0.5:
                exponent -=3D 1
            scale =3D 10**(-exponent)
            vmin =3D math.floor(scale*vmin)/scale
            vmax =3D math.ceil(scale*vmax)/scale

            self.viewlim.set_bounds(vmin, vmax)
        elif self._scale=3D=3D'log':
            if vmin=3D=3Dvmax:
                vmin =3D decade_down(vmin)
                vmax =3D decade_up(vmax)
                self.viewlim.set_bounds(vmin, vmax)


Let me know how this works for you,
JDH

>>>>> "matthew" == matthew arnison <ma...@ca...> writes:

    matthew> Hi again, I'm having more trouble with matplotlib ticks
    matthew> today. I wrote a little demo script that illustrates some
    matthew> of the problems:

Hi Matthew,

Thanks for sending me these example scripts - it really helps to have
complete examples when working on these problems.  I've made a few
changes to the tickval formatter.  The relevant function is
matplotlib.axis.format_tickval and is pretty simple conceptually.

Try replacing the default format_tickval with this one.  The d
variable gives the max-min distance of the view limits.  I use
different format strings depending on the size of the distance. 

    def format_tickval(self, x):
        'Format the number x as a string'
        d = self.viewlim.interval()
        if self._scale == 'log':
            # only label the decades
            fx = self.transData.func(x)
            isdecade = abs(fx-int(fx))<1e-10
            if self._ticklabelStrings is None and not isdecade: return ''

        #if the number is not too big and it's an int, format it as an
        #int
        if abs(x)<1e4 and x==int(x): return '%d' % x

        # if the value is just a fraction off an int, use the int
        if abs(x-int(x))<0.0001*d: return '%d' % int(x)

        # use exponential formatting for really big or small numbers,
        # else use float
        if d < 1e-2   : fmt = '%1.2e'
        elif d < 1e-1 : fmt = '%1.2f'
        elif d > 1e5  : fmt = '%1.3e'
        elif d > 10   : fmt = '%1.1f'
        elif d > 1    : fmt = '%1.2f'
        else          : fmt = '%1.3f'
        s =  fmt % x
        #print d, fmt, x, s
        # strip trailing zeros, remove '+', and handle exponential formatting
        m = self._zerorgx.match(s)
        if m:
            s = m.group(1)
            if m.group(2) is not None: s += m.group(2)
        s = s.replace('+', '')
        return s

And then feed it some more of your sadistic examples <wink>.  If you
don't like what you see, try tweaking the formats and the distance
values until you get sensible results.  Or feel free to provide more
comments and send more examples.

JDH