Most efficient way of loading formatted binary files in Python

Question

I have binary files no larger than 20Mb in size that have a header section and then a data section containing sequences of uchars. I have Numpy, SciPy, etc. and each library has different ways of loading in the data. Any suggestions for the most efficient methods I should use?

rmmh · Accepted Answer · 2009-03-31 22:05:12Z

8

Use the struct module, or possibly a custom module written in C if performance is critical.

answered Mar 31, 2009 at 22:05

rmmh

7,12529 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Theran · Accepted Answer · 2009-04-01 03:32:45Z

4

struct should work for the header section, while numpy's memmap would be efficient for the data section if you are going to manipulate it in numpy anyways. There's no need to stress out about being inconsistent here. Both methods are compatible, just use the right tool for each job.

answered Apr 1, 2009 at 3:32

Theran

3,87623 silver badges31 bronze badges

Comments

Toni Ruža · Accepted Answer · 2009-04-01 06:12:57Z

1

bdec seems promising.

answered Apr 1, 2009 at 6:12

Toni Ruža

7,5132 gold badges32 silver badges31 bronze badges

Comments

Eli Bendersky · Accepted Answer · 2009-04-01 00:27:11Z

0

I found that array.fromfile is the fastest methods for homogeneous data.

answered Apr 1, 2009 at 0:27

Eli Bendersky

276k92 gold badges372 silver badges427 bronze badges

Collectives™ on Stack Overflow

Most efficient way of loading formatted binary files in Python

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related