How to use binary files. Overwriting specific bytes

Question

I'm writing a program in python, and would like to be able to write to specific bytes in a binary file. I tried to do this in the shell with a small binary file containing the numbers 0 through 15, but I can't figure out how to do so. Below is the code I just entered into the shell with comments to demonstrate what I am trying to do:

>>> File=open("TEST","wb") # Opens the file for writing.
>>> File.write(bytes(range(16))) # Writes the numbers 0 through 15 to the file.
16
>>> File.close() # Closes the file.
>>> File=open("TEST","rb") # Opens the file for reading, so that we can test that its contents are correct.
>>> tuple(File.read()) # Expected output: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
>>> File.close() # Closes the file.
>>> File=open("TEST","wb") # Opens the file for writing.
>>> File.seek(3) # Moves the pointer to position 3. (Fourth byte.)
3
>>> File.write(b"\x01") # Writes 1 to the file in its current position.
1
>>> File.close() # Closes the file.
>>> File=open("TEST","rb") # Opens the file for reading, so that we can test that its contents are correct.
>>> tuple(File.read()) # Expected output: (0, 1, 2, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
(0, 0, 0, 1)
>>> File.close()
>>> File=open("TEST","wb") # I will try again using apend mode to overwrite.
>>> File.write(bytes(range(16)))
16
>>> File.close()
>>> File=open("TEST","ab") # Append mode.
>>> File.seek(3)
3
>>> File.write(b"\x01")
1
>>> File.close()
>>> File=open("TEST","rb")
>>> tuple(File.read()) # Expected output: (0, 1, 2, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1)
>>> File.close()

My desired output is as shown, but "wb" seems to erase all the data in the file, while "ab" can't seek backwards.

How would I achieve my desired output without rewriting the whole file?

Martijn Pieters · Accepted Answer · 2018-10-30 10:34:51Z

When you open a file for writing with w, the file is truncated, all contents removed. You need to open the file for reading and writing with r+ instead. From the open() function documentation:

'w' open for writing, truncating the file first

and

For binary read-write access, the mode 'w+b' opens and truncates the file to 0 bytes. 'r+b' opens the file without truncation.

Because the file was truncated first, seeking to position 3 then writing \x01 has the first few bytes filled in with \x00 for you.

Opening a file in append mode usually restricts access to the new portion of the file only, so anything past the first 16 bytes. Again, from the documentation:

Other common values are [...] and 'a' for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position).

(bold emphasis in quoted sections mine). This is why your \x01 byte ends up right at the end in spite of the File.seek(3) call.

r does not truncate a file and gives you full range of the contents with seek(); r+ adds write access to that mode. Demo with 'r+b':

>>> with open('demo.bin', 'wb') as f:
...     f.write(bytes(range(16)))
...
16
>>> with open('demo.bin', 'rb') as f:
...     print(*f.read())
...
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
>>> with open('demo.bin', 'r+b') as f:  # read and write mode
...     f.seek(3)
...     f.write(b'\x01')
...
3
1
>>> with open('demo.bin', 'rb') as f:
...     print(*f.read())
...
0 1 2 1 4 5 6 7 8 9 10 11 12 13 14 15

Programmer S · Accepted Answer · 2018-10-30 10:34:24Z

0

The solution is another mode: "r+b". (as shown by other answers.)

Here is the solution in the shell from where the file left off:

>>> File=open("TEST","r+b") # Opens file for reading and writing.
>>> File.seek(3)
3
>>> File.write(b"\x01")
1
>>> File.close()
>>> File=open("TEST","rb")
>>> tuple(File.read()) # Expected output: (0, 1, 2, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1)
(0, 1, 2, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1)
>>> File.close()

answered Oct 30, 2018 at 10:34

Programmer S

4897 silver badges25 bronze badges

Comments

Programmer S · Accepted Answer · 2018-10-30 11:39:44Z

0

If I remember properly, you have to open the file in "Append mode" or it will simply erase everything and start from scratch, and then when you use seek(3) you just create those 3 0's and then you write the 1. I'll investigate further on how to write directly to a position but you may have to read the whole file, modify, write the whole file again.

You can actually read about this behaviour in the documentation:

'w' for only writing (an existing file with the same name will be erased)

edited Oct 30, 2018 at 11:39

Programmer S

4897 silver badges25 bronze badges

answered Oct 30, 2018 at 10:24

epsilonmajorquezero

5991 gold badge5 silver badges20 bronze badges

3 Comments

Programmer S Over a year ago

I have shown in my question that append mode doesn't work in this situation.

Yann Vernier Over a year ago

Or rather, that append mode does work - it enforces that writes go to the end. It is specifically not meant for modifying existing contents.

epsilonmajorquezero Over a year ago

Yeah, maybe I wrote a bit too fast since I had to go toa meeting but wasn't intending that append mode was your solution here :) Martijin did a much better work at explaining how to solve your problem.

Collectives™ on Stack Overflow

How to use binary files. Overwriting specific bytes

3 Answers 3

Comments

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related