python file.write(bytes) works but file is missing bytes

Question

I wrote a bittorrent client and successfully downloaded all pieces for all files.

Before and after I write the files (mode='wr'), I print out filename, piece_index, number of bytes written, and the offset into the file where the bytes are written. After all files are written, I close the files.

BUT, when I look on the disk, only the first piece is written. The piece completely writes file 0 and the beginning bytes of file 1. Even though the print statements show all remaining pieces are written to file 1, file 1 does not have them. No error in file.seek, file.write. Here is some output:

-- first piece --    
offset: 0 piece_index: 0
about to write file 0: offset 0 start 0 nbytes 291
Distributed by Mininova.txt
just wrote 291 bytes at offset 0

about to write file 1: offset 0 start 291 nbytes 1048285    
DF self-extracting archive.exe
just wrote 1048285 bytes at offset 0

-- next piece --
offset: 1048285 piece_index: 1
about to write file 1: offset 1048285 start 1048576 nbytes 1048576
DF self-extracting archive.exe
just wrote 1048576 bytes at offset 1048285

-- next piece --
offset: 2096861 piece_index: 2 file_index: 1
about to write file 1: offset 2096861 start 2097152 nbytes 1048576
DF self-extracting archive.exe
just wrote 1048576 bytes at offset 2096861

The code:

def _write(self, fd, offset, start, num_bytes, row):
    print(fd.name[-30:])
    fd.seek(offset)  
    fd.write(self.buffer[row][start:start+num_bytes].tobytes())
    fd.seek(0)
    print('just wrote {} bytes at offset {}\n'.format(num_bytes, offset))

I have spent 2 days researching this. If f.write works, why doesn't file on filesystem have all the bytes? file is closed only after all the writes. — LeslieK
– LeslieK, Commented Dec 4, 2015 at 18:05
If a list doesn't contain all of the slice you ask for, you get a truncated result. In particular self.buffer[row][start:start+num_bytes] isn't guaranteed to return num_bytes worth of data. — Mark Ransom
– Mark Ransom, Commented Dec 4, 2015 at 18:38
Thanks @MarkRansom. Didn't realize that out-of-range slices are silent errors. The buffer returned 0 bytes since start was beyond the end of the row. I found the bug immediately after I printed out what I actually wrote. — LeslieK
– LeslieK, Commented Dec 5, 2015 at 0:01

Mike Müller · Accepted Answer · 2015-12-04 18:51:39Z

3

This should do:

def _write(self, fd, offset, start, num_bytes, row):
    print(fd.name[-30:])
    fd.seek(offset)
    bytes_ = self.buffer[row][start:start+num_bytes].tobytes()
    fd.write(bytes_)
    fd.flush()
    fd.seek(0)
    print('just wrote {} bytes at offset {}\n'.format(len(bytes_), offset))

Calculate the bytes you actually write: len(bytes_)
Flush the file after writing. You could also set buffering=0 when opening the file.

answered Dec 4, 2015 at 18:51

Mike Müller

86k21 gold badges174 silver badges165 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

LeslieK Over a year ago

Your answer led me to the bug. Thank you!!! The bug is the value of start. start should have been 'start modulo n'. Therefore, the start value was beyond the size of buffer[row]. As @MarkRansom said, a slice is silent when it is out of range. buffer[row] is an array. Since start was out of range, I was writing 0 bytes. I needed fresh eyes. I will never forget this lesson: print out what I actually wrote. Not what I thought I wrote.

LeslieK Over a year ago

By the way, the first down vote came instantly after I hit return on my question. Instantly. Hard for me to believe that a person could read the post and down vote it so fast. The second down vote came a few secs later.

Mike Müller Over a year ago

@LeslieK Glad to hear that this solved your problem. Often the solution is rather simple but the author of the program is too deep in the internals and does not see the problem. Don't let you be discouraged by the down votes. Sometimes they are really not justified. Seems like there are people who just down vote without actually reading the question. It should take at least one or two minutes to read and understand this question. Thinking through and figuring out if it is a "bad" question should take even longer.

Collectives™ on Stack Overflow

python file.write(bytes) works but file is missing bytes

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related