File Stream - ValueError: embedded null byte

Question

I'm trying to download a .png image via HTTP requests and upload it via HTTP to another location. My objective is to avoid saving the file on the disk so it's processed in-memory.

I have the code below:

Download the file and convert it into a byte array:

resp = requests.get(
    'http://www.personal.psu.edu/crd5112/photos/PNG%20Example.png',
    stream=True)

img = BytesIO(resp.content)

Upload the file to a remote HTTP repository

data=open(img.getvalue()).read()

r = requests.post(url=url, data=data, headers=headers, auth=HTTPBasicAuth('user', 'user'))

I'm getting a ValueError exception "embedded null byte" when reading the byte array.

If I save the file onto the disk and load it as below, then there is no error:

with open('file.png', 'wb') as pic:
  pic.write(img.getvalue())

Any advice on how I could achieve it without saving the file on the disk ?

Shrout1 · Accepted Answer · 2021-06-16 19:19:53Z

I believe that the embedded null byte error is caused by a filename input requirement of a library that is supporting whatever operation is being executed in your code. By using a BytesIO object this presents itself to that library "as if" it is wrapped inside a file.

Here is sample code that I used when trying to address this same issue with a tar file. This code should be able to satisfy most file input requirements for various other libraries.

The key that I found here was using the BytesIO object around the remote_file.content being passed into the tarfile.open as a file object. Other techniques I attempted did not work.

from io import BytesIO
import requests
import tarfile

remote_file=requests.get ('https://download.site.com/files/file.tar.gz')

#Extract tarball contents to memory
tar=tarfile.open(fileobj=BytesIO(remote_file.content))
#Optionally print all folders / files within the tarball
print(tar.getnames())
tar.extractall('/home/users/Documents/target_directory/')

This eliminated the ValueError: embedded null byte and expected str, bytes or os.PathLike object, not _io.BytesIO errors that I was experiencing with other methods.

AmilaMGunawardana · Accepted Answer · 2019-09-15 04:57:23Z

3

Yes, you can do this without saving to the disk. Before that, the error occurred in line

data=open(img.getvalue()).read()

Since the inbuild string operation is not good with different encodings this error occured. use the pillow library to meddle with image realated situations

from io import BytesIO
from PIL import Image    
img = BytesIO(resp.content)
-#data=open(img).read()
+data = Image.open(img)

this will give you a following object type

<class 'PIL.PngImagePlugin.PngImageFile'>

you can use this data variable as your data in the upload post request

edited Sep 15, 2019 at 4:57

answered Sep 15, 2019 at 3:05

AmilaMGunawardana

1,8752 gold badges16 silver badges34 bronze badges

2 Comments

rafi Over a year ago

How would this be solved if the file was a PDF instead of an image?

AmilaMGunawardana Over a year ago

@rafi you can use a method like this pdf_file = StringIO(r.content) existing_pdf = PdfFileReader(pdf_file) for PdfFileReader install PyPDF2 by pip install PyPDF2

Luca Brasi · Accepted Answer · 2019-09-15 04:55:27Z

1

@AmilaMGunawardana Thanks for the pointer.

I just had to save the image into a separate byte stream to get it uploaded properly:

img = BytesIO(resp.content)

data = Image.open(img, 'r')

buf = BytesIO()

data.save(buf, 'PNG')

r = requests.post(url=url, data=buf.getvalue(), headers=headers, auth=HTTPBasicAuth('user', 'user'))

answered Sep 15, 2019 at 4:55

Luca Brasi

7312 gold badges13 silver badges28 bronze badges

1 Comment

AmilaMGunawardana Over a year ago

Thats good but if you look into memory management use img variable to store the empty byte stream it will help towards speed and memory.

Collectives™ on Stack Overflow

File Stream - ValueError: embedded null byte

3 Answers 3

Comments

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related