uploading multiple files UploadFiles FastAPI [duplicate]

Question

Example

Here's my code:

from typing import  List
from fastapi import FastAPI, File, UploadFile
import asyncio
import concurrent.futures

app = FastAPI()
@app.post("/send_images")
async def update_item(
    files: List[UploadFile] = File(...),
):
    return {"res": len(files)}

And I send requests to this server with (these specific urls are just for example here):

import requests
import os 
import json
import numpy as np
import time
import io
import httpx
import asyncio
from datetime import datetime
import pandas as pd

from random import shuffle
import pandas as pd
import urllib
import io
from PIL import Image
from matplotlib import pyplot as plt
import concurrent.futures

urls = ['https://sun9-63.userapi.com/c638920/v638920705/1a54d/xSREwpakJD4.jpg',
 'https://sun9-28.userapi.com/c854024/v854024084/1160d8/LDMVHYgguAw.jpg',
 'https://sun9-54.userapi.com/c854220/v854220084/111f66/LdcbEpPR6tg.jpg',
 'https://sun9-40.userapi.com/c841420/v841420283/4c8bb/Mii6GSCrmpo.jpg',
 'https://sun6-16.userapi.com/CPQpllJ0KtaArvQKkPsHTZDCupqjRJ_8l07ejA/iyg2hRR_kM4.jpg',
 'https://sun9-1.userapi.com/c638920/v638920705/1a53b/SMta6Bv-k7s.jpg',
 'https://sun9-36.userapi.com/c857332/v857332580/56ad/rJCGKFw03FQ.jpg',
 'https://sun6-14.userapi.com/iPsfmW0ibE8RsMh0k2lUFdRxHZ4Q41yctB7L3A/ajJHY3WN6Xg.jpg',
 'https://sun9-28.userapi.com/c854324/v854324383/1c1dc3/UuFigBF7WDI.jpg',
 'https://sun6-16.userapi.com/UVXVAT-tYudG5_24FMaBWTB9vyW8daSrO2WPFQ/RMjv7JZvowA.jpg']

os.environ['NO_PROXY'] = '127.0.0.1'

async def request_get_4(list_urls):
    async with httpx.AsyncClient() as client:
        r = httpx.post("http://127.0.0.1:8001/send_images", files={f'num_{ind}': el for ind, el in enumerate(list_urls)})
        print(r.text)
        return r

async def request_get_3(url):
    async with httpx.AsyncClient() as client:
        return await client.get(url)
    
from collections import defaultdict

async def main():
    start = datetime.now()
    tasks = [asyncio.create_task(request_get_3(url)) for url in urls[0:10]]
    result = await asyncio.gather(*tasks)
    
    data_to_send = []
    for ind, resp in enumerate(result):
        if resp.status_code == 200:
            image_bytes = io.BytesIO(resp.content)
            image_bytes.seek(0)
            data_to_send.append(image_bytes)
        
    end = datetime.now()
    print(result)
    print(len(data_to_send))

    batch_size = 2
    batch_num = len(data_to_send) // batch_size
    tasks = [asyncio.create_task(request_get_4(data_to_send[i * batch_size: (i+1) * batch_size])) for i in range(batch_num)]
    result = await asyncio.gather(*tasks)
    
    left_data = data_to_send[batch_size*(batch_num):]
    print(len(left_data))
    print(result)

asyncio.run(main())

I am trying to load image which are contained in urls, then form batches of them and send them to FastAPI server. But it doesn't work. I get the following error:

{"detail":[{"loc":["body","files"],"msg":"field required","type":"value_error.missing"}]}

How can I fix my problem and be able to send multiple files through httpx to FastAPI?

My guess is that your endpoint expects a list of files, while you are passing a dictionary via the httpx.post request. Try something like files=[f for f in downloaded_files] , which, BTW, does not seem you are downloading them before sending them — lsabi
– lsabi, Commented Jul 28, 2020 at 20:11
and why do you think that I don't download urls? I use httpx.get. — Danil Kononyhin
– Danil Kononyhin, Commented Jul 29, 2020 at 9:53
so, yeah, this is httpx module limitation! right now it can't send more than one file in post request! — Danil Kononyhin
– Danil Kononyhin, Commented Jul 29, 2020 at 11:00
Future readers should have a look at this answer, as well as this answer and this answer — Chris
– Chris, Commented Feb 7, 2024 at 6:51

Danil Kononyhin · Accepted Answer · 2021-04-27 07:13:50Z

3

The problem is that HTTPX 0.13.3 doesn't support multiple file uploading as this arg wants dictionary and dictionary can't have same key values.

We can use this pull request https://github.com/encode/httpx/pull/1032/files to fix this problem (now it can also accept List[Tuple[str, FileTypes]]])!

UPDATE: this problem is solved now!

Update2: Here I show how I use httpx for different requests:

async def request_post_batch(fastapi_url: str, url_content_batch: List[BinaryIO]) -> httpx.Response:
    """
    Send batch to FastAPI server.
    """
    async with httpx.AsyncClient(timeout=httpx.Timeout(100.0)) as client:
        r = await client.post(
            fastapi_url,
            files=[('bytes_image', url_content) for url_content in url_content_batch]
        )
        return r


async def request_post_logs(logstash_url: str, logs: List[Dict]) -> httpx.Response:
    """
    Send logs to logstash
    """
    async with httpx.AsyncClient(timeout=httpx.Timeout(100.0)) as client:
        r = await client.post(
            logstash_url,
            json=logs
        )
        return r

edited Apr 27, 2021 at 7:13

answered Jul 29, 2020 at 11:14

Danil Kononyhin

3161 gold badge3 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Soerendip Over a year ago

Example, example, example :-) please :-)

Danil Kononyhin Over a year ago

what example do you want?

Ievgen Over a year ago

[('images', ('foo.png', open('foo.png', 'rb'), 'image/png')), ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))]

Ievgen · Accepted Answer · 2022-05-05 20:23:16Z

1

This example will pass multiple files under the files field.

async with httpx.AsyncClient(timeout=httpx.Timeout(100.0)) as client:    
     response = await client.post(f"/api/v1/upload-files",
                files=[("files", ("image.png", b"{}", "image/png")), ("files", ("image2.png", b"{}", "image/png"))],
            )

answered May 5, 2022 at 20:23

Ievgen

4,4837 gold badges81 silver badges130 bronze badges

Collectives™ on Stack Overflow

uploading multiple files UploadFiles FastAPI [duplicate]

Example

2 Answers 2

3 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Example

2 Answers 2

3 Comments

Comments

Linked

Related