Fetching multiple urls with aiohttp from a loop/generator

Question

All examples I saw for fetching multiple urls with aiohttp suggest to to the following:

async def fetch(session, url):
    async with session.get(url, ssl=ssl.SSLContext()) as response:
        return await response.json()


async def fetch_all(urls, loop):
    async with aiohttp.ClientSession(loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    urls = url_list
    htmls = loop.run_until_complete(fetch_all(urls, loop))
    print(htmls)

(https://stackoverflow.com/a/51728016/294103)

In practice, however, I typically have a generator (can be also async) returning domain objects from db, one attribute of which is url, but I also need access to other attributes later in the loop:

async for domain_obj in generator:
    url = domain_obj.url
    response = xxx # need to fetch single url here in async manner
    # do something with response

Of course I can batch collect domain_objs in a list, and fetch all of them like in example, but this doesn't feel right.

As an aside, do not use the deprecated asyncio.get_event_loop method. — Booboo
– Booboo, Commented Nov 21, 2024 at 13:51

Sam Bull · Accepted Answer · 2024-11-21 12:53:28Z

1

My first thought is that you probably want to use TaskGroup.

Something like:

    async with aiohttp.ClientSession() as session:
        tasks = []
        async with asyncio.TaskGroup() as tg:
            async for domain in generator:
                tasks.append(tg.create_task(fetch(session, domain)))
    return [t.result() for t in tasks]

See the linked reference for details on exception handling etc.

This will allow the tasks to start executing while waiting for more results from the generator, rather than needing to exhaust the generator before any tasks are started.

answered Nov 21, 2024 at 12:53

Sam Bull

3,3741 gold badge20 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

dragoon Over a year ago

Thank you, yes, this would be the same as for domain in generator: t = asyncio.create_task(adapter.fetch(domain) tasks.append((t, domain)) But what is my generator could be large/unknown? That means I have to manually do batching? Seems like a lot of code for such a common task.

dragoon Over a year ago

In the end I did just that with the task group, as I was able to separate my coroutines completely, such that they don't need to return anything.

Sam Bull Over a year ago

To limit the number of simultaneous requests, you can adjust the limit in the TCPConnector, or use an asyncio.Semaphore in the tasks.

Collectives™ on Stack Overflow

Fetching multiple urls with aiohttp from a loop/generator

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related