Fetching multiple urls with aiohttp in python

Question

In a previous question, a user suggested the following approach for fetching multiple urls (API calls) with aiohttp:

import asyncio
import aiohttp


url_list = ['https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530396000&before=1530436000', 'https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530436000&before=1530476000']
        
async def fetch(session, url):
    async with session.get(url) as response:
        return await response.json()['data']
        
        
async def fetch_all(session, urls, loop):
    results = await asyncio.gather(*[loop.create_task(fetch(session, url)) for url in urls], return_exceptions= True)
    return results

if __name__=='__main__':
    loop = asyncio.get_event_loop()
    urls = url_list
    with aiohttp.ClientSession(loop=loop) as session:
        htmls = loop.run_until_complete(fetch_all(session, urls, loop))
    print(htmls)

However, this results in only returning Attribute errors:

[AttributeError('__aexit__',), AttributeError('__aexit__',)]

(which I enabled, otherwise it would just break). I really hope there is somebody here, who can help with this, it is still kind of hard to find resources for asyncio etc. The returned data is in JSON format. In the end I would like to put all JSON dicts in a list.

Sam Bull · Accepted Answer · 2024-11-21 12:57:16Z

21

Working example:

import asyncio
import aiohttp
import ssl

url_list = ['https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530396000&before=1530436000',
            'https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530436000&before=1530476000']


async def fetch(session, url):
    async with session.get(url, ssl=ssl.SSLContext()) as response:
        return await response.json()


async def fetch_all(urls):
    async with aiohttp.ClientSession() as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results


if __name__ == '__main__':
    urls = url_list
    htmls = asyncio.run(fetch_all(urls))
    print(htmls)

edited Nov 21, 2024 at 12:57

Sam Bull

3,3741 gold badge20 silver badges23 bronze badges

answered Aug 7, 2018 at 13:29

Yurii Kramarenko

1,0746 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

py_dude Over a year ago

async with aiohhtp.ClientSession(connector=aiohttp.TCPConnector(verify_ssl=False)) no more tears to cry

Yurii Kramarenko Over a year ago

@Jannik looks strange, which version of aiohttp\python you are using?

Jannik Over a year ago

@YuriiKramarenko i am using aiohttp==0.7.2, asyncio 3.4.3 and python 3.6

Jannik Over a year ago

@YuriiKramarenko you were so right. I updated it, now it works! thank you so much. I have no idea, why my version was so outdated

Krissh Over a year ago

Not working for me got below error : RuntimeError: This event loop is already running

|

Collectives™ on Stack Overflow

Fetching multiple urls with aiohttp in python

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related