Asynchronous exception handling in Python

Question

I've the following code using asyncio and aiohttp to make asynchronous HTTP requests.

import sys
import asyncio
import aiohttp

@asyncio.coroutine
def get(url):
    try:
        print('GET %s' % url)
        resp = yield from aiohttp.request('GET', url)
    except Exception as e:
        raise Exception("%s has error '%s'" % (url, e))
    else:
        if resp.status >= 400:
            raise Exception("%s has error '%s: %s'" % (url, resp.status, resp.reason))

    return (yield from resp.text())

@asyncio.coroutine
def fill_data(run):
    url = 'http://www.google.com/%s' % run['name']
    run['data'] = yield from get(url)

def get_runs():
    runs = [ {'name': 'one'}, {'name': 'two'} ]
    loop = asyncio.get_event_loop()
    task = asyncio.wait([fill_data(r) for r in runs])
    loop.run_until_complete(task)   
    return runs

try:
    get_runs()
except Exception as e:
    print(repr(e))
    sys.exit(1)

For some reason, exceptions raised inside the get function are not caught:

Future/Task exception was never retrieved
Traceback (most recent call last):
  File "site-packages/asyncio/tasks.py", line 236, in _step
    result = coro.send(value)
  File "mwe.py", line 25, in fill_data
    run['data'] = yield from get(url)
  File "mwe.py", line 17, in get
    raise Exception("%s has error '%s: %s'" % (url, resp.status, resp.reason))
Exception: http://www.google.com/two has error '404: Not Found'

So, what is correct way to handle exceptions raised by coroutines?

dano · Accepted Answer · 2022-04-15 13:47:07Z

94

asyncio.wait doesn't actually consume the Futures passed to it, it just waits for them to complete, and then returns the Future objects:

coroutine asyncio.wait(futures, *, loop=None, timeout=None, return_when=ALL_COMPLETED)

Wait for the Futures and coroutine objects given by the sequence futures to complete. Coroutines will be wrapped in Tasks. Returns two sets of Future: (done, pending).

Until you actually yield from/await the items in the done list, they'll remain unconsumed. Since your program exits without consuming the futures, you see the "exception was never retrieved" messages.

For your use-case, it probably makes more sense to use asyncio.gather, which will actually consume each Future, and then return a single Future that aggregates all their results (or raises the first Exception thrown by a future in the input list).

def get_runs():
    runs = [ {'name': 'one'}, {'name': 'two'} ]
    loop = asyncio.get_event_loop()
    tasks = asyncio.gather(*[fill_data(r) for r in runs])
    loop.run_until_complete(tasks)
    return runs

Output:

GET http://www.google.com/two
GET http://www.google.com/one
Exception("http://www.google.com/one has error '404: Not Found'",)

Note that asyncio.gather actually lets you customize its behavior when one of the futures raises an exception; the default behavior is to raise the first exception it hits, but it can also just return each exception object in the output list:

asyncio.gather(*coros_or_futures, loop=None, return_exceptions=False)

Return a future aggregating results from the given coroutine objects or futures.

All futures must share the same event loop. If all the tasks are done successfully, the returned future’s result is the list of results (in the order of the original sequence, not necessarily the order of results arrival). If return_exceptions is True, exceptions in the tasks are treated the same as successful results, and gathered in the result list; otherwise, the first raised exception will be immediately propagated to the returned future.

edited Apr 15, 2022 at 13:47

answered May 21, 2015 at 3:39

dano

95.6k21 gold badges234 silver badges231 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Yury Bayda Over a year ago

Thanks for explanation, the documentation was not perfectly clear about exception handling

z0r Over a year ago

So how would one do it with wait? Is it something like yield from asyncio.wait(...)? Should await asyncio.wait(...) work too?

doak Over a year ago

By the way, the same issue arises if one of the couroutine is never actually started. Happens for me with two coroutines opening some socket (manually) and try to await <loop>.sock_recv(<socket>, <size>). If the socket is not switched to non-blocking (with <socket>.setblocking()), the second coroutine is not started and a KeyboardInterrupt results in "Task exception was never retrieved".

Rugnar Over a year ago

Inside async function one can't use yield from to resolve future or task. One should await instead. This was not clear for me.

pouya Over a year ago

thanks for providing the return_exceptions hint. Doing hundreds of head request I don't want an exception is thrown cause one of them is timeouted.

Grr · Accepted Answer · 2018-10-26 16:41:27Z

5

To debug or "handle" exceptions in callback:

Coroutine which return some result or raise exceptions:

@asyncio.coroutine
def async_something_entry_point(self):
    try:
        return self.real_stuff_which_throw_exceptions()
    except:
        raise Exception(some_identifier_here + ' ' + traceback.format_exc())

And callback:

def callback(self, future: asyncio.Future):
    exc = future.exception()
    if exc:
        # Handle wonderful empty TimeoutError exception
        if type(exc) == TimeoutError:
            self.logger('<Some id here> callback exception TimeoutError')
        else:
            self.logger("<Some id here> callback exception " + str(exc))

    # store your result where you want
    self.result.append(
        future.result()
    )

edited Oct 26, 2018 at 16:41

Grr

16.2k7 gold badges72 silver badges91 bronze badges

answered Mar 4, 2016 at 6:27

Oleg Neumyvakin

10.4k5 gold badges63 silver badges67 bronze badges

Collectives™ on Stack Overflow

Asynchronous exception handling in Python

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related