65

After perusing many docs on AsyncIO and articles I still could not find an answer to this : Run a function asynchronously (without using a thread) and also ensure the function calling this async function continues its execution.

Pseudo - code :

async def functionAsync(p):
    #...
    #perform intensive calculations
    #...
    print ("Async loop done")

def functionNormal():
    p = ""
    functionAsync(p)
    return ("Main loop ended")

 print ("Start Code")
 print functionNormal()

Expected Output :

Start code
Main loop ended
Async loop done

Searched examples where loop.run_until_complete is used, but that will not return the print value of functionNormal() as it is blocking in nature.

6
  • If it should run "without using a thread", how do you expect this to work? Or, are you saying that it's ok for the implementation to use threads under the hood, but you don't want to create a thread explicitly? Commented Apr 12, 2019 at 9:21
  • Yes that is what I mean. Explicitly I do not want to create a thread. If it does it under the hood it's fine (which acc to my knowledge it might not be, as per my reading concurrency does not always mean a new thread.) Commented Apr 12, 2019 at 9:55
  • Concurrency doesn't always mean a new thread if you use coroutines (async def) for all your code. But your requirement is to have a sync function executed concurrently with async code, and that will certainly require multiple threads or fibers. Commented Apr 12, 2019 at 11:05
  • 2
    Async code can be started in a new event loop too if i'm correct. loop = asyncio.new_event_loop() ..And yes you are right, the sync code should continue running and go to the next line of code as shown in the example. Commented Apr 12, 2019 at 11:18
  • 3
    new_event_loop only allocates an event loop. To actually run async code in it, you must use run_until_complete or run_forever, which blocks the current thread - so you need an additional thread to run sync code concurrently with async code. It will never work without threads. Commented Apr 12, 2019 at 11:43

9 Answers 9

39

Just use asyncio.run() inside a synchronous function.

def syncfunc():
    async def asyncfunc():
        await some_async_work()
    asyncio.run(asyncfunc())

syncfunc()

Note that asyncio.run() cannot be called when another asyncio event loop is running in the same thread.

Sign up to request clarification or add additional context in comments.

4 Comments

How to do while another loop is running?
This is blocking. Can you please provide an example the reproduces the required output?
To clarify the question from @villamejia, this won't work in the context of a running event loop. Notably, running syncfunc() (or any function calling it) from a Jupyter notebook will raise "RuntimeError: asyncio.run() cannot be called from a running event loop".
make sure to import asyncio before
28

The following should work in all relevant scenarios

import asyncio
import threading
from concurrent.futures import ThreadPoolExecutor
from typing import Any, Coroutine, TypeVar

__all__ = [
    "run_coroutine_sync",
]

T = TypeVar("T")


def run_coroutine_sync(coroutine: Coroutine[Any, Any, T], timeout: float = 30) -> T:
    def run_in_new_loop():
        new_loop = asyncio.new_event_loop()
        asyncio.set_event_loop(new_loop)
        try:
            return new_loop.run_until_complete(coroutine)
        finally:
            new_loop.close()

    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        return asyncio.run(coroutine)

    if threading.current_thread() is threading.main_thread():
        if not loop.is_running():
            return loop.run_until_complete(coroutine)
        else:
            with ThreadPoolExecutor() as pool:
                future = pool.submit(run_in_new_loop)
                return future.result(timeout=timeout)
    else:
        return asyncio.run_coroutine_threadsafe(coroutine, loop).result()

4 Comments

This made my day! I've tried a similar approaches but always resulting in one error or another. Even though I think it does not solve the problem explain in the original question, I believe it solves mine :) Thanks!
Glad to hear that! Keep in mind that it's not super well tested so please let me know if you run into any issues
what happens if you call asyncio.run_coroutine_threadsafe in the main thread (if you already have a loop)?
I was able to break this one as well with "Event Loop already closed"...
24

Here's a helper function to synchronize an async operation:

import asyncio

async def sleep_return(s):
    await asyncio.sleep(s)
    return 420

def async_to_sync(awaitable):
    loop = asyncio.get_event_loop()
    return loop.run_until_complete(awaitable)

async_result = async_to_sync(sleep_return(.69))
print(f"Result from async call: { async_result }")

# Output:
# Result from async call: 420

You can inline most of the calls to the asyncio api to simplify it even more, but this is a little bit cleaner, imho

4 Comments

"RuntimeError: This event loop is already running"
In case any one tries to use this example, you wouldn't want to call time.sleep from an async function because it would block the event loop. Use asyncio.sleep instead docs.python.org/3/library/asyncio-task.html#asyncio.sleep
This one saved my day. Neat!
checkout my answer for an implementation that works inside another loop
17

asyncio can't run arbitrary code "in background" without using threads. As user4815162342 noted, in asyncio you run event loop that blocks main thread and manages execution of coroutines.

If you want to use asyncio and take advantage of using it, you should rewrite all your functions that uses coroutines to be coroutines either up to main function - entry point of your program. This main coroutine is usually passed to run_until_complete. This little post reveals this topic in more detail.


Since you're interested in Flask, take a look Quart: it's a web framework that tries to implement Flask API (as much as it's possible) in terms of asyncio. Reason this project exists is because pure Flask isn't compatible with asyncio. Quart is written to be compatible.

If you want to stay with pure Flask, but have asynchronous stuff, take a look at gevent. Through monkey-patching it can make your code asynchronous. Although this solution has its own problems (which why asyncio was created).

2 Comments

asyncio.ensure_future(my_coro(param1, param2)) works when inside a sync function (like a callback) with a running event loop.
Supplement to comment of @ScottP.: if you have a coroutine without arguments, async def asyncfunc():, do not forget parenthesis in the call: asyncio.ensure_future(asyncfunc) will trigger "TypeError: An asyncio.Future, a coroutine or an awaitable is required", see stackoverflow.com/q/59481105 - instead use asyncio.ensure_future(asyncfunc())
16

Maybe it's a bit late, but I'm running into a similar situation and I solved it in Flask by using run_in_executor:

def work(p):
    # intensive work being done in the background


def endpoint():
    p = ""
    loop = asyncio.get_event_loop()
    loop.run_in_executor(None, work, p)

I'm not sure however how safe this is since the loop is not being closed.

1 Comment

The question's point is that work should be async
4

You want: Run a function asynchronously (without using a thread) and also ensure the function calling this async function continues its execution.

As user Mikhail Gerasimov has posted, this is impossible. You must have an event loop running in a different thread and the async function scheduled to run in that event loop. Function run_async is how I would do it:

import asyncio
import threading

_event_loop = None
_event_lock = threading.Lock()

def run_async(coro):
    """await a coroutine from a synchronous function/method."""

    global _event_loop

    if not _event_loop:
        with _event_lock:
            if not _event_loop:
                _event_loop = asyncio.new_event_loop()
                threading.Thread(target=_event_loop.run_forever,
                                 name="Async Runner",
                                 daemon=True
                                 ).start()

    return asyncio.run_coroutine_threadsafe(coro, _event_loop)

import time

async def async_fn():
    # simulate work:
    await asyncio.sleep(1)
    return time.time()  # When we finished

def sync_fn():
    waitable = run_async(async_fn())
    # simulate work
    import time
    time.sleep(1)
    t = time.time()  # when we finished
    # Wait for coroutine to complete and get its return value:
    return_value = waitable.result()
    print(t, return_value)

if __name__ == '__main__':
    sync_fn()

Prints:

1747424812.4480834 1747424812.4579432

Comments

1

I'm not sure what you are trying to achieve. By design coroutines won't run if the current thread is occupied by running some sync code. create_task as the name says just create a task entity somewhere inside asyncio and will run when you await something either returned task or any other awaitable object. To run it in current thread you need to explicitly switch context to it by calling await.

import asyncio


async def foo():
    await asyncio.sleep(5)
    return 'Hi there'

def bar():
    return asyncio.create_task(foo())

async def baz():
    print(await bar())

async def main():
    await baz()


if __name__ == '__main__':
    asyncio.run(main())

Comments

0

A slight variation on another answer, to make sure timeout is always taken into account, to simplify the code and explain the reasons for decisions used:

import asyncio
import typing as t
from concurrent.futures import ThreadPoolExecutor

if t.TYPE_CHECKING:
    from collections.abc import Coroutine


def run_coroutine_sync[T](coroutine: "Coroutine[t.Any, t.Any, T]", timeout: float = 30) -> T:
    assert asyncio.iscoroutine(coroutine)

    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        return asyncio.run(asyncio.wait_for(coroutine, timeout=timeout))

    if not loop.is_running():
        return loop.run_until_complete(asyncio.wait_for(coroutine, timeout=timeout))

    with ThreadPoolExecutor() as executor:
        fut = executor.submit(lambda: asyncio.run(coroutine))
        return fut.result(timeout=timeout)
  • When not in an async context, we just use asyncio.run which will start a new event loop and run the task.

  • If we are in an async context, and (in the very rare case) the loop is not yet running, start the loop and run our task inside it.

  • In the case that we are in an async context, with an already running loop, then start a new thread, and run the task in there.


The biggest hurdle is that asyncio loops are thread-specific, so you cannot have more than one running per thread.

Comments

-5

Assuming the synchronous function is inside an asynchronous function, you can solve it using exceptions. Pseudo code:

class CustomError(Exception):
pass


async def main():
    def test_syn():
        time.sleep(2)
        # Start Async
        raise CustomError
    try:
        test_syn()
    except CustomError:
        await asyncio.sleep(2)
    

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.