Implementing "SystemCalls" with Coroutines in Python

Question

Am currently reading through the tutorial doc http://www.dabeaz.com/coroutines/Coroutines.pdf and got stuck at the (pure coroutine) multitask part, in particular the System call section.

The part that got me confused is

class Task(object):
    taskid = 0
    def __init__(self,target):
        Task.taskid += 1
        self.tid = Task.taskid # Task ID
        self.target = target # Target coroutine
        self.sendval = None # Value to send

    def run(self):
        return self.target.send(self.sendval)


def foo():
    mytid = yield GetTid()
    for i in xrange(5):
        print "I'm foo", mytid
        yield


class SystemCall(object):
    def handle(self):
        pass

class Scheduler(object):
    def __init__(self):
        self.ready = Queue()
        self.taskmap = {}

    def new(self, target):
        newtask = Task(target)
        self.taskmap[newtask.tid] = newtask
        self.schedule(newtask)
        return newtask.tid

    def schedule(self, task):
        self.ready.put(task)

    def mainloop(self):
        while self.taskmap:
        task = self.ready.get()
        try:
            result = task.run()
            if isinstance(result,SystemCall):
                result.task = task
                result.sched = self
                result.handle()
                continue
            except StopIteration:
                self.exit(task)
                continue
        self.schedule(task)

And the actual calling

sched = Scheduler()
sched.new(foo())
sched.mainloop()

The part I don't understand is that how the tid got assigned to mytid in foo()? In the order of things, it seems to be like (starting from sched.mainloop()). Please correct me if I get the flow wrong.

Assumptions: let's name some of the things

the_coroutine = foo()
scheduler = Scheduler
the_task = scheduler.new(the_coroutine) # assume .new() returns the task instead of tid

scheduler: .mainloop() is called
scheduler: the_task.run() is called
the_task: the_coroutine.send(None) is called
the_corou: yield GetTid(), and it returns an instance of GetTid to scheduler before the None in 3 is sent to the yield statement in loop. (am i right?)
the_corou: (also at the same time?) myTid is assigned as the instance of GetTid()?
scheduler: result = ~~theTask.run()~~ ~~the_task.run()~~ <an instance of GetTid>
scheduler: result is indeed an instance of SystemCall
scheduler: result.handle() is called
GetTid instance: scheduler.schedule(the_task)
scheduler: current iteration is done, start a new one
scheduler: (assume no other task in queue) the_task.run() is called
scheduler: the_coroutine.send() is called
I am lost?

When it reaches step 12, apparently the loop is already started and is able to print the tid whenever the scheduler runs the respective task. However, when exactly was the value of tid assigned to mytid in foo()? I am sure I missed something in the flow, but where (or even completely wrong)?

Then I notice the part where the Task object calls .send(), it returns a value, so .send() returns a value?

zehnpaard · Accepted Answer · 2015-01-02 10:58:14Z

1

You seem to have omitted the Scheduler's new method, which is almost certainly where the assignment occurs. I imagine the result of GetTid() that is yielded is immediately sent back into the coroutine using .send().

Regarding .send(), yes you're correct, .send() returns a value.

Try the example below to see what's going on. .send assigns a value to the variable on the left side of the = yield, and execution of code within the coroutine resumes until it hits the next yield. At that point, the coroutine yields, and whatever it yields is the return value of the .send.

>>> def C():
...     x = yield
...     yield x**2
...     print 'no more yields left'
...
>>> cor = C()
>>> cor.next()
>>> yielded = cor.send(10)
>>> print yielded
100
>>> cor.next()
no more yields left
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Moving some of the in-comment points into the answer

So, how does x = yield y work?

When the coroutine hits that line of code, it yields the value y, then halts execution, waiting for someone to invoke its .send() method with an argument.

When someone does invoke .send() whatever is the argument in the .send is assigned to variable x, and the coroutine begins to execute code from there upto the point of its next yield statement.

Edit: Whoa boy it gets even more complex... I skimmed through this David Beazley presentation before, but to be honest I'm better acquainted with his other two talks on generators... Going through the linked material, it looks like this definition of GetTid is what we're after.

class GetTid(SystemCall):
    def handle(self):
        self.task.sendval = self.task.tid
        self.sched.schedule(self.task)

I quote from his presentation: "The operation of this is little subtle". haha.

Now look at the mainloop:

if isinstance(result,SystemCall):
    result.task = task
    result.sched = self
    result.handle() # <- This bit!
    continue

result here is the GetTid object, which runs its handle method which sets its' task's sendval attribute to the task's tid, and the schedules the task by putting it back in the queue.

Once the task is retrieved from the queue, the task.run() method is run again. Let's look at the Task object definition:

class Task(object):
...
    def run(self):
        return self.target.send(self.sendval)

When task.run() is invoked for this second time, it will send its sendval value (which was previously set by result.handle() to its tid) to its .target - the foo coroutine. This is where the foo coroutine object finally receives the value of its mytid.

The foo coroutine object runs until its next yield, printing its message along the way, and returns None (because there's nothing to the right of yield). That None is the return value of the task.run() method.

That is NOT an instance of a SystemCall, so the task is not handled/scheduled upon the second pass.

Other evil things probably happen too but that's the flow that I'm seeing for now.

edited Jan 2, 2015 at 10:58

answered Jan 2, 2015 at 9:24

zehnpaard

6,2732 gold badges28 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Jeffrey04 Over a year ago

oh, so the return value of .send is the next yield (in this case x**2), thanks (: (just added the missing method definition to the question).

Jeffrey04 Over a year ago

while I understand what you're trying to do here, I still don't see how yield GetTid() ends up returning the tid to mytid in foo()....

zehnpaard Over a year ago

Regarding your first question - None is never printed, it just gets passed around the scheduler, foo, task and GetTid objects, causing them to behave in certain ways.

zehnpaard Over a year ago

Regarding your second question, that's not how x = yield y works. The coroutine returns the value y, then waits for someone to invoke its .send() method. Whatever is the argument in the .send is assigned to variable x, and the coroutine begins to execute code from there upto the point of its next yield statement. At mytid = yield GetTid(), foo will yield the object created by GetTid(). Then it waits until .send(sendval) is called, and the value of sendval is assigned to mytid and code begins to execute in foo again.

zehnpaard Over a year ago

No problem! Coroutines are the most mind-boggling concept I've encountered in Python, but seemingly with some weird alien aesthetics - I'm glad I could articulate their mechanism.

|

Collectives™ on Stack Overflow

Implementing "SystemCalls" with Coroutines in Python

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related