4

Background

I have a Python 3 script called server.py that uses the built-in http.server module. The script boils down to the following:

from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler

class MyRequestHandler (BaseHTTPRequestHandler):
    def do_POST(self):
        # Code omitted for brevity

    def do_GET(self):
        # Code omitted for brevity

def start_server():

    # Begin serving
    # -------------
    server = HTTPServer(('', port), MyRequestHandler)
    print("server now running on port {0} ...".format(port))

    server.serve_forever()

# Start the Server
# ----------------
if __name__ == '__main__':
    start_server()

MyRequestHandler handles GET and POST requests by importing modules "on-the-fly" depending on the URI that was used for the request.

The above works fine, however, subsequent to the creation of that script, there has been a requirement to be able to remotely update the whole "package" of scripts (i.e. the server script, plus all of the "module scripts" that are loaded up "on-the-fly" and are located in sub-folders).

For this, I have written another server script in Python 3 (called updater.py) which, when instructed, will retrieve a zip file, which it then unzips to overwrite the original server.py script along with all the other associated scripts and sub-folders.

Question

This all works great, but I have now hit a wall. The best approach to this, I thought, would be to have the updater.py script control the running of server.py. It could shut down the server.py, and everything linked to it before overwriting it all and then give it a clean start after it is overwritten.

On that basis, the road that I have gone down is to use subprocess.Popen to start the server, believing that I could just kill the Python process before overwriting the server.py stuff, however, this is not working as hoped. Here is my trial.py script that I have written to test the theory:

import sys
import subprocess

def main():
    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'])
        print("Started process:")
        print(proc.pid)
        return proc

    def kill_process(the_process):
        print("Killing process:")
        print(the_process.pid)
        the_process.kill()

    process = None

    while True:
        user_input = input("Type something: ")

        if user_input == 'start':
            process = start_process()
        if user_input == 'kill':
            kill_process(process)
        if user_input == 'exit':
            break

if __name__ == '__main__':
    main()

This does appear to start and kill a Python process, but the server is not running while this script is running, so I am not sure what it is starting and killing! Typing "start" and then "exit" (and thus quitting the trial.py script) allows the server to run, though I can't understand why, since I thought subprocess.Popen should cause the spawned process to run independently of the parent process?

edit: Thanks to @Håken Lid's astute observation below, I noticed that all I was doing is breaking out of the while loop, not exiting the script. This leads me to believe that the while loop is somehow blocking the sub-process from running (since once the loop is exited, the server will start).

Improve this question
2
  • 2
    If the subprocess were independent of the parent process, it wouldn't be called a _sub_process, would it? Commented Nov 6, 2017 at 14:16
  • How embarrassing! I know that, yet my brain somehow went on a tangent. I was confusing starting a separate process with starting a separate independent process. I think it's fair to say (now that you've encouraged me to re-read my question and code) that the while loop is blocking the execution of the subprocess in some way. Commented Nov 6, 2017 at 14:46

2 Answers 2

Reset to default
1

per our discussion, I'd recommend some way to empty the stdio buffers from "server.py". If you want to also be able to give user input, you'll need a thread to do the printing (or just to empty the buffers into a black hole) while you wait for user input on the main thread. Here's a rough idea of how I might do it..

import sys
import subprocess
from threading import Thread 
#This could probably be solved with async, but I still
#haven't learned async as well as I know threads

def main():

    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'], 
                                stdin=subprocess.PIPE, 
                                stdout=subprocess.PIPE)
        print("Started process:")

        def buf_readerd(proc, inbuf, outbuf):
            while proc.poll() is None:
                outbuf.write(inbuf.readline()) #may need to add a newline.. I'm not sure if readline ends in a \n

        stdoutd = Thread(target=buf_readerd, args=(proc, proc.stdout, sys.stdout), daemon=True)
        stderrd = Thread(target=buf_readerd, args=(proc, proc.stderr, sys.stderr), daemon=True)
        stdoutd.start()
        stderrd.start()
        print("started pipe reader daemons")

        print(proc.pid)
        return proc
# ...
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

Okay, so I think I have fixed this myself now. The problem was that I was blocking the sub-process execution with the call to input() inside the while loop. The script does nothing until it receives some input. I made subprocess.poll() an integral part of the loop and put the call to input() inside this loop. If I hit <return> a few times, the server will start and I can use it. If I then type kill, the server will be killed, as expected.

I am not sure how this will work in the context of my updater.py script, but I have learnt a valuable lesson in "thinking more deeply before posting to StackOverflow"!

import sys
import subprocess

def main():
    def start_process():
        proc = subprocess.Popen([sys.executable, 'server.py'])
        print("Started process:")
        print(proc.pid)
        return proc

    def kill_process(the_process):
        print("Killing process:")
        print(the_process.pid)
        the_process.kill()

    user_input = input("Type something: ")

    if user_input == 'start':
        process = start_process()

        while process.poll() is None:
            user_input = input()
            if user_input == 'kill' or user_input == 'exit':
                kill_process(process)

if __name__ == '__main__':
    main()
Improve this answer

7 Comments

that behavior seems odd to me. have you tried different stdin and stdout configurations to see if you're waiting on a full buffer? that may become an issue later if it's not resolved correctly.
perhaps include stdin=subprocess.PIPE and stdout=subprocess.PIPE in the popen constructor and start a helper thread to read their contents and dump to sys.stdout and sys.stderr periodically while waiting for user input in the main thread
I find it a little strange too. As soon as the process is started, it polls and gets back "None", fair enough. It then waits for user input. At this point I would expect that the server process is running, but it's not. The "server now running ..." message does not appear until there has been 2/3 attempts at user input (i.e. hitting <return> 2 or 3 times). The server also does not respond until that message has appeared, so it is definitely not running prior to hitting <return> a few times.
what is printing "server now running"? the 'server.py' or the main process?
Actually, removing the "user_input = input()" line from within the while-loop causes the server to start immediately. To be fair, this is a distraction because I won't be using Popen in this context - this was just a test. I actually want to run the sub-process from my updater.py script and then when I make a "POST" to that script, have it kill the sub-process. In that context, the call to subprocess.poll() will be wrapped in an "http.server.serveforever" loop, which I hope does not block the subprocess in any way...
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.