Python and environment variables not working

Question

In a Docker container on Ubuntu, I'm running a .sh file which calls python3 my_script_file.py. In that file, I do:

commands = []
commands.append("export OMP_NUM_THREADS=4")
commands.append("echo $OMP_NUM_THREADS")

I then execute the commands array with:

def execute_commands_list(commands, no_bash_script = False):
    for command in commands:
        if not no_bash_script and gc.output_to_bash_script:
            write_command_to_bash_script(command)
        else:
            print("     [  " + command + "  ]")

            ret = subprocess.call(command, shell=True, executable="/bin/bash")

            assert ret==0 or ret==2, "bash return code error: " + str(ret) + " " + command

Here is the output:

     [  export OMP_NUM_THREADS=4  ]
     [  echo $OMP_NUM_THREADS  ]

(that's a blank line). I try running echo $OMP_NUM_THREADS in the terminal after execution myself, and get nothing. If I run those both commands manually however the env var is set.

(I have also tried os.environ, and that has failed as well)

What am I missing?

(For reference, the .sh file is:)

#!/bin/bash -p

python3 my_script_file.py

Avihoo Mamka · Accepted Answer · 2019-12-13 11:22:50Z

That's probably because the scope of export is limited only to the current shell it's being running on. Meaning that only the shell the subprocess ran from, this environment variable will be visible to.

If you want to make this environment variable globally exposed, and given you're running from a linux machine of some sort, I'd suggest writing it to /etc/environment in the format of KEY=VALUE.

For example in your case, instead of

export OMP_NUM_THREADS=4

Replace it with:

echo OMP_NUM_THREADS=4 >> /etc/environment

Note! Since environment is located under /etc there's a good chance its permissions are set to allow write only to a sudo user, so make sure you include that in your subprocess call.

You can use something like the following:

echo 'OMP_NUM_THREADS=4' | sudo tee -a /etc/environment > /dev/null

tee prints the text to stdout, too. In order to mute it so it behaves more similar to shell appending (>>), route the stdout to /dev/null.

Collectives™ on Stack Overflow

Python and environment variables not working

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related