Running Python Script from Bash - How to Redirect the Python Logging File

Question

I am running a series of python scripts from a bash script.

eg

nohup python $IMPORTER_PATH/importer.py -t styles -e $ELASTIC_URL -j $STYLES_DATA_PATH  -b -f &>> "$LOG_PATH/styles-$(date "+%d-%b-%Y").log"
nohup python $IMPORTER_PATH/importer.py -t objects -e $ELASTIC_URL -j $OBJECTS_DATA_PATH  -b -f &>> "$LOG_PATH/objects-$(date "+%d-%b-%Y").log"

At the moment I am successfully generating log output of stdout and stderr with the use of &>> My python script had been littered with useful print statements such as print("Process started") and therefore the bash generated log file did have some use.

However, I have recently been cleaning up my log files and using the python logging module to generate more focused log files.

Here's the extract from my importer.py script:

import argparse
import logging

def parse_args(args):
    parser = argparse.ArgumentParser()
    parser.add_argument("-t", "--type", help="Type of data to import")

def main(argv):
    # Set up Logging levels
    logging.basicConfig(filename='importer.log',
                        filemode='w', level=logging.INFO)
    es_log = logging.getLogger("elasticsearch")
    es_log.setLevel(logging.WARNING)

    args = parse_args(argv)
    logging.info('Started with %s', args)

The problem now is that the importer.log is created wherever the bash script is run from, and because I have set filemode='w', it gets overwritten as each python script is called.

Is there a way for me to redirect the output from importer.log to my dated log file such as "$LOG_PATH/styles-$(date "+%d-%b-%Y").log"?

Or should I add another argument to my importer.py ArgumentParser and pass it the destination path and file name?

Luca Angioloni · Accepted Answer · 2020-11-27 02:33:52Z

1

You can format the name of the log file as you want. Here is an example:

from datetime import datetime
log_name = f"importer_{datetime.strftime(datetime.now(), '%Y-%m-%d')}.log"
# This becomes: importer_2020-11-27.log

You can of course change the format of the date ecc...

You could also use this:

import os
script_path = os.path.realpath(__file__)
# This points to your script: /path/to/script.py

To get the path of your script if you want to save the logs there.

log_name = os.path.join(os.path.dirname(script_path), log_name)
# now together with the first snippet of code you get: /path/to/importer_2020-11-27.log

answered Nov 27, 2020 at 2:33

Luca Angioloni

2,2532 gold badges21 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

whatapalaver Over a year ago

Thanks this is helpful, renaming the logs would ensure I had a different one for each process.

Collectives™ on Stack Overflow

Running Python Script from Bash - How to Redirect the Python Logging File

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related