how do i process multiple files with an existing python script

Question

i have >100 .cwa files (raw accelerometery data) and have a .py programme which processes the .cwa data and produces a .csv file. How do i write a script in python which can process all the .cwa files with my .py without having to individually insert the filename into the command line.

i am new to programming, i have looked at glob.glob and os. walk but don't understand how to use these process multiple files. i want to read in 1 x .cwa file at a time, process it using my .py and then move onto the next .cwa until all have been read and there are the corresponding number of csv files as an output. i do not want to merge the data into one file.

glob and os.walk are good tools for a single directory or whole subtree search. I suggest you learn those tools. Searching "python glob" and "python os.walk" gives lots of examples and tutorials. You could put [python] glob in the search bar here on stackoverflow and get lots of good hits. — tdelaney
– tdelaney, Commented Jun 9, 2020 at 22:33

Benjamin · Accepted Answer · 2020-06-09 22:24:21Z

1

You can use shell scripting (here is a bash example for Linux) to list files in a directory and call a command for each file:

find /path/to/cwa/files/ -type f -name "*.cwa" -exec /path/to/script.py {} \;

This example assumes that script.py takes a single argument which is a CWA filepath, e.g. script.py /my/cwa/file.cwa

If you want to stick with Python, a for-loop with os.listdir might be the easiest approach. Let's assume that your CWA-processing code is in the process_cwa() function:

import os

CWA_DIR = '/path/to/cwa/files'

def process_cwa(path):
    # process CWA file
    pass

# process CWA files from CWA_DIR on disk
for file in os.listdir(CWA_DIR):
    if file.endswith(".cwa"):
        path = os.path.join(CWA_DIR, file)
        process_cwa(path)

edited Jun 9, 2020 at 22:24

answered Jun 9, 2020 at 22:01

Benjamin

5364 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SD12 Over a year ago

thankyou - the processing cwa code is however in a .py file rather than a function, so I would need to call the .py.

tdelaney Over a year ago

By the time OP figures out find on linux or equivalent on Windows, s/he might as well have figured out glob or os.walk.

Collectives™ on Stack Overflow

how do i process multiple files with an existing python script

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related