Import Multiple Text files (Large Number) using numpy and Post Processing

Question

This forum has been extremely helpful for a python novice like me to improve my knowledge. I have generated a large number of raw data in text format from my CFD simulation. My objective is to import these text files into python and do some postprocessing on them. This is a code that I have currently.

import numpy as np
from matplotlib import pyplot as plt
import os

filename=np.array(['v1-0520.txt','v1-0878.txt','v1-1592.txt','v1-3020.txt','v1-5878.txt'])

for i in filename:
    format_name= i
    path='E:/Fall2015/Research/CFDSimulations_Fall2015/ddn310/Autoexport/v1'
    data= os.path.join(path,format_name)
    X,Y,U,V,T,Tr = np.loadtxt(data,usecols=(1,2,3,4,5,6),skiprows=1,unpack = True) # Here X and Y represents the X and Y coordinate,U,V,T,Tr represents the Dependent Variables
    plt.figure(1)
    plt.plot(T,Y)
    plt.legend(['vt1a','vtb','vtc','vtd','vte','vtf'])
    plt.grid(b=True)

Is there a better way to do this, like importing all the text files (~10000 files) at once into python and then accessing whichever files I need for post processing (maybe indexing). All the text files will have the same number of columns and rows.

I am just a beginner to Python.I will be grateful if someone can help me or point me in the right direction.

Welcome to SO! that code is not valid python; thus one could assume it has not been run once yet; maybe you want to clean the code and avoid that impression. — decltype_auto
– decltype_auto, Commented Nov 9, 2015 at 18:42
Welcome to StackOverflow. Please read and follow the posting guidelines in the help documentation. Minimal, complete, verifiable example applies here. We cannot effectively help you until you post your code and accurately describe the problem. For instance, your post-processing here consists of five plots from a single data file; there is no interaction among the data files; hence, no reason to have more than one active at any time. We have no idea why you're writing out data. — Prune
– Prune, Commented Nov 9, 2015 at 19:00
Thanks for the reply. It consists of 6 plots from six different text files. For eq, in 'v1-0001.txt' file 0001 represents the time step associated with the data. So I want to be able to see the variation of a dependent variable in time. So in this present code, it is plotting the variation of temperature along x at six different timesteps. Please let me know where I am getting this wrong. — narayan.p
– narayan.p, Commented Nov 9, 2015 at 19:58

hpaulj · Accepted Answer · 2015-11-09 20:38:10Z

1

Your post needs to be edited to show proper indentation.

Based on a quick read, I think you are:

reading a file, making a small edit, and write it back
then you load it into a numpy array and plot it

Presumably the purpose of your edit is to correct some header or value.

You don't need to write the file back. You can use content directly in loadtxt.

content = content.replace("nodenumber","#nodenumber") # Ignoring Node number column
data1=np.loadtxt(content.splitlines())
Y=data1[:,2]
temp=data1[:,5]

loadtxt accepts any thing that feeds it line by line. content.splitlines() makes a list of lines, which loadtxt can use.

the load could be more compact with:

Y, temp = np.loadtxt(content.splitlines(), usecols=(2,5), unpack=True)

With usecols you might not even need the replace step. You haven't given us a sample file to test.

I don't understand your multiple file needs. One way other you need to open and read each file, one by one. And it would be best to close one before going on to the next. The with open(name) as f: syntax is great for ensuring that a file is closed.

You could collect the loaded data in larger lists or arrays. If Y and temp are identical in size for all files, they can be collected into larger dimensional array, e.g. YY[i,:] = Y for the ith file, where YY is preallocated. If they can vary in size, it is better to collect them in lists.

edited Nov 9, 2015 at 20:38

answered Nov 9, 2015 at 20:32

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

narayan.p Over a year ago

Thank you hpaulj for your suggestions. I have included an improved code with proper indentations. I request you for some clarifications. How can I provide a sample file to test? Is there a way to upload the text files in SO? What do you mean by "collect the data in larger lists or arrays"? Can you show me an example code? Is it like, I am reading all the text files and then assigning them to some arrays? Pardon me if some of my questions are redundant or stupid.

Collectives™ on Stack Overflow

Import Multiple Text files (Large Number) using numpy and Post Processing

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related