1

I have list of file names file1,file2,file3
I want to pass these file names in the script and remove the special characters

I prepared sed command to remove the special characters 

sed  -i -e 's/^B/,/g' /home/data/nfiledata/ 
hdfs dfs -put -f /home/data/nfiledata/*  user/sai/table1/nfiledata/
gzip  /home/data/nfiledata/*

sed  -i -e 's/^B/,/g' /home/data/marginfile/  
hdfs dfs -put -f /home/data/marginfile/*  user/sai/table2/marginfile/
gzip  /home/data/marginfile/*

sed  -i -e 's/^B/,/g' /home/data/calldata/  
hdfs dfs -put -f /home/data/calldata/*  user/sai/table3/calldata/
gzip  /home/data/calldata/*

My question is instead of writing multiple times same command can i write in one command and loop the process for each file using Shell script 

nfile = (nfiledata,margindata, calldata)
while IFS= read -r nfile
do
  sed  -i -e 's//,/g' /home/data/$nfile/
  hdfs dfs -put -f /home/data/$nfile/*  user/sai/table$/$nfile/
  gzip  /home/data/$nfile/*
done < "home/data/$nfile"
Improve this question
4
  • 2
    What are you hoping sed 's//,/g' will do? Commented May 26, 2016 at 20:08
  • files=( file1 file2 file3 ) is correct array assignment syntax. Can't have whitespace around the =s, can't have commas. Commented May 26, 2016 at 20:17
  • 1
    I strongly suggest running your code through shellcheck.net and fixing everything it finds before asking questions here. Commented May 26, 2016 at 20:17
  • sed normally operates on files not directories. In the code sed -i -e 's/^B/,/g' /home/data/file1/ , what are you expecting sed to do? Commented May 26, 2016 at 20:25

2 Answers 2

Reset to default
1

A for loop, not a while read loop, is appropriate here:

nfile=(file1 file2 file3)
for f in "${nfile[@]}"; do
  sed  -i -e 's/^B/,/g' /home/data/"$f"/ # should this be "$f"/* ?
  hdfs dfs -put -f /home/data/"$f"/*  user/sai/table1/"$f"/
  gzip /home/data/"$f"/*
done

Noteworthy components:

  • Assignments must not have spaces around the =. Commas are not part of array syntax in bash -- unquoted, unescaped whitespace acts as a separator in that case as elsewhere.
  • Expansions such as $f must be inside double quotes to be performed safely (without string-splitting or globbing).
  • Glob expansions, such as *, must be outside quotes to be honored.
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thank you Charles. Even if my file names are different i can use this script? correct? My file names are like (nfile_dat, fileidentifier,margindata,calldata) . Each file represent a table in my hadoop.
@Sai, of course; put the actual directory names you want into the array contents. (Your code is written with file1, file2 and file3 being directory names, not file names; if they are in fact filenames, the original proposed commands wouldn't be correct either because of the trailing /s).
@Sai, ...so, if they're actually files, make it /home/data/"$f" where currently written /home/data/"$f"/* or /home/data/"$f"/.
1

Answer for original version of question

The same sed command can be applied to edit-in-place several files with a single invocation:

sed  -i -e 's/old/new/g' /home/data/file1 /home/data/file2 /home/data/file3

Also, if the file names really are that simple, then brace expansion can be used:

sed  -i -e 's/old/new/g' /home/data/file{1..3}

Or,

sed  -i -e 's/old/new/g' /home/data/file[123]

Or, if there are no other similarly named files that you want to exclude, pathname expansion may be sufficient:

sed  -i -e 's/old/new/g' /home/data/file?

Example of real file names

sed  -i -e 's/old/new/g' nfile_dat fileidentifier margindata calldata
Improve this answer

4 Comments

what about /home/data/file[123] or /home/data/file? ?
@MaxU Yes. I didn't show that originally because the OP's it is likely that file1, file2, etc are just stand-ins and the real file names are more complex. If not, however, I added your suggestions to the answer.
Thank you for your answer John. What if my file names are different like (nfile_dat, fileidentifier,margindata,calldata) ? Also how can i loop the data
@Sai I just added to the answer an example using those file names.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.