0

I have following data in a unix flatfile and I want to suppress duplicate values in first two columns and convert it as shown below:

1:x:4:3:2:y
1:x:7:9:l:z
1:0:3:j:k:m
2:9:r:s:6:u
2:m:y:5:7:9
2:u:7:9:7:6
3:a:b:c:d:e
3:a:b:d:e:f
3:a:b:n:r:s

To:

1:x:4:3:2:y
 : :7:9:l:z
 :0:3:j:k:m
2:9:r:s:6:u
 :m:y:5:7:9
 :u:7:9:7:6
3:a:b:c:d:e
 : :b:d:e:f
 : :b:n:r:s
Improve this question
7
  • 1
    What have you tried so far and where are you stuck? Commented Oct 17, 2019 at 5:56
  • I might help you if you are allowed to use Ruby programming language... Is that allowed? Commented Oct 17, 2019 at 5:57
  • @s.goswami yes pls go ahead Commented Oct 17, 2019 at 6:02
  • bash-4.4$ which ruby /usr/bin/ruby bash-4.4$ ruby -v ruby 1.9.3p551 (2014-11-13 revision 48407) [sparcv9-solaris2.11] bash-4.4$ Commented Oct 17, 2019 at 6:03
  • 1
    Are you only concerned with consecutive duplicates, or would an x in column 2 in a line further down also be deleted? Commented Oct 17, 2019 at 7:22

2 Answers 2

Reset to default
3

Straightforwardly with awk:

$ awk -F':' '++a[$1] > 1{ $1=" " }++b[$2] > 1{ $2=" " }1' OFS=':' inp_file
1:x:4:3:2:y
 : :7:9:l:z
 :0:3:j:k:m
2:9:r:s:6:u
 :m:y:5:7:9
 :u:7:9:7:6
3:a:b:c:d:e
 : :b:d:e:f
 : :b:n:r:s
Improve this answer
7
  • bash-4.4$ awk -F':' '++a[$1] > 1{ $1=" " }++b[$2] > 1{ $2=" " }1' OFS=':' Array_e.log awk: syntax error near line 1 awk: bailing out near line 1 bash-4.4$ Commented Oct 17, 2019 at 6:35
  • @Naga, post your awk version Commented Oct 17, 2019 at 7:04
  • that will also show an extra line like 3:x:b:n:r:s at the end with x being a duplicate of the a above it. I'm not sure that's what's intended -- it's hard to say based on the OP. Commented Oct 17, 2019 at 9:15
  • @RomanPerekhrest . how to find awk version ? awk --version hangs bash-4.4$ which awk /usr/bin/awk bash-4.4$ /usr/bin/awk --version Commented Oct 17, 2019 at 10:01
  • @Naga, and what's the output of /usr/bin/awk --version ? Commented Oct 17, 2019 at 10:03
0

As @Naga mentioned in the comments that a Ruby script is allowed. Here I have came up with a solution.

Ruby Script

#!/usr/bin/env ruby
# Frozen_String_Literal: false
$-v = nil
d, $-s, i, $,, $; = [], ?\s.freeze, '', ?:.freeze, ?:.freeze
::FILENAME = $*[0] || abort("Pass the filename as Argument.\nExample:\n\t#{File.basename($0)} hello.rb")

puts IO.readlines(File.join(Dir.pwd, FILENAME)).map! { |x|
    c = x.tap(&:strip!).split
    d.clear && i.replace(c[0].to_s) if i != c[0]
    a = c.first(2).map { |y| d.find { |z| z.include?(y) } ? $-s : y }.join << $, << c.drop(2).join unless c.empty?
    d << c.first(2).join
    a
} rescue exit!

Running

ruby scriptname.rb inputfile.txt

Output

[Assuming inputfile.txt contains the same data in the question]

1:x:4:3:2:y
 : :7:9:l:z
 :0:3:j:k:m
2:9:r:s:6:u
 :m:y:5:7:9
 :u:7:9:7:6
3:a:b:c:d:e
 : :b:d:e:f
 : :b:n:r:s

Hope this helps!

Improve this answer
3
  • ruby script is working fine, but i have challenge 1:x,a:4:3:2:y 1:x,1:7:9:l:z 1:0,2:3:j:k:m 2:9:r:s:6:u 2:m:y:5:7:9 2:u,1:7:9:7:6 3:a:b:c:d:e 3:a,2:b:d:e:f 3:a,1:b:n:r:s Commented Oct 17, 2019 at 10:03
  • is that space splitted instead of new line characters? Commented Oct 17, 2019 at 15:45
  • I have edited the answer so that if the given file doesn't exist (or for any other error), it will exit with return value 1. Also, the script will not raise errors for invalid files... Commented Oct 17, 2019 at 16:09

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.