I have been running below script on a Red Hat server, and it works fine and finishes the job. The file I am feeding it, contains half a million lines in it (approximately 500000 lines), and that's why (to finish it faster) I have added an '&' at the end of while loop block

But now I have setup a Desktop with 8 GB of RAM running Ubuntu 18.04 on it, and running the same code only finishes a few thousand lines and then hangs. I read a bit about it and increased the stack limit to unlimited as well and still it hung after 80000 lines or so, Any suggestions about how can I optimize the code or tune my PC parameters to always finish the job?

while read -r CID60

do    

 { 

       OLT=$(echo "$CID60" | cut -d"|" -f5) 

       ONID=${OLT}:$(echo "$CID60" | cut -d, -f2 | sed 's/ //g ; s/).*|//') 

       echo $ONID,$(echo "$CID60" | cut -d"|" -f3) >> $localpath/CID_$logfile.csv       

  } &     

done < $localpath/$CID7360

Input:

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ASSN45| Unlocked|12-654-0330|Up|202-00_MSRFKH00OL6|P282018767.C2028 ( network, R1.S1.LT7.PON8.ONT81.SERV1 )|



202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ASSN46| Unlocked|12-654-0330|Down|202-00_MSRFKH00OL6|P282017856.C881 ( local, R1.S1.LT7.PON8.ONT81.C1.P1 )|



202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ASSN52| Unlocked|12-664-1186|Up|202-00_MSRFKH00OL6|P282012623.C2028 ( network, R1.S1.LT7.PON8.ONT75.SERV1 )|

output:

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330



202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330



202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186

my output of interest is 5th column ( separated with pipe | ) being concatenated with part of last column, and then the third column

Perl solution

This script doesn't do anything in parallel but is quite fast regardless.
Save it as filter.pl (or whatever name you prefer) and make it executable.

#!/usr/bin/env perl



use strict;

use warnings;



while( <> ) {

    if ( /^(?:[^|]+|){2}([^|]+)|[^|]+|([^|]+)|[^,]+,s*(S+)/ ) {

        print "$2:$3,$1n";

    }

}

I copied your sample data until I got 1,572,864 lines and then ran it as follows:

me@ubuntu:~> time ./filter.pl < input.txt > output.txt

real    0m3,603s

user    0m3,487s

sys     0m0,100s



me@ubuntu:~> tail -3 output.txt

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186

If you prefer one-liners, do:

perl -lne 'print "$2:$3,$1" if /^(?:[^|]+|){2}([^|]+)|[^|]+|([^|]+)|[^,]+,s*(S+)/;' < input.txt > output.txt

A pure sed solution:

sed -r 's/^[^|]+|[^|]+|([^|]+)|[^|]+|([^|]+)|.+( .+, ([^ ]+).+/2:3,1/' <in.dat >out.dat

Oneliner

If the order of the items and the separators can be different from what you specify in the question, I thought the following one-liner would do it,

< input tr ' ' '|' | cut -d '|' -f 4,6,10 > output

but in a comment you wrote that you need exactly the specified format.

I added a solution with 'awk', which is approximately on par with PerlDuck's solution with perl. See the end of this answer.

< input awk '{gsub("\|"," "); print $5 ":" $9 "," $3}' > output

Test

The test was done in my computer with Lubuntu 18.04.1 LTS, 2*2 processors and 4 GiB RAM.

I made a huge infile by 'doubling 20 times' from your demo input (1572864 lines), so some margin to your 500000 lines,

Oneliner with cut and sed:

$ < infile cut -d '|' -f 3,5,6 | sed -e 's/|[A-Z].*, /|/' -e 's/ )$//' > outfile

$ wc -l infile

1572864 infile

$ wc -l outfile

1572864 outfile

$ tail outfile

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

Timing

We might expect, that a pure sed solution would be faster, but I think that reordering of the data slows it down, so that the cut and sed solution is faster. Both solutions work without any problem in my computer.

Oneliner with cut and sed:

$ time < infile cut -d '|' -f 3,5,6 | sed -e 's/|[A-Z].*, /|/' -e 's/ )$//' > outfile



real    0m8,132s

user    0m8,633s

sys     0m0,617s

A pure sed oneliner by xenoid:

$ time sed -r 's/^[^|]+|[^|]+|([^|]+)|[^|]+|([^|]+)|.+( .+, ([^ ]+).+/2:3,1/' <infile > outfile-sed 



real    1m8,686s

user    1m8,259s

sys     0m0,344s

A perl oneliner by PerlDuck is faster than the previous oneliners:

$ time perl -lne 'print "$2:$3,$1" if /^(?:[^|]+|){2}([^|]+)|[^|]+|([^|]+)|[^,]+,s*(S+)/;' < infile > outfile.perl



real    0m5,929s

user    0m5,339s

sys     0m0,256s

Oneliner with tr and cut with a tr -s command:

I used tr to convert the spaces in the input file to pipeline characters and then cut could do it all without sed. As you can see, tr is much faster than sed. The tr -s command removes double pipes in the input, which is a good idea, particularly if there can be repeated spaces or pipes in the input file. It does not cost much.

$ time < infile tr ' ' '|' | tr -s '|' '|' | cut -d '|' -f 3,5,9 > outfile-tr-cut



real    0m1,277s

user    0m1,781s

sys     0m0,925s

Oneliner with tr and cut without the tr -s command, fastest so far:

time < infile tr ' ' '|' | cut -d '|' -f 4,6,10 > outfile-tr-cut



real    0m1,199s

user    0m1,020s

sys     0m0,618s





$ tail outfile-tr-cut

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.SERV1

12-654-0330|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT81.C1.P1

12-664-1186|202-00_MSRFKH00OL6|R1.S1.LT7.PON8.ONT75.SERV1

Oneliner with awk, fast but not the fastest,

< input awk '{gsub("\|"," "); print $5 ":" $9 "," $3}' > output



$ time < infile awk '{gsub("\|"," "); print $5 ":" $9 "," $3}' > outfile.awk



real    0m5,091s

user    0m4,724s

sys     0m0,365s

Speed summary: the 'real' time according to time rounded to 1 decimal

1m 8.7s - sed

   8.1s - cut & sed

   5.9s - perl

   5.1s - awk

   1.2s - tr & cut

Finally, I note that the oneliners with sed, perl and awk create an output file with the prescribed format.

$ tail outfile.awk

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.SERV1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT81.C1.P1,12-654-0330

202-00_MSRFKH00OL6:R1.S1.LT7.PON8.ONT75.SERV1,12-664-1186