UNIX : Sed command & useful examples

SYNTAX  :   sed [options] '{command}' [filename]

sed is very useful stream editor that accepts a series of commands and executes them on a file (or set of files) non-interactively.

How sed Works

The sed utility works by sequentially reading a file, line by line, into memory. It then performs all actions specified for the line and places the line back in memory to dump to the terminal with the requested changes made.

After all actions have taken place to this one line, it reads the next line of the file and repeats the process until it is finished with the file.

As mentioned, the default output is to display the contents of each line on the screen.

Two important factors

1. The output can be redirected to another file to save the changes;

2.  The original file, by default, is left unchanged.

Few basic & Advanced examples of sed :

1) Substitute command 's/{old value}/{new value}/'

i] Single change (changes dba with admin )

echo "peoplesoft dba" | sed 's/dba/admin'

ii] Multiple changes

type 1 :

$ echo I’ll go to gym after cricket match  | sed -e 's/gym/office/' -e 's/after/before/'

type 2:

$ echo I’ll go to gym after cricket match | sed 's/gym/office/; s/after/before/'

type 3:

$ echo I’ll go to gym after cricket match | sed '

> s/gym/office/

> s/after/before/'

iii] For global changes throughout file through all lines just append g at the end of command 's/after/before/g'

iv] replace tab by spaces

sed 's/    / /g'

2) Replacement with particular patterns

i]this will replace only in lines containing string "wow"

$ sed '/wow/ s/1/2/' sample_one

ii] will replace 1 by 2 in lines having "abc" and 1 by 3 with lines containing "xyz"

$ sed '

> /abc/ s/1/2/

> /xyz/ s/1/3/' sample_one

iii] using script file as sed options

$ cat sedlist

/abc/ s/1/2/

/xyz/ s/1/3/

and use that file as

$ sed -f sedlist sample_one

iv] substitute "html" with "xml" only in the fifth and sixth line

$ sed '5,6 s/html/xml/' sample_one

v] prohibit display(verbose) use -n

sed -n -f sedlist sample_one

vi] display 2nd to 6th lines

sed -n '2,6p' sample_one

3) deleting lines with syntax-> '{what to find} d'

i] To remove lines containing "man"

sed '/man/ d' sample_one

ii] delete first 3 lines

sed '1,3 d' sample_one

iii] delete the line if "sta" were the first three characters of the line

sed '/^sta/ d' sample_one

iv] delete the line only if "pky" were the last three characters of the line

sed '/pky$/ d' sample_one

v) delete all blank lines from file

sed '/^$/ d' {filename}

iv] delete first line throught to the first blank line

sed '1,/^$/ d' {filename}

vii] combined command 

sed '/man/ s/1/2/; /sta/ s/1/3/; /^$/ d' sample_one

4) Huge files sized in GBs cant be opened in vi editor. Here is trick to get desired log context from file.

i.    grep –in “JavaException” hugefile

      will return JavaException matching line with line number.

ii    Now reduce the filelength by getting few lines before and after JavaException line number as

      sed –n ‘no1,no2p’ hugefile < newtempfile      ( no1 & no2 are starting & ending numbers )

      Now you can open newtempfile and troubleshoot errors backward & forward .

5) To comment lines from 4th to 22nd in a script and uncomment lines from 26th to 34th

sed -e '4,22 s/^/#/' -e '26,34 s/^#//' abc.sh > newfile

6) You want to read script but dont want unneccessary comments

sed -e '/^#/d' nirajScript.ksh | more

7) To display only clients status for App server & removing headers & any blank lines from output.

psadmin -c cstatus –d FSDEV 2> /dev/null | sed -e '1,3 d' -e '/^$/ d' -e '$ d'

Use of Next parameter in sed

The Next command is used for multiline pattern space manipulation.

it allows you to juggle text that you wish to control over multiple lines.

Where the next command outputs the contents of the pattern space then reads the next line...

the Next command reads the contents of the next line then appends it to the pattern space

separating first line from the second with a "\n" character.

8) To delete two consecutive empty lines in a file

sed  ‘/^$/{

N

/^\n$/D 

}’ datafile123

9) To replace a path in a file from  /home/psoft/output/ to /var/spool/report/

As / is special shell character it has to be escaped either by ‘\’ or we can use @ here instead of ‘/’

Type 1:

sed ‘s@/home/psoft/output/@/var/spool/report/@g’ datafile123

Type 2:

sed ‘s/\/home\/psoft\/output\//\/var\/spool\/report\//g’ datafile123

10)sed & Usage: Substitute /home/psoft/ to /home/psoft/log

sed 's@/home/psoft@&/log@g' file123.txt

11)sed & usage : Match the whole line 

sed 's@^.*$@<<<&>>>@g' path.txt

In the above example regexp has “^.*$” which matches the whole line. Replacement part <<<&>>> writes the whole line with <<< and >>> in the beginning and end of the line respectively.

Regular expressions behave slight differently in sed, awk,grep, shell.
So here is how they can be used in sed

Character	Description
^	Matches the beginning of the line
$	Matches the end of the line
.	Matches any single character
*	Will match zero or more occurrences of the previous character
[ ]	Matches all the characters inside the [ ]

Examples:

Reg Expression	Description
/./	Will match any line that contains at least one character
/../	Will match any line that contains at least two characters
/^#/	Will match any line that begins with a '#'
/^$/	Will match all blank lines
/}$/	Will match any lines that ends with '}' (no spaces)
/} *$/	Will match any line ending with '}' followed by zero or more spaces
/[abc]/	Will match any line that contains a lowercase 'a', 'b', or 'c'
/^[abc]/	Will match any line that begins with an 'a', 'b', or 'c'