sed Stream Edit non-interactively processes an editing script of functions and copies stdin to stdout applying the functions

Stream Edit non-interactive edit

As of 12/12/10 the current version of sed is 4.2.1.
Version 4.2.1 is a minor version that includes these bug fixes:
A fix to a parsing problem.
A fix to ensure the security contexts are preserved by -i under SELinux.
A fix that ensures the temporary files for sed -i are not made group or world-readable until they are complete.
Version 4.1.5 is a minor bug fix release which fixes the handling of a negative character class when it doesn't include a closed bracket.
Most of this is based on the 3.02 version and still is applicable.

sed [-n] [-e script [-e script]] [-f sfile] [file ...]

sed copies file^† to STDOUT, editing it according to a script of commands.

Examples
sed "s/input/output/" infile > outfile sed -e "s/old/older/" -e "s/new/old" needsMultipleChanges.txt sed -f bunch-o-cmds.sed dafile otherfile > da-SEDed-file sed "s/;/;\n/" > myprog.c make multi-statment-lines be on seperate lines)

Warning: sed "s/xx/yy/" file > file results in an empty file!

note: sed "s/xx/yy/" file doesn't update file, rather outputs to STDOUT i.e terminal

-n no default output.
-excute command
-f scriptfile

Normally, sed compiles the entire script then cycles through the input :

reading a line into a WORK^† area (unless there is something left after a D ),

deletes trailing \n

applying, in sequence, all commands whose address select the WORK area , and

writing the resulting WORK area to standard output^† including a \n

A script consists of editing commands, one per line:

[address [,address] ] function [arguments]

Omitting the addresses selects every line, ( unlike ed which expects %).
s/xxx/yyy/
address:
- lineNumber, continuing across input files, starting from 1
- 1 i <table> first line ( it seems that functions that insert should be before functions that reference the entire file.
- 324 s/xxx/yyy/ A particular line by number (looks dangereous).
- . [dot] current line
- $ a </table> last line of input
- Context Address selects all lines that match a regular expression^†
  /<LI>/ all LIst tags
- AddressExpression
  .+1 s/"/'/ on the next line
range: ( two addresses select the inclusive lines)
- 12,33 s/xxx/yyy
- .,$ s/xxx/yyy/ i.e. thisLine through lastInputLine
- context address range :
  from the first line that matches the first address,
  through the next line that matches the second address.
/PRE/,/\/PRE/ s/<BR>// i.e. within ALL PRE blocks remove <BR> tags

If the second address is BEFORE the first address, only the line at the first address is selected.

In a context address, the construction
\?regular expression?
where ? is any character, is identical to
/regular expression/
So in this context address \xabc\xdefx , the second x stands for itself,
i.e. the regular expression is abcxdef.
Use a ! at the end of the address range to select all lines except those that match.

A function includes substitute, append, change, delete, and others
Some of the functions use a HOLD area to save all or part of the WORK area for subsequent retrieval.

arguments depend on the function

* Function Description
2 s/regexp/replacement/f
The character immediating following the s
Substitute the replacement string for instances of the regexp in WORK.
Another character may be used instead of /
df | sed "s?/Volumes/??"
For a fuller description see ed(1).
flags are:
g Global. Substitute all nonoverlapping instances of regexp.
n = 1 - 512. Substitute for just the nth occurrence of the regexp.
n.b. using 2 for flags changes
the second occurance IN THE RANGE OF LINES,
not the second occurance in every line.
i.e. only one string will be changed unless you use g
p Print WORK if a replacement was made.
w file Write the WORK area, appending to file if a replacement was made.
& in the replacment signifies the matched regexp
Example: s/<td/& align=right/ changes <td> to <td align=right>
2 n W → STDOUT
in → W next. Output WORK ; read the next line of input to WORK.
2 N W + \n + in → W Append a new-line and the Next line of input to WORK.
Resulting in an embedded new-line, that can be substituted by null, joining the current WORK and the next input line.
Example: Join continuation of a span tag had been split onto next line
/<span$/N; s/<span\n/<span/
2 G _{W +} ^H → _W WORK Gets [newline] and HOLD appended .
2 h _W → ^H HOLD gets contents of WORK .
2 H ^H+_W → ^H Append WORK to HOLD .
2 g ^H → _W contents of the HOLD get put to WORK
2 x W ← X → H Exchange the contents of WORK and HOLD .
2 P Print. Copy WORK to the standard output.
(useful if -n option was used to supress normal output.)
2 p print. Copy the initial segment of the WORK area through the first new-line to the standard output.
1 a text append text after outputting the selcted line.
If the text is several lines, all (but the last) must end with \ .
After the head tag, append a title tag, "nice document" and the ending title tag:
/<head>/a <title>\ nice document\ <title>

1 i text insert text before outputting the selected line.
'</body>'i <style>\ td {text-align:right}\ </style>

2 d delete WORK. Start the next cycle.
2 c\ text change. Delete WORK. Output text. Start the next cycle.
2 D Delete the initial segment of WORK through the first new-line. Start the next cycle.
1 rfile read file and output it.
file must be the last thing on the line and be preceeded by exactly one blank.
1 = output the current line number of input.
(good for debugging) (The current line number changes.)

2 b label Branch to the command bearing the :label.
If label is empty, branch to the end of the script.
2 t label test. Branch to the command bearing the :label if any substitutions have been made since the most recent reading of an input line or execution of a t.
If label is empty, branch to the end of the script.
0 :label define a label for b and t commands to branch to.
1 q quit.

1 wfile write. Append WORK to file .
file must be the last thing on the line and be preceded by exactly one blank.
The first occurrence of w causes file to be cleared.
Each time sed is used, file is overwritten.

Each w file is created before processing begins.
There can be at most 10 distinct w file arguments.

2 { Execute the following commands through a matching } only when WORK is selected.
2 ! function Don't apply the function (or group, if function is {) only to lines not selected by the address(es).
# comment On the first line #n, the default output will be suppressed.
The rest of the line after #n is ignored.
2 y/string1/string2/ Transform. Replace all occurrences of characters in string1 with the corresponding characters in string2.
string1 and string2 must have the same number of characters.
example:
to change all uppercase to lowercase

y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/

2 l (ell) output WORK area in an unambiguous form.
Non-printable^† characters are displayed in octal notation and
long lines are folded.
consider the strings command.
example:
create a file with control V, C, and M using:
ex -c p funnyfile ---^V,,,^C+++^M===
now see the control characters using sed:
sed "l" funnyfile ---\026,,,\003+++\r===\n$
\026 is octal ^V
\003 is octal ^C
\r is the ^M (return character)
\n is the linefeed
$ is the

Example:
change
Dec 31 19:37
to
19:37 Dec 31

command results
s/... .. ..:../& &/ duplicates the entire "date time" string
Dec 31 19:37 Dec 31 19:37
s/... ..// deletes the first Dec 31 19:37 Dec 31 19:37
s/..:..//2 deletes the 2^nd 19:37 19:37 Dec 31
(I expect you could use the X command)

command		results
`s/... .. ..:../& &/`	duplicates the entire "date time" string	`Dec 31 19:37 Dec 31 19:37`
`s/... ..//`	deletes the first `Dec 31`	`19:37 Dec 31 19:37`
`s/..:..//2`	deletes the 2^nd `19:37`	`19:37 Dec 31`

Change 123,456 to 123456
substitute from 1 to 3 characters followed by a comma to the same 1 to 3 characters comma and a zz substitute ,zz to null
s/.\{1,3\},/&zz/ ; s/,zz//
The new-line escape sequence, \n, matches a new-line embedded in WORK.

s/\n>/>/ joins a line with a leading > to the previous by omitting the new-line.

A period . matches any character except the terminal new-line of WORK.
Use the negation function ! to have commands applied only to non-selected WORK areas .
Backslashes in text are treated like backslashes in the replacement string of an s command, and may be used to protect initial blanks and tabs against the stripping that is done on every script line.

command list example:

s/xxx/yyy/ { n b top }

Not all versions of sed will handle [tab]s as \t Pre process files TRanslating tabs to spaces;

tr '\t' ' ' < $1 |sed -f ~/xml.sed

or y/tab/ /

On single-line patterns, . (dot) matches character.
It does not match the [NewLine] at the end of the line because the newline is removed when the line is put into the work space.
sed adds a newline when the work space is output.
On multi-line patterns obtained with N or G , dot matchs a newline in the middle of the pattern space.
If there are 3 lines in the pattern space, s/.*// deletes all 3 lines.

Example using of a "bracketed expression" character classes s/[[:digit:]]/N/

see awk, ed, grep(includes regular expression discussion), environ

9 May 10, copy pasted list from a PDF and ended up with x'EF 82 B7 ' . Only way I could get them out was to copy the file and delete everything except the x'EF 82 B7 ' then prefix it with s/ and suffix it with /zzz/g

Gotta be a better way

Modified to Real HTML by Dennis German 12 Mar 2002 see also sed-3.02, see also sedFAQ.html, sed-gnu

Some information taken from SunOS 5.4 man page of 14 Sep 1992