May 24, 2006

Useful AWK script

HANDY ONE-LINERS FOR AWK                                  22 July 2003
compiled by Eric Pement version 0.22
Latest version of this file is usually at:


Unix: awk '/pattern/ {print "$1"}' # standard Unix shells
DOS/Win: awk '/pattern/ {print "$1"}' # okay for DJGPP compiled
awk "/pattern/ {print \"$1\"}" # required for Mingw32

Most of my experience comes from version of GNU awk (gawk) compiled for Win32. Note in particular that DJGPP compilations permit the awk script to follow Unix quoting syntax '/like/ {"this"}'. However, the user must know that single quotes under DOS/Windows do not protect the redirection arrows (<, >) nor do they protect pipes (|). Both are special symbols for the DOS/CMD command shell and their special meaning is ignored only if they are placed within "double quotes." Likewise, DOS/Win users must remember that the percent sign (%) is used to mark DOS/Win environment variables, so it must be doubled (%%) to yield a single percent sign visible to awk.

If I am sure that a script will NOT need to be quoted in Unix, DOS, or CMD, then I normally omit the quote marks. If an example is peculiar to GNU awk, the command 'gawk' will be used. Please notify me if you find errors or new commands to add to this list (total length under 65 characters). I usually try to put the shortest script first.


# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
# NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are
# often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'

# triple space a file
awk '1;{print "\n"}'


# precede each line by its line number FOR THAT FILE (left alignment).
# Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*

# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*

# number each line of a file (number on left, right-aligned)
# Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'

# number each line of file, but only print numbers if line is not blank
# Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};{print}'
awk '{print (NF? ++a " :" :"") $0}'

# count lines (emulates "wc -l")
awk 'END{print NR}'

# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum awk
'{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value awk
'{for (i=1; i<=NF; i++) if ($i < i =" -$i;" i="1;" i =" ($i" total =" total"> max {max=$1; maxline=$0};
END{ print max, maxline}'

# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '

# print the last field of each line
awk '{ print $NF }'

# print the last field of the last line
awk '{ field = $NF }; END{ print field }'

# print every line with more than 4 fields
awk 'NF > 4'

# print every line where the value of the last field is > 4
awk '$NF > 4'


# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"");print}' # assumes EACH line ends with Ctrl-M

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r");print}

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile

# Use "tr" instead.
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
awk '{sub(/^[ \t]+/, ""); print}'

# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "");print}'

# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'
awk '{$1=$1;print}' # also removes extra space between fields

# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ");print}'

# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*

# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*

# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar");print}' # replaces only 1st instance
gawk '{$0=gensub(/foo/,"bar",4);print}' # replaces only 4th instance
awk '{gsub(/foo/,"bar");print}' # replaces ALL instances in a line

# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")};{print}'

# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")};{print}'

# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red"); print}'

# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*

# if a line ends with a backslash, append the next line to it
# (fails if there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*

# print and sort the login names of all users
awk -F ":" '{ print $1 | "sort" }' /etc/passwd

# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file

# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file

# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'

# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file

# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'

# remove duplicate, nonconsecutive lines
awk '! a[$0]++' # most concise script
awk '!($0 in a) {a[$0];print}' # most efficient script

# concatenate every 5 lines of input, using a comma separator
# between fields
awk 'ORS=%NR%5?",":"\n"' file


# print first 10 lines of file (emulates behavior of "head")
awk 'NR <>1{exit};1'

# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'

# print the last line of a file (emulates "tail -1")
awk 'END{print}'

# print only lines which match regular expression (emulates "grep")
awk '/regex/'

# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'

# print the line immediately before a regex, but not the line
# containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (x=="" ? "match on line 1" : x)};{x=$0}'

# print the line immediately after a regex, but not the line
# containing the regex
awk '/regex/{getline;print}'

# grep for AAA and BBB and CCC (in any order)
awk '/AAA/; /BBB/; /CCC/'

# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'

# print only lines of 65 characters or longer
awk 'length > 64'

# print only lines of less than 65 characters
awk 'length < nr="=" nr="=" nr="=" nr="=">

May 22, 2006



原 料:
材料: 鱿鱼,芹菜各150克,红辣椒1个,大蒜2粒,姜3片 调味料: (A)沙茶酱,酱油各1/2小匙,米酒,鸡精各1小匙。 (B)淀粉水1小匙。


Free Anti-Virus Software

Free Anti-Virus Software:
AVG Free Edition, (High recommended)
avast! Home Edition
AntiVir Personal Edition.

May 16, 2006


Some Useful VIMRC Definitions:

function! ShowFunc()
let oldmake= &makeprg
let &makeprg= 'listfunc'

silent! make%

let &makeprg=oldmake

"To enable restoring (for an xterm):
"set t_ti=^[7^[[r^[[?47h t_te=^[[?47l^[8
"(Where ^[ is an , type CTRL-V to insert it)
set t_ti=^[7^[[r^[[?47h
set t_te=^[[?47l^[8

set ignorecase
"map j_
"map k_
"set wmh=0
map ^[j j
map ^[k k

map :cn
map :cp
map :grep * -r :cw :set syntax=off k
map :on
map :w :make :cw
map :call ShowFunc()

augroup filetypedetect
au! BufRead,BufNewFile *.inc setfiletype make
augroup END

" Automatically reload .vimrc when changing
autocmd! bufwritepost .vimrc source %

The listfunc script:
ctags -x --c-types=f -u $1 | gawk '{printf("%s:%s: %s\n",$4,$3,$1) > "/dev/stderr"}'

May 13, 2006



#do not change unless you know what you are doing


postscript("drum.eps", horizontal=FALSE, onefile=FALSE,paper="special",height=10,width=14);

for (ii in levels(f)){
lines(t,i*grid+v,type="s"); text(-5,i*grid+grid/2,tones[as.numeric(ii)-tones_start+1],cex=1.0);

May 12, 2006


MIDI Format is explaine in this website.

The clock unit in the CSV file generated by the midicsv tool is in terms of PPQN Clock.

The rate PPQN ticks is calcuated as follows:

MicrosPerPPQN = MicroTempo/Division

MicroTempo=60,000,000/BPM(Beat Per Minute)

Divison is also called Timebase. It is specified in the midi file header. For hardware Midi equipments, this is a fixed value. For software generated midi files, this usually is also a application-dependent fixed value

May 11, 2006

Plot Midi Drum Line

Following the story of midicsv that converts midi to csv, I wrote some scripts to plot the drum line from an exising midi file.

First, run the midicsv tool to creat the csv file:

Second, run the awk script to filter out everything other than drum beats. The script (filter.awk)

if ($3 ~ /ote/ &&amp;amp; NF==6 && $4==9)
Now, plot the drum line in R (get it from Here is the script (midiplot.r)
for (i in levels(f)){
Above is the plot result.


This must be the first tool for midi: midicsv. It converts any midi to a human-readable CSV file, and also converts the CSV file back to midi. This is fantastic because now I can get a really-good-sound midi, convert it to midi, and see now it beats the drum. I was trying to write a midi parser based on midi2abc, but thought there must already exist something like it giving that MIDI has been around for a while.

What a delightful day!

Thanks, John Walker!


A typical Makefile with Patterns:


%.o: %.c $(HEADERS)
$(CC) $(CFLAGS) -c $<

$@: The target
$<: The name of the first prerequisite
$^: The names of all the prerequisites

Less common:

$?: The names of al lthe prerequisites that are newer than the target

AutoIt3 with DLL

Finally I got AutoIt3 to work with a DLL I wrote from C and compiled with MingW. Now I can start writing native windows applications with integration of native C code. Nice.


MMA is a great midi generator. You give it the chord, and the style (rock, folk, waltz,etc), and it generates a rich midi file for you. It is open source software with fantastic documentation and code. If you are even remotely interested in MIDI and music, you definitely should check it out.

abcm2ps with UTF-8 Support

Two days ago I got an email from the Jef, the author of abcm2ps, about adding UTF-8 support to abcm2ps. This is something I emailed him a while ago and he never got a chance to do it. Now he is planning to do it. That's great. We are starting with Chinese UTF-8 support, with help from the package called "cnprint", which produces postscript file from a text file and a TTF font file.

May 8, 2006

槐花吃法 ( Black Locust Flowers Recipe)

嫩绿色的浓荫里, 串串白色的槐花香气浓郁,味道香甜。槐花还含有多种微生素和微量元素,具有较高的营养价值。槐花有多种吃法既可加面粉蒸食、做饼,又可凉拌、炒食。

* 烙槐花

主料 槐花。最好是尚未完全开放的槐花。择净,用开水烫过,漂洗(清除槐毒)。


* 槐花鸡蛋汤
1. 汤锅上火,葱花、花椒爆锅,加水、海米、精盐、烧开。把准备好的槐花入锅。
2. 鸡蛋2个,加水三四滴打散,待锅中槐花汤烧开,立即将鸡蛋液均匀的浇入锅内,用勺推开,烧开即可。
3. 盛碗时,点大红樱桃二三个配色。