December 19, 2006

Bind: Address Already in Use

Bind: Address Already in Use

Or How to Avoid this Error when Closing TCP Connections

Normal Closure

In order for a network connection to close, both ends have to send FIN (final) packets, which indicate they will not send any additional data, and both ends must ACK (acknowledge) each other's FIN packets. The FIN packets are initiated by the application performing a close(), a shutdown(), or an exit(). The ACKs are handled by the kernel after the close() has completed. Because of this, it is possible for the process to complete before the kernel has released the associated network resource, and this port cannot be bound to another process until the kernel has decided that it is done.

TCP State Diagram
Figure 1

Figure 1 shows all of the possible states that can occur during a normal closure, depending on the order in which things happen. Note that if you initiate closure, there is a TIME_WAIT state that is absent from the other side. This TIME_WAIT is necessary in case the ACK you sent wasn't received, or in case spurious packets show up for other reasons. I'm really not sure why this state isn't necessary on the other side, when the remote end initiates closure, but this is definitely the case. TIME_WAIT is the state that typically ties up the port for several minutes after the process has completed. The length of the associated timeout varies on different operating systems, and may be dynamic on some operating systems, however typical values are in the range of one to four minutes.

If both ends send a FIN before either end receives it, both ends will have to go through TIME_WAIT.

Normal Closure of Listen Sockets

A socket which is listening for connections can be closed immediately if there are no connections pending, and the state proceeds directly to CLOSED. If connections are pending however, FIN_WAIT_1 is entered, and a TIME_WAIT is inevitable.

Note that it is impossible to completely guarantee a clean closure here. While you can check the connections using a select() call before closure, a tiny but real possibility exists that a connection could arrive after the select() but before the close().

Abnormal Closure

If the remote application dies unexpectedly while the connection is established, the local end will have to initiate closure. In this case TIME_WAIT is unavoidable. If the remote end disappears due to a network failure, or the remote machine reboots (both are rare), the local port will be tied up until each state times out. Worse, some older operating do not implement a timeout for FIN_WAIT_2, and it is possible to get stuck there forever, in which case restarting your server could require a reboot.

If the local application dies while a connection is active, the port will be tied up in TIME_WAIT. This is also true if the application dies while a connection is pending.

Strategies for Avoidance

SO_REUSEADDR

You can use setsockopt() to set the SO_REUSEADDR socket option, which explicitly allows a process to bind to a port which remains in TIME_WAIT (it still only allows a single process to be bound to that port). This is the both the simplest and the most effective option for reducing the "address already in use" error.

Oddly, using SO_REUSEADDR can actually lead to more difficult "address already in use" errors. SO_REUSADDR permits you to use a port that is stuck in TIME_WAIT, but you still can not use that port to establish a connection to the last place it connected to. What? Suppose I pick local port 1010, and connect to foobar.com port 300, and then close locally, leaving that port in TIME_WAIT. I can reuse local port 1010 right away to connect to anywhere except for foobar.com port 300.

A situation where this might be a problem is if my program is trying to find a reserved local port (<>SO_REUSADDR, then each time I run the program on my machine, I'll keep getting the same local reserved port, even if it is stuck in TIME_WAIT, and I risk getting a "connect: Address already in use" error if I go back to any place I've been to in the last few minutes. The solution here is to avoid SO_REUSEADDR.

Some folks don't like SO_REUSEADDR because it has a security stigma attached to it. On some operating systems it allows the same port to be used with a different address on the same machine by different processes at the same time. This is a problem because most servers bind to the port, but they don't bind to a specific address, instead they use INADDR_ANY (this is why things show up in netstat output as *.8080). So if the server is bound to *.8080, another malicious user on the local machine can bind to local-machine.8080, which will intercept all of your connections since it is more specific. This is only a problem on multi-user machines that don't have restricted logins, it is NOT a vulnerability from outside the machine. And it is easily avoided by binding your server to the machine's address.

Additionally, others don't like that a busy server may have hundreds or thousands of these TIME_WAIT sockets stacking up and using kernel resources. For these reasons, there's another option for avoiding this problem.

Client Closes First

Looking at the diagram above, it is clear that TIME_WAIT can be avoided if the remote end initiates the closure. So the server can avoid problems by letting the client close first. The application protocol must be designed so that the client knows when to close. The server can safely close in response to an EOF from the client, however it will also need to set a timeout when it is expecting an EOF in case the client has left the network ungracefully. In many cases simply waiting a few seconds before the server closes will be adequate.

It probably makes more sense to call this method "Remote Closes First", because otherwise it depends on what you are calling the client and the server. If you are developing some system where a cluster of client programs sit on one machine and contact a variety of different servers, then you would want to foist the responsibility for closure onto the servers, to protect the resources on the client.

For example, I wrote a script that uses rsh to contact all of the machines on our network, and it does it in parallel, keeping some number of connections open at all times. rsh source ports are arbitrary available ports less than 1024. I initially used "rsh -n", which it turns out causes the local end to close first. After a few tests, every single free port less than 1024 was stuck in TIME_WAIT and I couldn't proceed. Removing the "-n" option causes the remote (server) end to close first (understanding why is left as an exercise for the reader), and should've eliminated the TIME_WAIT problem. However, without the -n, rsh can hang waiting for input. And, if you close input at the local end, this can again result in the port going into TIME_WAIT. I ended up avoiding the system-installed rsh program, and developing my own implementation in perl. My current implementation, multi-rsh, is available for download

Reduce Timeout

If (for whatever reason) neither of these options works for you, it may also be possible to shorten the timeout associated with TIME_WAIT. Whether this is possible and how it should be accomplished depends on the operating system you are using. Also, making this timeout too short could have negative side-effects, particularly in lossy or congested networks.

In Linux, issue the following command to set the timeout_timewait parameter to 30 seconds:
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

From: http://hea-www.harvard.edu/~fine/Tech/addrinuse.html

November 13, 2006

Search Engine-Friendly URLs

Search Engine-Friendly URLs

By Chris Beasley
August 10th 2001

On today’s Internet, database driven or dynamic sites are very popular. Unfortunately the easiest way to pass information between your pages is with a query string. In case you don’t know what a query string is, it's a string of information tacked onto the end of a URL after a question mark.

So, what’s the problem with that? Well, most search engines (with a few exceptions - namely Google) will not index any pages that have a question mark or other character (like an ampersand or equals sign) in the URL. So all of those popular dynamic sites out there aren’t being indexed - and what good is a site if no one can find it?

The solution? Search engine friendly URLs. There are a few popular ways to pass information to your pages without the use of a query string, so that search engines will still index those individual pages. I'll cover 3 of these techniques in this article. All 3 work in PHP [1] with Apache [2] on Linux [3] (and while they may work in other scenarios, I cannot confirm that they do).

Method 1: PATH_INFO

Implementation:

If you look above this article on the address bar, you’ll see a URL like this: http://www.webmasterbase.com/article.php/999/12. SitePoint actually uses the PATH_INFO method to create their dynamic pages.

Apache has a "look back" feature that scans backwards down the URL if it doesn’t find what it's looking for. In this case there is no directory or file called "12", so it looks for "999". But it find that there's not a directory or file called "999" either, so Apache continues to look down the URL and sees "article.php". This file does exist, so Apache calls up that script. Apache also has a global variable called $PATH_INFO that is created on every HTTP request. What this variable contains is the script that's being called, and everything to the right of that information in the URL. So in the example we've been using, $PATH_INFO will contain article.php/999/12.

So, you wonder, how do I query my database using article.php/999/12? First you have to split this into variables you can use. And you can do that using PHP’s explode function:

$var_array = explode("/",$PATH_INFO);

Once you do that, you’ll have the following information:

$var_array[0] = "article.php"

$var_array[1] = 999

$var_array[2] = 12

So you can rename $var_array[1] as $article and $var_array[2] as $page_num and query your database.

Drawback:

There was previously one major drawback to this method. Google, and perhaps other search engines, would not index pages set up in this manner, as they interpreted the URL as being malformed. I contacted a Software Developer at Google and made them aware of the problem and I am happy to announce that it is now fixed.

There is the potential that other search engines may ignore pages set up in this manner. While I don't know of any, I can't be certain that none do. If you do decide to use this method, be sure to monitor your server logs for spiders to ensure that your site is being indexed as it should.

Method 2: .htaccess Error Pages

Implementation:

The second method involves using the .htaccess file. If you're new to it, .htaccess is a file used to administer Apache access options for whichever directory you place it in. The server administrator has a better method of doing this using his or her configuration files, but since most of us don't own our own server, we don't have control over what the server administrator does. Now, the server admin can configure what users can do with their .htaccess file so this approach may not work on your particular server, however in most cases it will. If it doesn't, you should contact your server administrator.

This method takes advantage of .htaccess’ ability to do error handling. In the .htaccess file in whichever directory you wish to apply this method to, simply insert the following line:

ErrorDocument 404 /processor.php

Now make a script called processor.php and put it in that same directory, and you're done! Lets say you have the following URL: http://www.domain.com/directory/999/12/. And again in this example "999" and "12" do not exist, however, as you don't specify a script anywhere in the directory path, Apache will create a 404 error. Instead of sending a generic 404 header back to the browser, Apache sees the ErrorDocument command in the .htaccess file and calls up processor.php.

Now, in the first example we used the $PATH_INFO variable, but that won’t work this time. Instead we need to use the $REQUEST_URI variable, which contains everything in the URL after the domain. So in this case, it contains: /directory/999/12/.

The first thing you need to do in processor.php is send a new HTTP header. Remember, Apache thought this was a 404 error, so it wants to tell the browser that it couldn’t find a page.

So, put the following line in your processor.php:

header("HTTP/1.1 200 OK");

At this time I need to point out an important fact. In the first example you could specify what script processed your URL. In this example all URLs must be processed by the same script, processor.php, which makes things a little different. Instead of creating different URLs based on what you want to do, such as article.php/999/12 or printarticle.php/999/12 you only have 1 script that must do both.

So you must decide what to do based on the information processor.php receives - more specifically, by counting how many parameters are passed. For instance on my site [4], I use this method to generate my pages: I know that if there's just one parameter, such as http://www.online-literature.com/shakespeare/, that I need to load an author information page; if there are 2 parameters, such as http://www.online-literature.com/shakespeare/hamlet/, I know that I need to load a book information page; and finally if there are 3 parameters, such as http://www.online-literature.com/shakespeare/hamlet/3/, I know I need to load a chapter viewing page. Alternatively, you can simply use the first parameter to indicate the type of page to display, and then process the remaining parameters based on that.

There are 2 ways you can accomplish this task of counting parameters. First you need to use PHP’s explode function to divide up the $REQUEST_URI variable. So if $REQUEST_URI = /shakespeare/hamlet/3/:

$var_array = explode("/",$REQUEST_URI);

Now note that, because of the positioning of the /’ there are actually 5 elements in this array [5]. The first element, element 0, is blank, because it contains the information before the first /. The fifth element, element 4, is also blank, because it contains the information after the last /.

So now we need to count the elements in our $var_array. PHP has two functions that let us do this. We can use the sizeof() function as in this example:

$num = sizeof($var_array); // 5

You’ll notice that the sizeof() function counts every item in the array regardless of whether it's empty. The other function is count(), which is an alias for the sizeof() function.

Some search engines, like AOL, will automatically remove the trailing / from your URL, and this can cause problems if you’re using these functions to count your array. For instance http://www.online-literature.com/shakespeare/hamlet/ becomes http://www.online-literature.com/shakespeare/hamlet, and as there are 3 total elements in that array our processor.php would load an author page instead of a book page.

The solution is to create a function that will count only the elements in an array that actually hold data. This will allow you to leave off the ending / or allow any links from AOL'’s search engine to point to the proper place. An example of such a function is:

function count_all($arg)
{
// skip if argument is empty
if ($arg) {
// not an array, return 1 (base case)
if(!is_array($arg))
return 1;
// else call recursively for all elements $arg
foreach($arg as $key => $val)
$count += count_all($val);
return $count;
}
}

To get your count, access the function like this:

$num = count_all($url_array);

Once you know how many parameters you need, you can define them like this:

$author=$var_array[1];
$book=$var_array[2];
$chapter=$var_array[3];

Then you can use includes to call up the appropriate script, which will query your database and set up your page. Also if you get a result you’re not expecting, you can simply create your own error page for display to the browser.

Drawback:

The drawback of this method is that every page that's hit is seen by Apache as an error. Thus every hit creates another entry in your server error logs, which effectively destroys their usefulness. So if you use this method you sacrifice your error logs.

Method 3: The ForceType Directive

Implementation:

You'll recall that the thing that trips up Google, and maybe even other search engines, when using the PATH_INFO method is the period in the middle of the URL. So what if there was a way to use that method without the period? Guess what? There is! Its achieved using Apache'’s ForceType directive.

The ForceType directive allows you to override any default MIME types you have set up. Usually it may be used to parse an HTML [6] page as PHP or something similar, but in this case we will use it to parse a file with no extension as PHP.

So instead of using article.php, as we did in method 1, rename that file to just "article". You will then be able to access it like this: http://www.domain.com/article/999/12/, utilizing Apache's look back feature and PATH_INFO variable as described in method 1. But now, Apache doesn’t know to that "article" needs to be parsed as php. To tell it that, you must add the following to your .htaccess file.


ForceType application/x-httpd-php

This is known as a "container". Instead of applying directives to all files, Apache allows you to limit them by filename, location, or directory. You need to create a container as above and place the directives inside it. In this case we use a file container, we identify “article” as the file we're concerned with, and then we list the directives we want applied to this file before closing off the container.

By placing the directive inside the container, we tell Apache to parse "article" as a PHP script even though it has no file extension. This allows us to get rid of the period in the URL that causes the problems, and yet still use the PATH_INFO method to manage our site.

Drawback:

The only drawback to this method as compared with method 2 is that your URLs will be slightly longer. For instance, if I were to use this method on my site, I'd have to use URLs like this: http://www.online-literature.com/ol/homer/odyssey/ instead of http://www.online-literature.com/homer/odyssey/. However if you had a site like SitePoint and used this method it wouldn't be such a problem, as the URL (http://www.SitePoint.com/article/755/12/) would make more sense.

Conclusion

I have outlined 3 methods of making search engine friendly URLs - along with their drawbacks. Obviously, you should evaluate these drawbacks before deciding which method to implement. And if you have any questions about the implementation of these techniques, they are oft-discussed topics on the SitePoint Forums [7] so just stop in and make a post.

November 3, 2006

Use Wget to find site broken links

wget --recursive -nd -nv --delete-after --domains=domain.com http://domain.com/ | tee wget.out 2>&1
Got it from: http://www.shallowsky.com/blog/tech/web/webstats.html

October 2, 2006

FreeCall to phones

This Service seems to be good. Needs to try it out.

September 16, 2006

菜品走味怎么办

放有辣椒的菜太辣时或炒辣椒时加点醋,辣味大减。

烹调时,放酱油若错倒了食醋,可撒放少许小苏打,醋味即可消除。

菜太酸,将一只松花蛋捣烂放入。

菜太辣,放一只鸡蛋同炒。

菜太苦,滴入少许白醋。

汤太咸又不宜兑水时,可放几块豆腐或土豆或几片蕃茄到汤中;也可将一把米或面粉用布包起来放入汤中。

汤太腻,将少量紫菜在火上烤一下,然后撒入汤中。

The BusinessWeek Best-Sellers of 2003

The top-selling business books as reported January through December, 2003

Paperback Business Books


1. FAST FOOD NATION By Eric Schlosser (HarperCollins -- $13.95) The bad news on burgers and fries, by an Atlantic Monthly writer.

2. NICKEL AND DIMED By Barbara Ehrenreich (Owl Books -- $13) How the working poor struggle to make ends meet.

3. THE TIPPING POINT By Malcolm Gladwell (Back Bay -- $14.95) What turns an idea into a hot trend, by a New Yorker writer.

4. THE E-MYTH REVISITED By Michael E. Gerber (HarperBusiness -- $16) A systems approach for small-business success.

5. WHAT COLOR IS YOUR PARACHUTE? By Richard Nelson Bolles (Ten Speed Press -- $17.95) The 2003 edition of the enduring job-search bible.

6. REAL ESTATE RICHES By Dolf de Roos (Warner -- $17.95) Why buildings are better investments than stocks.

7. REAL ESTATE LOOPHOLES By Diane Kennedy, C.P.A., and Garrett Sutton, Esq. (Warner -- $16.95) Tax and legal knowhow.

8. THE RICHEST MAN IN BABYLON By George S. Clason (Signet -- $6.99) Save 10% of what you earn, then pay your bills.

9. RICH DAD'S RETIRE YOUNG, RETIRE RICH By Robert T. Kiyosaki, with Sharon L. Lechter, C.P.A. (Warner -- $17.95) Plan ahead.

10. THE ERNST & YOUNG TAX GUIDE 2003 By Ernst & Young, LLP (Wiley -- $16.95) Dig out those W2s and 1099s.

11. EFFECTIVE PHRASES FOR PERFORMANCE APPRAISALS By James E. Neal Jr. (Neal Publications -- $10.95) How about: ``attaboy''?

12. THE INTELLIGENT INVESTOR, REVISED EDITION By Benjamin Graham, with Jason Zweig (HarperBusiness -- $19.95) The classic explanation of ``value investing.''

13. J.K. LASSER'S YOUR INCOME TAX 2003 By the J.K. Lasser Institute (Wiley -- $16.95) Home-office deductions, the alternative minimum tax, and other traps for the unwary.

14. IT'S NOT HOW GOOD YOU ARE, IT'S HOW GOOD YOU WANT TO BE By Paul Arden (Phaidon -- $7.95) An ad man's aphorisms on success.

15. THE POWER OF FOCUS By Jack Canfield, Mark Victor Hansen, and Les Hewitt (Health Communications -- $12.95) Zoning in on achievement.

Hardcover Business Books


1. GOOD TO GREAT By Jim Collins (HarperBusiness -- $27.50) How run-of-the-mill companies make the leap to excellence.

2. EXECUTION By Larry Bossidy and Ram Charan (Crown Business -- $27.50) Translating business strategies into results.

3. WHAT SHOULD I DO WITH MY LIFE? By Po Bronson (Random House -- $24.95) Overcoming confusion to find your true calling.

4. THE GREAT UNRAVELING By Paul Krugman (Norton -- $25.95) A Princeton University economist takes on the Bush Administration.

5. LEADERSHIP By Rudolph W. Giuliani (Talk Miramax -- $25.95) Hizzoner speaks.

6. THE PRESENT By Spencer Johnson (Doubleday -- $19.95) The pursuit of happiness and success, as described in a fable.

7. THE ONE MINUTE MILLIONAIRE By Mark Victor Hansen and Robert G. Allen (Harmony Books -- $21) Chicken soup for the investor.

8. REEFER MADNESS By Eric Schlosser (Houghton Mifflin -- $23) The underground economy in porn, pot, and migrant labor.

9. THE FIVE DYSFUNCTIONS OF A TEAM By Patrick Lencioni (Jossey-Bass -- $22) Overcoming behavior that obstructs teamwork.

10. TOTAL MONEY MAKEOVER By Dave Ramsey (Thomas Nelson -- $24.99) Getting rid of debt and building up your reserves.

11. THE POWER OF FULL ENGAGEMENT By Jim Loehr and Tony Schwartz (Free Press -- $26) Apply the rigors of the gym to your work and personal life.

12. THE LAWS OF MONEY, THE LESSONS OF LIFE By Suze Orman (Free Press -- $26) More maxims for prospering, from the queen of personal finance.

13. IN AN UNCERTAIN WORLD By Robert E. Rubin and Jacob Weisberg (Random House -- $35) The former Treasury Secretary tells why Clintonomics worked.

14. PURPLE COW By Seth Godin (Portfolio -- $19.95) Emulate Krispy Kreme and Dutch Boy paints, and astonish your customers.

15. RE-IMAGINE! By Tom Peters (DK Publishing -- $30) ``Think beautiful...think weird'' and other instructions, in an eye-popping format.

BusinessWeek's Best-Seller List is based on a survey of chain and independent booksellers that carry a broad selection of books on economics, management, sales and marketing, small business, investing, personal finance, and careers. Well over 1,000 retail outlets nationwide are represented. Current rankings are based on a weighted analysis of unit sales.

August 20, 2006

腌黄瓜

独门秘技腌黄瓜——酸,甜,辣,爽!

偶经常去前门的都一处烧麦馆,到不是贪恋里边的烧麦,而是钟情于里边的酸辣黄瓜条,自认为虽然价格贵了点(五快钱一小碟,直径不超过十厘米的那种碟子,而 且也只是码放着寥寥数根而已),但是味道超级爽口,舔中带酸,还伴着淡淡的辣味,冰冰凉凉清爽的不得了,在别的饭店从未吃到过。于是今天就琢磨着在家里自 己做了一盘,没想到效果非常好,和店里的味道一模一样,晚餐的时候摆上桌得到了老公的强烈支持!

现在就把方法介绍给大家,有兴趣的朋友不妨一试。

主料:鲜嫩黄瓜两至三根(一定要顶花带刺的做出来才爽口哦)

调料:A.盐一勺

B.白醋五勺,冰糖一块(核桃大小),蜂蜜一勺,味精小半勺,干红辣椒四个(掰成小段)

制作方法:1,将黄瓜洗净,切成大约八厘米长手指头粗细的小段,放在干净容器内用盐搅拌均匀,待用。

2,将B里面的材料一起放进一个小碗里边,搅拌均匀,并放置到冰糖完全溶解。

3,黄瓜用盐腌二十分钟左右会渗出很多水分,把这些水分倒掉,并将刚才调好的B调味料到入黄瓜中搅拌均匀。

4,放置冰箱中冷藏一天,即可。

特点:酸中带甜,辣味适中,冰爽可口,开胃消食。

需要注意的几点:1,一定要用冰糖和白醋,不要用白糖和米醋/陈醋,否则作不出那种清爽的味道。

2,用盐腌黄瓜的时候,盐的用量切勿过多,否则会影响最后清甜的口感。

3,此道凉菜放置在冰箱中腌制,所以特别适合夏天食用,根据我在都一处的经验,在保证卫生的情况下腌至的时间如果超过两天效果会更好。如果只腌一天,也已经很不错了。

4,和别的凉菜不同的是,这道凉菜不需要放香油,以强调不带一滴油脂的清爽感。


August 9, 2006

GVIM 7.0 and netrw

Recently I installed gvim7.0 on my windows XP and I am happy with the new spell check feature. However, I can not remotely write to a ftp file anymore by using "e ftp://www.abc.com/file.txt". I used to be able to do that with gvim6.3. The issues lies in the netrw vim plugin.

To solve this problem, I had to get to the netrw.vim plugin webpage (http://www.vim.org/scripts/script.php?script_id=1075) and install the latest version (version 102).

To install it, you have to gunzip it first. This is insane because gunzip is not native to Windows. Although I do have it on my machine through unixutils, I don't think many other Windows XPs have it.

Anyway, after installing the latest version, the problem seems to be fixed. I miss the old good days that vim (and gvim) just works out of box.

August 1, 2006

mail-to-blog for wordpress

http://www.economysizegeek.com/?page_id=395 has a good plugin (named postie) for wordpress to support mail-to-blog, supporting picture and attachment.

June 29, 2006

Pregnancy

My wife might be pregnant yesterday. At least according to the testing tick we bought. How wonderful that is! We are being cautious and will wait for a few more days to confirm it before announcing it. :-) Sweet. Shu......, Keept it a serect!

iLBC Codec

iLBC is the Best Codec over IP network.

It is the codec used by Skype and many other IM clients. It is open, FREE, and a standard.

Find more about it, incuding the the source code of a reference implementation at:

http://www.ilbcfreeware.org/index.html

June 28, 2006

CSS, List and Float

Very nice tutorial and examples on CSS, List and Float
http://css.maxdesign.com.au/floatutorial/

June 27, 2006

SNMP On Linux

To Add a mib file to the tool, do the following:
       
mkdir $HOME/.snmp
mkdir $HOME/.snmp/mibs
cp MY-MIB.txt $HOME/.snmp/mibs

And Then Add the following lines to the file ~/.snmp/snmp.conf

 mibs +MY-MIB 


Note MY-MIB here is the MIB Module name, not the Mib file name.

You can use the following command to list your new mib

 snmptranslate -Tp -IR ModuleName

June 26, 2006

Fix for VIM7 HTML Syntax File

A minor bug in the default VIM7 html syntax file is that spell check is not applied to paragraphs. Adding the following line to line 157 fixes that:


Write using HTML and CSS

Using HTML and CSS to write a book or an article
http://www.alistapart.com/articles/boom

June 22, 2006

Power Talk

Watched a video seminar on how to communicate better by George Walther.

I found the summary sheeet on his website very useful:
http://www.georgewalther.com/images/50WaystoSayWhatYouMean.pdf

June 12, 2006

June 10, 2006

Remebering Key Signature

  • No sharps or flats is C major
  • One flat is F major
  • For more than one flat, the (major) key is the next-to-last flat.
  • For any number of sharps, take the last sharp and go up one semitone to get the (major) key.
More at http://en.wikipedia.org/wiki/Key_signature

怎样为一首歌曲配和声?

作者:淦立康 来源:中国原创音乐联盟

我们在给一首歌配和弦时,应先分析一下歌曲的风格、调式、调性,这样我们才能够把握歌曲的总体。一般说来,如果歌曲为大调歌曲,我们应该在歌曲的第一和最后一小节配上I级和弦也就是主和弦,而在歌曲的倒数第二小节配属(七)和弦,倒数第三小节配下属和弦。但这只是一般规律,具体配什么和弦还得按组成音来定。另外,就是关于和弦密度的问题,到底一小节配多少和弦才算合适呢?一般说来,它主要取决于歌曲的速度,较快的歌曲和弦变换不要过快,这样容易使歌曲变浑浊;相反,速度较慢的歌曲应把和弦配的活跃一些,这样使别人感觉有新意。当然,这也是一般性规律,具体问题还得具体分析。

那么,怎么配法呢?下面有些规律供参考。

要以强拍上的音为先,兼顾弱拍。所谓强拍是指小节中的第一拍(次强拍也算)弱拍上的音可做为和弦外音不与考虑。

以整小节中出现最多的音为准。

要考虑所选和弦的功能性。

然而一直按照既定的主旋律编配,是很被动的。我们知道,其实有很多和弦进行的模式,俗称“套子”。例如:经典的I-VI-IV-V进行,很多人在利用这个进行已经创作了不少曲子。很多优美的音乐在创作的时候其实已经有意无意间考虑了和声的因素。那么,我们在编配的时候只要找出这个规律就可以了。前提条件是大脑里已经有了这方面的想法和经验。例如《童年》这首歌曲,原来的和弦编配已经有很多种。这里我们把它全部按照I-VI-IV-V的模式套进来看看有什么奇怪的现象。




在第三行左数第三小节,旋律写的是D这个音,按照一般的配法,这个地方使用II级和弦是非常恰当的,前后的连接也非常正统。似乎用I-VI-IV-V这个和弦进行模式不恰当。但是听实际的音响效果又别有一番风味。这是为什么呢?因为旋律不是孤独存在的,它是和和声结合在一起,形成了总体的音响效果,最终传递到人耳。这里用的是IV级和弦。IV级和弦的构成音是“F、A、C”,现在加入旋律的“D”音。使得人耳听到的组成音变成“F、A、C、D”,这是一个新的和弦,是II级小七和弦的一个转位。由于后面是V级,因此在这里,所配和弦与旋律构成的新和弦进行恰好是一个典型的重属七和弦解决到属和弦的过程。

通过上例可以总结出两点,一是要不为旋律所盲动,尽量寻找相应的固有的套路去解决问题。二是要和旋律结合起来,从总体上把握和声。

吉他弹唱进阶:如何为歌曲编配和弦

首先我们来看看歌曲与和弦之间是一种什么关系。打个比方来说歌曲与和弦之间的关系就象是人与衣服之间的关系一样:

★一个形象好的人穿上得体的衣服会给人以美的享受;同样一首好的歌曲配上适合的和弦会给人以舒服的感觉。
★精心选配的衣服可以弥补一个人形象上的缺陷;同样精妙的和弦编配可以弥补歌曲的某些不足。
★一个人可以穿上不同风格的衣服以展现不同的风采;同样一首歌可以采用多种配和弦的方法以取得不同的效果。
★有的人因身材、相貌等原因,为他设计服装较容易,而有的人则正相反;同样有的歌曲比较容易配和弦,而有的歌曲则不容易配和弦。
所以一个编曲人就象是一名服装设计师,不管这个人的形象如何,你都能为他设计出好的、满足他要求的、流行的服装。
近年来流行歌曲的和弦编配技术有了较大的发展,这种发展在我国大致经历了这样的几个阶段:

(1)只会使用主和弦、下属和弦、属和弦(或属七和弦)等正和弦及Ⅱ级小和弦、 Ⅲ级小和弦、Ⅵ级小和弦等副和弦的初级模仿阶段。即在C调中只会使用:C、Am、Em、F、G、G7、Am7、Em7、Dm7、D7、C7和弦。
如:《军港之夜》、《让我再看你一眼》的和弦配置。

(2)在上个阶段的基础上学会使用了挂留和弦、大七和弦、六和弦、九和弦的中级阶段。即在C调中会使用:Csus4、C大七、F大七、C和弦。
如:崔健的早期作品的和弦配置。

(3)之后是学会了降Ⅶ级和弦、增、减和弦的成熟阶段。即在C调中会使用:和弦。
如:《长江之歌》《透过开满鲜花的月亮》《祝你平安》《牵挂你的人是我》等歌曲的和弦配置。
(若你需要上述歌曲的谱子,请读由我编纂的配有原版和弦的简谱歌本《97上榜金曲》《98上榜金曲》,价格详见本书第4页的“邮购目录”)

(4)现在又进入了创新阶段。如:①创造性地使用5和弦,②和弦不变而外声部音下行。
本 书中涉及到“5和弦”的歌曲有:《愚公移山》《孔雀东南飞》《为你》《干杯朋友》《只有你陪我一起唱歌》《自由自在》。本书中涉及到和弦不变而外声部音下 行的歌曲有:《为你》《你哪里下雪了吗》,乐曲有:《人们的梦》《旅途》。和弦编配技术的发展除了使我们会用更多的和弦外,还使我们对和弦之间的连接有了 更深的认识,以前似是而非的东西经过一番争论也基本有了定论。

▲和弦之间较好的连接方式(按程度排列):
(纯)四度连接最好
如:C类→F类,D类→G类,E类→A类,F类→bB类,G类→C类,A类→D类,bB类→F类,等。
(大)三度连接好
如:C→Em,D→#Fm,E→#Gm,F→Am,G→Bm,A→#Cm,bB→ Dm等。
(大)六度连接好
如:C→Am,D→Bm,E→#Cm,F→Dm,G→Em,A→#Fm,bB→ Gm,等。
挂4和弦转同名大三和弦好
如:Csus4→C,Dsus4→D
(纯)五度连接较好
如:C→G,D→A,E→B,F→C,G→D,A→E,#B→F,等。

在二度连接中,这样的二度连接是好的:

①下属和弦→属和弦,属和弦→下属和弦。如在C大调中,F→G、G→F是好的。在Am小调中,Dm→Em、Em→Dm是好的。

②Ⅲ级和弦→下属和弦,下属和弦→Ⅲ级和弦。如在C调中,Em→F,F→Em。同名和弦之间的“顺向连接”是好的:
如:G→G7,C→C大七,Am→Am7等。


▲和弦之间不好的连接方式(按程度排列):
“关系小和弦”转接“关系大和弦”不好,
即Ⅵm→Ⅰ不好。
如:Am→C、Dm→F、Em→G这样的连接都是不好的甚至可以说是非常恶劣的,要避免使用。
注:段落间出现这种情况可除外。如:《天蓝蓝海蓝蓝》
而A→C、E→G、D→F这样的连接在某种特定情况下是好的。
如:在G调中D→F是好的,在D调中A→C是好的,在A调中E→G是好的等。属和弦后接二级小和弦不好,即:Ⅴ→Ⅱm不好。
如:在C调中G→Dm,在G调中D→Am等,这样的连接是非常别扭的,因为它属于“逆向连接”。
二级小和弦后接一级大和弦不好,即Ⅱm→Ⅰ不好。
如:在C调中Dm→C,在G调中Am→G等,这样的连接也是恶劣的,要避免使用。注意:Dm→C这样的连接在C调中是不好的,而在F调中却是好的,同样,Am→G这样的连接在G调中是不好的,而在C调中却是好的。
同名和弦之间的“逆向连接”是不好的
如:G7→G,C大七→C,Am7→Am等。
在大调歌曲中三级和弦后接二级和弦一般来说是不好的,即在大调歌曲中,Ⅲm→ Ⅱm不好、Ⅲm→Ⅱ7不好。
如:在C大调中Em→Dm、Em→D7,在G大调中Bm→Am、Bm→A7等,这样的连接也是非常别扭的。
(注:在小调歌曲中Ⅲm→Ⅱm是可以的)
一级大和弦后接二级小和弦一般来说也是不好的,即Ⅰ→Ⅱm不好。
如:在C调中C→Dm,在G调中G→Am等,这样的连接已被越来越多的编曲人所抛弃。
不过类似这样的连接是可以的:Ⅰ→Ⅱm→Ⅴ→Ⅰ。
此外,Ⅰ级大和弦后接Ⅱ级大和弦则是可以的,即Ⅰ→Ⅱ是可以接受的。
如:在C调中C→D,在G调中G→A等。

★介于上述“好”与“不好”之间的其他和弦连接都是可以的。
如:bB→G,Em→bB等。
上 述所述是当前比较流行的和弦编配原则,这些原则得到了绝大部分编曲人的认可。你可能会举出一些不符合这些原则的例子,但我要告诉你有这样的情况存在:①一 些编曲人执迷不悟②一些编曲人死不改悔③一些编曲人初学乍练④所举的例子是以前编曲水平不高时的作品。⑤特殊情况的灵活处理。


自测题:
  指出下列各个和弦走向中哪些是好的哪些是不好的(若是不好的请具体指出何处不好),并指出各个和弦的级数:
注:以上这些好与不好的原则或例子都是从纯和声的角度上来说的,没有考虑旋律的具体情况。
★我们要努力掌握并深刻理解这些原则,但又不要被这些原则所束缚,具体情况是千变万化的。
★配和弦时要尽量避开不好的连接,多用好的连接。
★你要想打破规则,必须首先要掌握规则。在你一知半解的时候,不能用“打破规则”来为自己不好的和弦编配辨护。

吉他中国论坛

制作MIDI-实战篇(上集)

接着就是对音乐的分析,由于此教材面对的接触电脑音乐不长 时间的朋友,所以不能太复杂的,大家细细听这首歌,听听伴奏是由哪几个乐器组成的?不要求把所有的乐器都作出来,可以有自己编曲方法,如果对编曲没把握的 朋友,就参考音乐提供的乐器,能作得有多象就作多象,毕竟那是音乐制作人作出来的作品,不管经验还是各方面都比我们强,不知道大家听出来了没?有几种乐器 声音?是不是有钢琴的声音,弦乐的声音.贝司和鼓的声音?相信这几种乐器大家都可以听的出来,那好,我们就按这几种乐器来学习怎么用软件制作MIDI,制 作MIDI要求各种乐器的弹奏方法和在歌曲的作用了解清楚,这很重要,否则都无从下手,如果对此方面还不了解的朋友,建议多听音乐,听听配器,且每种乐器 所起的重要和弹奏方法,在此教材稍微介绍上一点,主要是抛砖引玉作用,作得好不好还看大家以后的努力程度.细细听完这首<童话>以后,我们对 配器也了解了,不了解的朋友继续听,然后用乐器吉他或者电子琴把谱子写出来,写的越详细越好,这也就是扒带,这对学习有很大的帮助,多扒好的歌曲,对提高 配器各方面提高特别有效,好吧,那我们就开始制作吧,估计大家都等急了.

新建MIDI轨,按上面提到的乐器,我们需要建立四个MIDI轨,建MIDI大家都懂了吧, 点击右键选择Add midi track,我们一轨一轨来制作,首先是钢琴轨,钢琴在这首歌曲中很重要,是整个歌曲节奏和旋律的形成,所有需要两轨钢琴,一轨是旋律,另一轨是节奏.我 们先来作节奏部分的,在我们新建的第一轨作为钢琴节奏制作,制作前要先调入软音源,按F11,出来以后鼠标左建点击音色栏,选择Hypersonic音 源,进去音源界面以后钢琴音色,都知道哪个了吧?也就是Acoustic piano,大钢琴音色,选择好以后,回到MIDI轨页面,在轨道的左边选择上所选的音源,也就上MIDI轨左边设置里面的OUT:里面选择我们所用的音 源,在通道设置里面选择通道1,这是我们要做的.这个选择好以后就开始音符的输入,选择菜单栏的MIDI选项,进去后打开第一个选项Open key editor,进行音符的输入,还有就是歌曲的速度设置,这首歌曲在75左右,关于谱子那就是自己所扒的内容了,实在扒不出来就到网上下载谱子,但那是简 谱,还是需要自己配上和弦节奏,这都是关于乐理方面的知识了,学过吉他的朋友应该很容易明白,没有学过吉他的朋友就学些基础乐理吧,很容易明白的,这是制 作所必须懂的东西,还要根据自己对钢琴的弹奏方法进行音符的输入,这是一个示范曲,大家可以听听,作得不是很好,主要是为教材所用.示范曲如下:

示范曲收听

作完第一轨钢琴以后,接着作的就是第二轨旋律钢琴,制作步骤跟上面一样,选好音源以后就开始 写入音符,音色和上面的一样,是大钢琴音色,所写的音符也就是自己所扒的内容了,具体方法上面也提过了,多扒慢慢的速度就快了,所有这些都需要反复的去 练,关于扒带的方法我会在另一篇教材说,详细的跟大家说扒带的具体步骤.在这同样为大家提供一个示范曲,作的很粗糙,希望大家别见笑,示范曲02如下:

示范曲收听

以上说的钢琴音轨的制作,通过这样大体上是完成,当然这是针对接触软件时间不长的朋友,真正 作起来还有很多东西,比如说每个音符的音量,力度等等参数的调节,这是一方面,另外所输入的音符也不是那么简单,这里主要是配合教材和初学的朋友,音符还 编得更细腻些,这样听来感觉会更好,象是一个钢琴师在演奏,这是我们学习的目标.

好吧,钢琴制作部分就想讲到这,现在已经是深夜2点半了,自己也瞌睡了,明天我会继续把剩下 的部分讲完,剩下部分包括弦乐,贝司和鼓的制作,还有最后的导出保存,都在明天的课里讲给大家,感谢所有看此教材的朋友,我的目的只有一个,方便大家学 习,少走弯路,然后实现自己的音乐之梦,期待你们的原创作品!

http://edu.chinaz.com

软件制作MIDI-实战篇(下集)

继续:上节课我们讲了钢琴部分的制作,相信大家对MIDI 制作都有所了解,也应该知道作MIDI需要哪些知识,接着我们这节课讲的是剩下乐器的MIDI制作,包括弦乐,贝司和鼓.讲完这些以后就是讲最后的后期调 整和导出作品,通过这两集教材,大家对MIDI制作有个初步的了解,且自己要动手制作,不作永远都不会作,多实践,多思考,这就是学习方法.

关于弦乐的制作,弦乐在歌曲主要是铺底的作用,使整首歌曲听起来不是很空,感觉会更好些,开 始制作弦乐的部分可以根据和弦走向来制作,当然弦乐不是简单的和弦,它还含有很多成分,这要靠自己以后去细细听每一首作品,然后把它扒下来自己练习,多练 习就会了,在这首<童话>的弦乐制作中,我选用了两轨来制作,一轨是铺底,另一轨是作旋律,让歌曲的旋律感更加明显些,关于弦乐该怎么作?我 会以另一个教材跟大家说明,到时候会集中我们常用的几种乐器的编曲方法,好的,我们先做的铺底弦乐,制作方法跟前面所说的一样,把音色选好,注意通道的对 应,否则会没声音输出,然后就是输入音符,这里所作的也是根据和弦的走向也配的弦乐,以下是示范曲03,作的有些急,为教材所用.

示范曲收听

作完铺底弦乐以后,我们来作旋律弦乐,这主要是为了突出旋律性,大家每作一轨目的都要明确, 到底这轨是作什么?在歌曲起的什么作用,该怎么作?从歌曲哪个地方开始进等等,把这些问题考虑清楚以后,就可以往钢琴卷里写入音符,注意音符的节拍等等, 同样,在这举出了一个示范曲04,这弦乐是在歌曲中间加入的,突出歌曲的层次感,作得不是很细,仅供大家参考.

示范曲收听

上面介绍的是弦乐部分的制作,接下来我们讲的是贝司的制作,贝司是低音吉他,它主要作用是弥 补鼓的低音部分,加入贝司以后歌曲总体感觉会更稳重,更扎实些,开始接触MIDI制作的朋友,贝司可以采用走跟音的方法,等以后熟练以后,自己再尝试加些 花子,这很重要,有好多歌曲贝司的花子就是一个亮点,很不错的效果,这就要靠大家以后听音乐去领会,学习贝司该怎么作,在这里同样提供一个示范曲05,仅 供大家参考.这贝司做法就是走跟音,作的比较简单,这样大家更容易明白!

示范曲收听

最后一轨就是鼓的制作了,这稍微复杂些,刚开始接触电脑音乐的朋友作起来稍微麻烦些,首先你 要对架子鼓有所了解,鼓是哪几部分组成的,每部分都起什么作用,这首歌曲该配什么鼓节奏,这都是制作鼓之前要考虑好的,看平时鼓Loop的积累了,要作到 听到一段旋律就该配什么的鼓节奏,现在大部分软件都提供有鼓Loop,这样作起来就比较方便,所以大家还是需要经常练,随便作一段旋律,或者用吉他弹奏一 段旋律,然后自己配上鼓,弹吉他的朋友这部分很容易学会,因为他们听的歌曲多,各种各样的鼓节奏都比较熟悉些,不会吉他的朋友也没关系,多扒带,然后多积 累些鼓的Loop,这对鼓轨的制作很有帮助,这首<童话>的鼓是中间才起,也是为了突出层次感的作用,这里所才用的鼓音源是常见的软音源 Lm7,这鼓音源不是很好,还有更好的音源,虚拟鼓手等等,大家以后可以多尝试,选择自己喜欢的音源,为了大家更能简单操作,就把所有鼓部分都在一轨内制 作完成,记住鼓的通道是第十通道,这也别弄错了.同样在这里也提供有示范曲06,这是写教材时作的鼓,制作得不细腻,仅供大家参考.

示范曲收听

通过以上的介绍,全部乐器的制作就完成了,接着就是后期工作,如果想作的细腻的话,要对音符 进行力度和音量等调节,还有音规的声相调节,这很重要,如果不进行声相的调整,整个乐器就会混都一块,给人感觉会比较乱,还有音轨间的均衡,哪些音量需要 大些,那些的音量需要小些,这都需要我们后期的调节,调节可以在软件里完成,最好就是在软件里完成,当然也有些朋友是分轨导出,然后输入音频编辑软件,在 音频编辑软件里进行声相和均衡的调节,这也行的,看自己哪样更方便,效果更好就怎样?制作音乐本来就不是在一个软件里完成,需要几个软件综合起来使用,这 样出来的作品才是好的作品,每个软件都有它的长处,我们就好好利用软件的长处来制作相关部分,发挥好每个软件的长处,例如SAM2496混音效果很好,大 家可以分轨导出音乐,然后导入SAM2496里进行混音处理,这都要需要自己灵活去处理,多动脑子,多想想该怎么作更好,怎样好我们就怎样作就行了。

最后要讲的是导出作品,当我们调好各各乐器的均衡以后就可以导出作品了,导出时有些地方需要 注意的,否则会出现导不出音乐的现象, 用鼠标点一下工具栏的箭头工具,然后按住鼠标左键不放,在音轨的上面把有音乐的部分全部选择上,选择后会变成暗蓝色,接着在菜单栏里的File下面选择 Export里的Audio Mixdown,打开对话框后,选择保存类型,一般都选择MP3格式.然后还有精度的选择,集成声卡的朋友可以选择 64KBit/m.44.100khz.stereo.在Output里选择Stereo out(Stereo),Channes也选择Stereo,别的选项就用默认值吧.然后选择路径,按Save就可以把作品保存好了.最后就慢慢欣赏自己 的作品吧.好吧,本次教材就讲到这吧,由于写的时间比较紧,所以难免有漏的或者说不对的地方,还希望大家多多包涵。

http://edu.chinaz.com

May 24, 2006

Useful AWK script

HANDY ONE-LINERS FOR AWK                                  22 July 2003
compiled by Eric Pement version 0.22
Latest version of this file is usually at:
http://www.student.northpark.edu/pemente/awk/awk1line.txt

USAGE:

Unix: awk '/pattern/ {print "$1"}' # standard Unix shells
DOS/Win: awk '/pattern/ {print "$1"}' # okay for DJGPP compiled
awk "/pattern/ {print \"$1\"}" # required for Mingw32


Most of my experience comes from version of GNU awk (gawk) compiled for Win32. Note in particular that DJGPP compilations permit the awk script to follow Unix quoting syntax '/like/ {"this"}'. However, the user must know that single quotes under DOS/Windows do not protect the redirection arrows (<, >) nor do they protect pipes (|). Both are special symbols for the DOS/CMD command shell and their special meaning is ignored only if they are placed within "double quotes." Likewise, DOS/Win users must remember that the percent sign (%) is used to mark DOS/Win environment variables, so it must be doubled (%%) to yield a single percent sign visible to awk.

If I am sure that a script will NOT need to be quoted in Unix, DOS, or CMD, then I normally omit the quote marks. If an example is peculiar to GNU awk, the command 'gawk' will be used. Please notify me if you find errors or new commands to add to this list (total length under 65 characters). I usually try to put the shortest script first.



FILE SPACING:

# double space a file
awk '1;{print ""}'
awk 'BEGIN{ORS="\n\n"};1'

# double space a file which already has blank lines in it. Output file
# should contain no more than one blank line between lines of text.
# NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are
# often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'

# triple space a file
awk '1;{print "\n"}'

NUMBERING AND CALCULATIONS:

# precede each line by its line number FOR THAT FILE (left alignment).
# Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*

# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*

# number each line of a file (number on left, right-aligned)
# Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'

# number each line of file, but only print numbers if line is not blank
# Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};{print}'
awk '{print (NF? ++a " :" :"") $0}'

# count lines (emulates "wc -l")
awk 'END{print NR}'

# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum awk
'{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'
# print every line after replacing each field with its absolute value awk
'{for (i=1; i<=NF; i++) if ($i < i =" -$i;" i="1;" i =" ($i" total =" total"> max {max=$1; maxline=$0};
END{ print max, maxline}'


# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '

# print the last field of each line
awk '{ print $NF }'

# print the last field of the last line
awk '{ field = $NF }; END{ print field }'

# print every line with more than 4 fields
awk 'NF > 4'

# print every line where the value of the last field is > 4
awk '$NF > 4'


TEXT CONVERSION AND SUBSTITUTION:

# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"");print}' # assumes EACH line ends with Ctrl-M

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r");print}

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
# Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile

# Use "tr" instead.
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line
# aligns all text flush left
awk '{sub(/^[ \t]+/, ""); print}'

# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "");print}'

# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'
awk '{$1=$1;print}' # also removes extra space between fields

# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ");print}'

# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*

# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*

# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar");print}' # replaces only 1st instance
gawk '{$0=gensub(/foo/,"bar",4);print}' # replaces only 4th instance
awk '{gsub(/foo/,"bar");print}' # replaces ALL instances in a line

# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")};{print}'

# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
awk '!/baz/{gsub(/foo/, "bar")};{print}'

# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarlet|ruby|puce/, "red"); print}'

# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*

# if a line ends with a backslash, append the next line to it
# (fails if there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*

# print and sort the login names of all users
awk -F ":" '{ print $1 | "sort" }' /etc/passwd

# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file

# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file

# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'

# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file

# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'

# remove duplicate, nonconsecutive lines
awk '! a[$0]++' # most concise script
awk '!($0 in a) {a[$0];print}' # most efficient script

# concatenate every 5 lines of input, using a comma separator
# between fields
awk 'ORS=%NR%5?",":"\n"' file



SELECTIVE PRINTING OF CERTAIN LINES:

# print first 10 lines of file (emulates behavior of "head")
awk 'NR <>1{exit};1'

# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'

# print the last line of a file (emulates "tail -1")
awk 'END{print}'

# print only lines which match regular expression (emulates "grep")
awk '/regex/'

# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'

# print the line immediately before a regex, but not the line
# containing the regex
awk '/regex/{print x};{x=$0}'
awk '/regex/{print (x=="" ? "match on line 1" : x)};{x=$0}'

# print the line immediately after a regex, but not the line
# containing the regex
awk '/regex/{getline;print}'

# grep for AAA and BBB and CCC (in any order)
awk '/AAA/; /BBB/; /CCC/'

# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'

# print only lines of 65 characters or longer
awk 'length > 64'

# print only lines of less than 65 characters
awk 'length < nr="=" nr="=" nr="=" nr="=">

May 22, 2006

芹菜鱿鱼

芹菜鱿鱼


原料:
原 料:
材料: 鱿鱼,芹菜各150克,红辣椒1个,大蒜2粒,姜3片 调味料: (A)沙茶酱,酱油各1/2小匙,米酒,鸡精各1小匙。 (B)淀粉水1小匙。


制法:
(1)红辣椒去蒂及籽,切片,大蒜去皮切片,芹菜去叶洗净,切段。
(2)鱿鱼划交叉刀蚊,切成片,放入热水汆烫,捞起,泡入冷水,沥干备用。
(3)锅中倒入1大匙油烧热,爆香红辣椒,大蒜,姜,加入鱿鱼,芹菜及A料翻炒入味,淋入B料勾芡即可。
特点:
鱿鱼肉遇热会急速紧缩,可让切刻的花纹更加漂亮,再泡入冷水即可保持弹性,让口感更爽脆。

May 16, 2006

VIMRC

Some Useful VIMRC Definitions:

function! ShowFunc()
let oldmake= &makeprg
let &makeprg= 'listfunc'

silent! make%
cwindow

let &makeprg=oldmake
endfunc

"To enable restoring (for an xterm):
"set t_ti=^[7^[[r^[[?47h t_te=^[[?47l^[8
"(Where ^[ is an , type CTRL-V to insert it)
set t_ti=^[7^[[r^[[?47h
set t_te=^[[?47l^[8

set ignorecase
"map j_
"map k_
"set wmh=0
map ^[j j
map ^[k k

map :cn
map :cp
map :grep * -r :cw :set syntax=off k
map :on
map :w :make :cw
map :call ShowFunc()

augroup filetypedetect
au! BufRead,BufNewFile *.inc setfiletype make
augroup END

" Automatically reload .vimrc when changing
autocmd! bufwritepost .vimrc source %

============================
The listfunc script:
#!/bin/sh
ctags -x --c-types=f -u $1 | gawk '{printf("%s:%s: %s\n",$4,$3,$1) > "/dev/stderr"}'

May 13, 2006

PLOTMIDI V2

#configurable
x=read.csv("in2.csv");
timebase=384;
tempo=789494;

#do not change unless you know what you are doing
grid=150;
tones_start=27;
tones=c("HighQ","Slap","ScratchPush","ScratchPull","Sticks",
"SquareClick","MetronomeClick","MetronomeBell","KickDrum2",
"KickDrum1","SideKick","SnareDrum1","HandClap","SnareDrum2",
"LowTom2","ClosedHiHat","LowTom1","PedalHiHat","MidTom2",
"OpenHiHat","MidTom1","HighTom2","CrashCymbal1","HighTom1",
"RideCymbal1","ChineseCymbal","RideBell","Tambourine","SplashCymbal",
"CowBell","CrashCymbal2","VibraSlap","RideCymbal2","HighBongo",
"LowBongo","MuteHighConga","OpenHighConga","LowConga",
"HighTimbale","LowTimbale","HighAgogo","LowAgogo",
"Cabasa","Maracas","ShortHiWhistle","LongLowWhistle",
"ShortGuiro","LongGuiro","Claves","HighWoodBlock",
"LowWoodBlock","MuteCuica","OpenCuica","MuteTriangle",
"OpenTriangle","Shaker","JingleBell","Castanets","MuteSudro","OpenSudro")


tick=tempo/timebase;
x$time=x$time*tick/1000000;
xmax=max(x$time);
ymin=0;

postscript("drum.eps", horizontal=FALSE, onefile=FALSE,paper="special",height=10,width=14);

tone=rep(0,128);
f=factor(x$tone);
ymax=length(levels(f))*grid;
i=0;
plot(0,0,xlim=c(-10,xmax),ylim=c(ymin,ymax),xlab="",ylab="");
for (ii in levels(f)){
t=x$time[x$tone==ii];
v=x$velocity[x$tone==ii];
lines(t,i*grid+v,type="s"); text(-5,i*grid+grid/2,tones[as.numeric(ii)-tones_start+1],cex=1.0);
i=i+1;
}

dev.off()

May 12, 2006

MIDI FORMAT

MIDI Format is explaine in this website.

The clock unit in the CSV file generated by the midicsv tool is in terms of PPQN Clock.


The rate PPQN ticks is calcuated as follows:

MicrosPerPPQN = MicroTempo/Division

MicroTempo=60,000,000/BPM(Beat Per Minute)

Divison is also called Timebase. It is specified in the midi file header. For hardware Midi equipments, this is a fixed value. For software generated midi files, this usually is also a application-dependent fixed value

May 11, 2006

Plot Midi Drum Line


Following the story of midicsv that converts midi to csv, I wrote some scripts to plot the drum line from an exising midi file.

First, run the midicsv tool to creat the csv file:

Second, run the awk script to filter out everything other than drum beats. The script (filter.awk)

BEGIN{
FS=",";
printf("time,tone,velocity\n");
}
{
if ($3 ~ /ote/ &&amp;amp; NF==6 && $4==9)
printf("%s,%s,%s\n",$2,$5,$6);
}
Now, plot the drum line in R (get it from www.r-project.org) Here is the script (midiplot.r)
grid=200;
x=read.csv("in2.csv");
xmax=max(x$time);
ymax=max(x$tone)*grid+200;
plot(0,0,xlim=c(0,xmax),ylim=c(35*grid,ymax));
tone=rep(0,128);
f=factor(x$tone);
for (i in levels(f)){
t=x$time[x$tone==i];
v=x$velocity[x$tone==i];
lines(t,as.numeric(i)*grid+v,type="s");
}
Above is the plot result.

MIDICSV

This must be the first tool for midi: midicsv. It converts any midi to a human-readable CSV file, and also converts the CSV file back to midi. This is fantastic because now I can get a really-good-sound midi, convert it to midi, and see now it beats the drum. I was trying to write a midi parser based on midi2abc, but thought there must already exist something like it giving that MIDI has been around for a while.

What a delightful day!

Thanks, John Walker!

Makefile

A typical Makefile with Patterns:

Pattern:

%.o: %.c $(HEADERS)
$(CC) $(CFLAGS) -c $<

$@: The target
$<: The name of the first prerequisite
$^: The names of all the prerequisites

Less common:

$?: The names of al lthe prerequisites that are newer than the target

AutoIt3 with DLL

Finally I got AutoIt3 to work with a DLL I wrote from C and compiled with MingW. Now I can start writing native windows applications with integration of native C code. Nice.

MMA

MMA is a great midi generator. You give it the chord, and the style (rock, folk, waltz,etc), and it generates a rich midi file for you. It is open source software with fantastic documentation and code. If you are even remotely interested in MIDI and music, you definitely should check it out.

abcm2ps with UTF-8 Support

Two days ago I got an email from the Jef, the author of abcm2ps, about adding UTF-8 support to abcm2ps. This is something I emailed him a while ago and he never got a chance to do it. Now he is planning to do it. That's great. We are starting with Chinese UTF-8 support, with help from the package called "cnprint", which produces postscript file from a text file and a TTF font file.

May 8, 2006

槐花吃法 ( Black Locust Flowers Recipe)

嫩绿色的浓荫里, 串串白色的槐花香气浓郁,味道香甜。槐花还含有多种微生素和微量元素,具有较高的营养价值。槐花有多种吃法既可加面粉蒸食、做饼,又可凉拌、炒食。

* 烙槐花

主料 槐花。最好是尚未完全开放的槐花。择净,用开水烫过,漂洗(清除槐毒)。

做法
1.槐花控去大部分水。鸡蛋一个,打开打散,加入花椒粉、精盐搅匀,倒入盛槐花的盆中,搅匀。面粉均匀的撒入盆中,与槐花搅成稠糊状。
2.平底锅加少量花生油,烧热。将调好的槐花糊倒入锅内,均匀摊开煎烙。待下面一层可以晃动时,反转过来再烙另一面。将火调小,反复两面烙,烙至两面深黄,内里已熟时,直接上盘。

* 槐花鸡蛋汤
做法
1. 汤锅上火,葱花、花椒爆锅,加水、海米、精盐、烧开。把准备好的槐花入锅。
2. 鸡蛋2个,加水三四滴打散,待锅中槐花汤烧开,立即将鸡蛋液均匀的浇入锅内,用勺推开,烧开即可。
3. 盛碗时,点大红樱桃二三个配色。

March 5, 2006

Web Font

Some examples of good web font:

* body{font-family:arial,sans-serif}
* TD.txt{font-family:verdana,sans-serif;font-size:small;)
*

Arial: a sans serif (meaning without serifs—the little doohickeys at the ends of each letter) face that has a streamlined, more modern look, but isn't necessarily easier to read on screen because it's on the narrow side and can look very light in smaller sizes. It's also incredibly boring and looks far too much like the type used by the IRS and other tax agencies, so it can have negative subliminal connotations.

Helvetica: a sans serif face similar to Arial.

Verdana: An extremely easy-to-read sans serif face that's included with the Internet Explorer. Other fonts: Tahoma (a face almost identical to Verdana that comes with Microsoft Office), Arial, or Helvetica.

Comic Sans: This face is informal and friendly, which is great for some sites, but not professional enough for others.

February 27, 2006

Recipe for Korean Cooking

韩式料理食谱

1.卤牛蒡(4人份)

材料:牛蒡1条(40公分)白芝麻少许。

白醋水:白醋3大匙、开水3杯。

卤汁:酱油4大匙、果糖1.5匙、开水1杯(230cc)。

做法:

1.牛蒡洗净用菜瓜布轻刷表皮。

2.横切段后再切成粗丝。

3.将牛蒡放入白醋水中,浸泡20分钟后,从醋水中捞出,放入卤汁中。

4.卤汁煮滚后转小火煮,煮至剩下少许汤汁后熄火即可。 

5.可放冰箱储存,食用时撒上白芝麻。

料理秘诀:

白醋水可避免牛蒡丝氧化变黑。

牛蒡外皮营养成分高,不需削皮,准备乾净菜瓜布刷净即可。


2.萝卜丝泡菜(4人份)

这是一道可以现做现吃的“新鲜”泡菜,尤其冬季萝卜特别鲜甜好吃,可以多做一些,放在冰箱两、三天也不会坏。

材料:萝卜40克、盐巴20克、碎白芝麻8克。

调味腌料:韩国辣椒粉24克、砂糖40克、白醋20cc、芝麻油8cc、蒜泥8公克、老姜末8公克。

做法:

1.白萝卜洗净用菜瓜皮轻刷表皮。

2.将白萝卜刨成丝,用盐腌5分钟后,洗净沥乾水分。

3.加入调味腌料拌匀即可。

4.上桌前加上碎芝麻拌在一起。


3.海带芽冷汤(4人份)

酸酸凉凉的海带芽汤很开胃,整个口腔的感觉像被涤清一样,很适合搭配味道较重的菜肴。

材料:海带芽7克、小黄瓜1/2条、开水3杯。

调味料:白芝麻2小匙、蒜泥2小匙、白醋5大匙、砂糖1大匙、盐巴少许。

做法:

1.海带芽先泡冷水10分钟后洗净。

2.小黄瓜洗净切丝。

3.准备3杯白开水加入调味料及海带芽即可。一般放冷藏食用最好。

4.食用前撒上黄瓜丝,也可加入冰块。


4.拌冬粉(4人份)

拌冬粉是韩国最常见的家庭料理之一,由绿豆、荞麦粉做成的韩国冬粉较粗且有弹性。

材料:橄榄油2大匙、冬粉2卷、空心菜5根(切段)、红萝卜1/4根(切丝)、葱1根(切段)、洋葱1/4颗(切丝)、盐少许、肉片5片(狗、鸡、猪均可,也可以不加)。

调味料:酱油2大匙、砂糖1大匙、麻油1小匙、黑胡椒粉、白芝麻少许。

做法:

1.水煮沸,放入冬粉,直到冬粉呈透明状后捞出,沥乾水分。

2.橄榄油2大匙炒肉片和蔬菜。


5.蔬菜饼(4人份12片)

调面糊的技巧很重要,也需要多练习,但蔬菜饼做法简单又健康,当成宵夜点心都很合适。

材料:油3大匙、高筋面粉20大匙、鸡蛋1个、冷开水150cc、韭菜、葱(切小段)、红萝卜切丝(任何蔬菜均可以放入,份量可依个人喜好增多减少)、盐少许。

做法:

1.面粉加开水、蛋,用打蛋器慢慢调成面糊。面糊稠度以打蛋器可以拉起,慢慢滴下最佳。

2.将面糊放入蔬菜、盐搅拌均匀。

3.放油热锅,用汤杓将面糊舀出煎饼,中火将饼煎至两面呈金黄色即可。


6.韩式炒年糕(2人份)

甜甜辣辣的炒年糕,是韩国年轻人的最爱。比较特别的是,韩国年糕不用油炒,而是利用水煮方式,让年糕饱吸酱料,搭配脆脆的青菜,口感层次很丰富。

材料:宁波年糕12片、甜不辣或竹轮8个、开水240cc、白芝麻2小匙。

酱料:韩式辣椒酱4小匙、砂糖8小匙、洋葱1/4颗(切丝)、葱1根(切段)、高丽菜1碗(切成小方块)。

做法:

1.将开水240cc、调味料及年糕放入锅中。

2.开火煮至水少掉1/3后,放入甜不辣再拌炒。

3.待水减少到1/2时,熄火,放入洋葱、葱及高丽菜拌拌即可。食用时撒上芝麻。

料理秘诀:

蔬菜最后放,可以保持鲜脆。



7. 热热的松子茶是韩国冬天的驱寒甜品,核果类富含优质油脂成分,对于心血管疾病、美容、抗老化等都有助益。

做法:

1.花生、核桃、松子、栗子(也可加薏仁粉)四种核果类等量磨粉。

 2.饮用时用热开水冲泡,70cc水中可放入2茶匙的松子茶粉,并可依个人喜好加入砂糖。

3.准备一个大碗,先将冬粉和炒好的菜肉混合搅拌均匀,再加入调味料拌匀即可。

February 18, 2006

Running Vmware Player as a service

credit to http://www.windowsitpro.com/Windows/Article/ArticleID/42607/42607.html
for this acticle.


If you're a VMware enthusiast, you've probably on more than one occasion
wanted to log off from your computer while leaving your virtual machines
(VMs) running. Or, maybe you've wanted selected VMs to start as soon as
your system boots so that your host system can log on to a domain
controller (DC) running inside one of the host machine's VMs. Sound too
good to be true? That's what I thought. I assumed that logging off of my
computer and having my VMs remain running was an unattainable dream. But
I discovered that getting VMs to run as services is possible and very
easy to configure.

Tools for Service
VMware doesn't natively support running its software as a service, but
configuring VMware Workstation 4.0 VMs to run as services is almost as
easy as tying your shoes. All you need to get started are two
tried-and-true Windows resource kit tools: instsrv.exe and srvany.exe.
Both tools are available as free downloads. Go to
http://www.microsoft.com/downloads, enter Windows 2003 Resource Kit
Tools in the Keywords field, and click Go. Then, click the Windows
Server 2003 Resource Kit Tools Download button at the Windows Server
2003 Resource Kit Tools Web page to download rktools.exe-which contains
the most recent versions of Instsrv and Srvany-and run the executable to
install the tools on your system.

Note that you can install the Windows 2003 resource kit tools on a
Windows 2003 or Windows XP system. If your host system runs Windows 2000
or Windows NT, you can acquire Instsrv and Srvany from the Win2K or NT
resource kit CD-ROMs or you can install the Windows 2003 resource kit
tools on an XP system and just copy Instsrv and Srvany from the XP
system to the %windir% folder on your Win2K or NT host system. The
Windows 2003 versions of Instsrv and Srvany run on the earlier OSs
without any problems.

Getting Started
Installing the resource kit tools updates the system path to include the
resource kit installation folder. Updating the path requires a reboot,
so be sure to reboot your system after installing the resource kit.
Alternatively, you can copy Instsrv and Srvany to a folder already in
the path, such as the folder C:\windows\system32.

With the resource kit files in place, your next task is to determine the
location of the VMware application's vmware.exe file. I used the default
settings when installing VMware, so the path I needed was C:\program
files\vmware\vmware workstation\vmware.exe.

The last bit of information that you need before you configure the new
service is the path to the configuration file of the VM that you want to
turn into a service. This file is in the folder in which the VM was
created and has a .vmx extension. All my VMs are stored on my system's E
drive, so the path to the .vmx file of the VM that I want to run as a
service is E:\vms\w2k1\w2k1.vmx. When you have the vmware.exe path and a
VM's .vmx path information, you're ready to create the service.

Creating the Service
First, decide on a name for the service. I prefer to preface the name of
the VM with VM_ to form the service name. For example, I would give my
VM named W2K1 the service name VM_W2K1. After you decide on the service
name, you can use the following syntax to set up the service:

instsrv

So a sample command might be

instsrv VM_W2K1
c:\windows\srvany.exe
Now you need to modify the service's parameters by using a registry
editor and the Microsoft Management Console (MMC) Windows Services
snap-in. In the registry editor, navigate to the
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\
subkey. Right-click the VM service name, select New, then click Key.
Name the new subkey Parameters.

Right-click the Parameters subkey, select New, then click String Value.
Name the new value Application. Double-click the Application value and
enter the path to the vmware.exe file on your host system (put the
pathname in double quotation marks), followed by -x, followed by the
path to the VM's .vmx file (put the pathname in double quotation marks).
For my configuration, I used the string value "C:\program
files\vmware\vmware workstation\vmware.exe" -x "e:\vms\w2k1\w2k1.vmx".
Close the registry editor.

Open the Windows Services snap-in. Locate and right-click the newly
created VM service and select Properties. In the service's Properties
dialog box, click the Log On tab. Ensure that Local System account is
selected, and select the Allow service to interact with desktop check
box, which Figure 1 shows. Click OK to close the service Properties
dialog box. You can now use the Windows Services snap-in to start your
VM service. By default, the service is configured as automatic, so the
VM will start when your system starts. Each VM that you configure to run
as a service will appear in its own window on the desktop. Because the
VM is running as a service, you'll now be able to log off of your
system, and the VM will continue to run.

February 15, 2006

How to Install FreeNX on Ubuntu

1. Update your /etc/apt/sources.list with the following settings. (This is from
the Ubuntu Guide)

-----------------cut-------------------
## Add comments (##) in front of any line to remove it from being checked.
## Use the following sources.list at your own risk.
## You may replace "us" with your country code to get the closest mirror.

deb http://us.archive.ubuntu.com/ubuntu breezy main restricted
deb-src http://us.archive.ubuntu.com/ubuntu breezy main restricted

## MAJOR BUG FIX UPDATES produced after the final release
deb http://us.archive.ubuntu.com/ubuntu breezy-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu breezy-updates main restricted

## UBUNTU SECURITY UPDATES
deb http://security.ubuntu.com/ubuntu breezy-security main restricted
deb-src http://security.ubuntu.com/ubuntu breezy-security main restricted

deb http://security.ubuntu.com/ubuntu breezy-security universe
deb-src http://security.ubuntu.com/ubuntu breezy-security universe

## UNIVERSE AND MULTIVERSE REPOSITORY (Unsupported by Ubuntu. Use at own risk.)
deb http://archive.ubuntu.com/ubuntu breezy universe multiverse
deb-src http://archive.ubuntu.com/ubuntu breezy universe multiverse

## BACKPORTS REPOSITORY (Unsupported. May contain illegal packages. Use at own risk.)
deb http://archive.ubuntu.com/ubuntu breezy-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu breezy-backports main restricted universe multiverse

## PLF REPOSITORY (Unsupported. May contain illegal packages. Use at own risk.)
deb http://packages.freecontrib.org/ubuntu/plf breezy free non-free
deb-src http://packages.freecontrib.org/ubuntu/plf breezy free non-free

## For FreeNX only
deb http://seveas.ubuntulinux.nl/ breezy-seveas freenx

-----------------cut-------------------

2. apt-get update
3. apt-get install binutils
4. apt-get install ssh
5. apt-get install freenx, (choose custom keys after installation)

6. add the user you want to give access. This will copy the generated public key to the right place (.ssh/authorized_keys2) of their home directories.

This command adds the user "ubuntu"

root@vmware:/home/admin# nxserver --adduser ubuntu
NX> 100 NXSERVER - Version 1.4.0-44 OS (GPL)
NX> 1000 NXNODE - Version 1.4.0-44 OS (GPL)
NX> 716 Public key added to: /home/admin/.ssh/authorized_keys2
NX> 1001 Bye.
NX> 999 Bye

7. Now update the nxserver with the user's password
root@vmware:/home/admin# nxserver --passwd ubuntu
NX> 100 NXSERVER - Version 1.4.0-44 OS (GPL)
New password:
Password changed.
NX> 999 Bye

Once that’s been done, install an NX client, there are plenty available from the NoMachine web site.

At this point you should be able to configure your connection and you’re almost ready to go. If you try to log in it won’t let you as you need to copy the key. The custom key lives in /var/lib/nxserver/home/.ssh/client.id_dsa.key. Copy this over to your client host and import it. You should now be ready to log into a fully functional NX environment.

Reference:
http://www.snakeoillabs.com/2005/10/27/freenx-on-ubuntu-breezy-howto