December 19, 2006

Bind: Address Already in Use

Bind: Address Already in Use

Or How to Avoid this Error when Closing TCP Connections

Normal Closure

In order for a network connection to close, both ends have to send FIN (final) packets, which indicate they will not send any additional data, and both ends must ACK (acknowledge) each other's FIN packets. The FIN packets are initiated by the application performing a close(), a shutdown(), or an exit(). The ACKs are handled by the kernel after the close() has completed. Because of this, it is possible for the process to complete before the kernel has released the associated network resource, and this port cannot be bound to another process until the kernel has decided that it is done.

TCP State Diagram
Figure 1

Figure 1 shows all of the possible states that can occur during a normal closure, depending on the order in which things happen. Note that if you initiate closure, there is a TIME_WAIT state that is absent from the other side. This TIME_WAIT is necessary in case the ACK you sent wasn't received, or in case spurious packets show up for other reasons. I'm really not sure why this state isn't necessary on the other side, when the remote end initiates closure, but this is definitely the case. TIME_WAIT is the state that typically ties up the port for several minutes after the process has completed. The length of the associated timeout varies on different operating systems, and may be dynamic on some operating systems, however typical values are in the range of one to four minutes.

If both ends send a FIN before either end receives it, both ends will have to go through TIME_WAIT.

Normal Closure of Listen Sockets

A socket which is listening for connections can be closed immediately if there are no connections pending, and the state proceeds directly to CLOSED. If connections are pending however, FIN_WAIT_1 is entered, and a TIME_WAIT is inevitable.

Note that it is impossible to completely guarantee a clean closure here. While you can check the connections using a select() call before closure, a tiny but real possibility exists that a connection could arrive after the select() but before the close().

Abnormal Closure

If the remote application dies unexpectedly while the connection is established, the local end will have to initiate closure. In this case TIME_WAIT is unavoidable. If the remote end disappears due to a network failure, or the remote machine reboots (both are rare), the local port will be tied up until each state times out. Worse, some older operating do not implement a timeout for FIN_WAIT_2, and it is possible to get stuck there forever, in which case restarting your server could require a reboot.

If the local application dies while a connection is active, the port will be tied up in TIME_WAIT. This is also true if the application dies while a connection is pending.

Strategies for Avoidance

SO_REUSEADDR

You can use setsockopt() to set the SO_REUSEADDR socket option, which explicitly allows a process to bind to a port which remains in TIME_WAIT (it still only allows a single process to be bound to that port). This is the both the simplest and the most effective option for reducing the "address already in use" error.

Oddly, using SO_REUSEADDR can actually lead to more difficult "address already in use" errors. SO_REUSADDR permits you to use a port that is stuck in TIME_WAIT, but you still can not use that port to establish a connection to the last place it connected to. What? Suppose I pick local port 1010, and connect to foobar.com port 300, and then close locally, leaving that port in TIME_WAIT. I can reuse local port 1010 right away to connect to anywhere except for foobar.com port 300.

A situation where this might be a problem is if my program is trying to find a reserved local port (<>SO_REUSADDR, then each time I run the program on my machine, I'll keep getting the same local reserved port, even if it is stuck in TIME_WAIT, and I risk getting a "connect: Address already in use" error if I go back to any place I've been to in the last few minutes. The solution here is to avoid SO_REUSEADDR.

Some folks don't like SO_REUSEADDR because it has a security stigma attached to it. On some operating systems it allows the same port to be used with a different address on the same machine by different processes at the same time. This is a problem because most servers bind to the port, but they don't bind to a specific address, instead they use INADDR_ANY (this is why things show up in netstat output as *.8080). So if the server is bound to *.8080, another malicious user on the local machine can bind to local-machine.8080, which will intercept all of your connections since it is more specific. This is only a problem on multi-user machines that don't have restricted logins, it is NOT a vulnerability from outside the machine. And it is easily avoided by binding your server to the machine's address.

Additionally, others don't like that a busy server may have hundreds or thousands of these TIME_WAIT sockets stacking up and using kernel resources. For these reasons, there's another option for avoiding this problem.

Client Closes First

Looking at the diagram above, it is clear that TIME_WAIT can be avoided if the remote end initiates the closure. So the server can avoid problems by letting the client close first. The application protocol must be designed so that the client knows when to close. The server can safely close in response to an EOF from the client, however it will also need to set a timeout when it is expecting an EOF in case the client has left the network ungracefully. In many cases simply waiting a few seconds before the server closes will be adequate.

It probably makes more sense to call this method "Remote Closes First", because otherwise it depends on what you are calling the client and the server. If you are developing some system where a cluster of client programs sit on one machine and contact a variety of different servers, then you would want to foist the responsibility for closure onto the servers, to protect the resources on the client.

For example, I wrote a script that uses rsh to contact all of the machines on our network, and it does it in parallel, keeping some number of connections open at all times. rsh source ports are arbitrary available ports less than 1024. I initially used "rsh -n", which it turns out causes the local end to close first. After a few tests, every single free port less than 1024 was stuck in TIME_WAIT and I couldn't proceed. Removing the "-n" option causes the remote (server) end to close first (understanding why is left as an exercise for the reader), and should've eliminated the TIME_WAIT problem. However, without the -n, rsh can hang waiting for input. And, if you close input at the local end, this can again result in the port going into TIME_WAIT. I ended up avoiding the system-installed rsh program, and developing my own implementation in perl. My current implementation, multi-rsh, is available for download

Reduce Timeout

If (for whatever reason) neither of these options works for you, it may also be possible to shorten the timeout associated with TIME_WAIT. Whether this is possible and how it should be accomplished depends on the operating system you are using. Also, making this timeout too short could have negative side-effects, particularly in lossy or congested networks.

In Linux, issue the following command to set the timeout_timewait parameter to 30 seconds:
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

From: http://hea-www.harvard.edu/~fine/Tech/addrinuse.html

November 13, 2006

Search Engine-Friendly URLs

Search Engine-Friendly URLs

By Chris Beasley
August 10th 2001

On today’s Internet, database driven or dynamic sites are very popular. Unfortunately the easiest way to pass information between your pages is with a query string. In case you don’t know what a query string is, it's a string of information tacked onto the end of a URL after a question mark.

So, what’s the problem with that? Well, most search engines (with a few exceptions - namely Google) will not index any pages that have a question mark or other character (like an ampersand or equals sign) in the URL. So all of those popular dynamic sites out there aren’t being indexed - and what good is a site if no one can find it?

The solution? Search engine friendly URLs. There are a few popular ways to pass information to your pages without the use of a query string, so that search engines will still index those individual pages. I'll cover 3 of these techniques in this article. All 3 work in PHP [1] with Apache [2] on Linux [3] (and while they may work in other scenarios, I cannot confirm that they do).

Method 1: PATH_INFO

Implementation:

If you look above this article on the address bar, you’ll see a URL like this: http://www.webmasterbase.com/article.php/999/12. SitePoint actually uses the PATH_INFO method to create their dynamic pages.

Apache has a "look back" feature that scans backwards down the URL if it doesn’t find what it's looking for. In this case there is no directory or file called "12", so it looks for "999". But it find that there's not a directory or file called "999" either, so Apache continues to look down the URL and sees "article.php". This file does exist, so Apache calls up that script. Apache also has a global variable called $PATH_INFO that is created on every HTTP request. What this variable contains is the script that's being called, and everything to the right of that information in the URL. So in the example we've been using, $PATH_INFO will contain article.php/999/12.

So, you wonder, how do I query my database using article.php/999/12? First you have to split this into variables you can use. And you can do that using PHP’s explode function:

$var_array = explode("/",$PATH_INFO);

Once you do that, you’ll have the following information:

$var_array[0] = "article.php"

$var_array[1] = 999

$var_array[2] = 12

So you can rename $var_array[1] as $article and $var_array[2] as $page_num and query your database.

Drawback:

There was previously one major drawback to this method. Google, and perhaps other search engines, would not index pages set up in this manner, as they interpreted the URL as being malformed. I contacted a Software Developer at Google and made them aware of the problem and I am happy to announce that it is now fixed.

There is the potential that other search engines may ignore pages set up in this manner. While I don't know of any, I can't be certain that none do. If you do decide to use this method, be sure to monitor your server logs for spiders to ensure that your site is being indexed as it should.

Method 2: .htaccess Error Pages

Implementation:

The second method involves using the .htaccess file. If you're new to it, .htaccess is a file used to administer Apache access options for whichever directory you place it in. The server administrator has a better method of doing this using his or her configuration files, but since most of us don't own our own server, we don't have control over what the server administrator does. Now, the server admin can configure what users can do with their .htaccess file so this approach may not work on your particular server, however in most cases it will. If it doesn't, you should contact your server administrator.

This method takes advantage of .htaccess’ ability to do error handling. In the .htaccess file in whichever directory you wish to apply this method to, simply insert the following line:

ErrorDocument 404 /processor.php

Now make a script called processor.php and put it in that same directory, and you're done! Lets say you have the following URL: http://www.domain.com/directory/999/12/. And again in this example "999" and "12" do not exist, however, as you don't specify a script anywhere in the directory path, Apache will create a 404 error. Instead of sending a generic 404 header back to the browser, Apache sees the ErrorDocument command in the .htaccess file and calls up processor.php.

Now, in the first example we used the $PATH_INFO variable, but that won’t work this time. Instead we need to use the $REQUEST_URI variable, which contains everything in the URL after the domain. So in this case, it contains: /directory/999/12/.

The first thing you need to do in processor.php is send a new HTTP header. Remember, Apache thought this was a 404 error, so it wants to tell the browser that it couldn’t find a page.

So, put the following line in your processor.php:

header("HTTP/1.1 200 OK");

At this time I need to point out an important fact. In the first example you could specify what script processed your URL. In this example all URLs must be processed by the same script, processor.php, which makes things a little different. Instead of creating different URLs based on what you want to do, such as article.php/999/12 or printarticle.php/999/12 you only have 1 script that must do both.

So you must decide what to do based on the information processor.php receives - more specifically, by counting how many parameters are passed. For instance on my site [4], I use this method to generate my pages: I know that if there's just one parameter, such as http://www.online-literature.com/shakespeare/, that I need to load an author information page; if there are 2 parameters, such as http://www.online-literature.com/shakespeare/hamlet/, I know that I need to load a book information page; and finally if there are 3 parameters, such as http://www.online-literature.com/shakespeare/hamlet/3/, I know I need to load a chapter viewing page. Alternatively, you can simply use the first parameter to indicate the type of page to display, and then process the remaining parameters based on that.

There are 2 ways you can accomplish this task of counting parameters. First you need to use PHP’s explode function to divide up the $REQUEST_URI variable. So if $REQUEST_URI = /shakespeare/hamlet/3/:

$var_array = explode("/",$REQUEST_URI);

Now note that, because of the positioning of the /’ there are actually 5 elements in this array [5]. The first element, element 0, is blank, because it contains the information before the first /. The fifth element, element 4, is also blank, because it contains the information after the last /.

So now we need to count the elements in our $var_array. PHP has two functions that let us do this. We can use the sizeof() function as in this example:

$num = sizeof($var_array); // 5

You’ll notice that the sizeof() function counts every item in the array regardless of whether it's empty. The other function is count(), which is an alias for the sizeof() function.

Some search engines, like AOL, will automatically remove the trailing / from your URL, and this can cause problems if you’re using these functions to count your array. For instance http://www.online-literature.com/shakespeare/hamlet/ becomes http://www.online-literature.com/shakespeare/hamlet, and as there are 3 total elements in that array our processor.php would load an author page instead of a book page.

The solution is to create a function that will count only the elements in an array that actually hold data. This will allow you to leave off the ending / or allow any links from AOL'’s search engine to point to the proper place. An example of such a function is:

function count_all($arg)
{
// skip if argument is empty
if ($arg) {
// not an array, return 1 (base case)
if(!is_array($arg))
return 1;
// else call recursively for all elements $arg
foreach($arg as $key => $val)
$count += count_all($val);
return $count;
}
}

To get your count, access the function like this:

$num = count_all($url_array);

Once you know how many parameters you need, you can define them like this:

$author=$var_array[1];
$book=$var_array[2];
$chapter=$var_array[3];

Then you can use includes to call up the appropriate script, which will query your database and set up your page. Also if you get a result you’re not expecting, you can simply create your own error page for display to the browser.

Drawback:

The drawback of this method is that every page that's hit is seen by Apache as an error. Thus every hit creates another entry in your server error logs, which effectively destroys their usefulness. So if you use this method you sacrifice your error logs.

Method 3: The ForceType Directive

Implementation:

You'll recall that the thing that trips up Google, and maybe even other search engines, when using the PATH_INFO method is the period in the middle of the URL. So what if there was a way to use that method without the period? Guess what? There is! Its achieved using Apache'’s ForceType directive.

The ForceType directive allows you to override any default MIME types you have set up. Usually it may be used to parse an HTML [6] page as PHP or something similar, but in this case we will use it to parse a file with no extension as PHP.

So instead of using article.php, as we did in method 1, rename that file to just "article". You will then be able to access it like this: http://www.domain.com/article/999/12/, utilizing Apache's look back feature and PATH_INFO variable as described in method 1. But now, Apache doesn’t know to that "article" needs to be parsed as php. To tell it that, you must add the following to your .htaccess file.


ForceType application/x-httpd-php

This is known as a "container". Instead of applying directives to all files, Apache allows you to limit them by filename, location, or directory. You need to create a container as above and place the directives inside it. In this case we use a file container, we identify “article” as the file we're concerned with, and then we list the directives we want applied to this file before closing off the container.

By placing the directive inside the container, we tell Apache to parse "article" as a PHP script even though it has no file extension. This allows us to get rid of the period in the URL that causes the problems, and yet still use the PATH_INFO method to manage our site.

Drawback:

The only drawback to this method as compared with method 2 is that your URLs will be slightly longer. For instance, if I were to use this method on my site, I'd have to use URLs like this: http://www.online-literature.com/ol/homer/odyssey/ instead of http://www.online-literature.com/homer/odyssey/. However if you had a site like SitePoint and used this method it wouldn't be such a problem, as the URL (http://www.SitePoint.com/article/755/12/) would make more sense.

Conclusion

I have outlined 3 methods of making search engine friendly URLs - along with their drawbacks. Obviously, you should evaluate these drawbacks before deciding which method to implement. And if you have any questions about the implementation of these techniques, they are oft-discussed topics on the SitePoint Forums [7] so just stop in and make a post.

November 3, 2006

Use Wget to find site broken links

wget --recursive -nd -nv --delete-after --domains=domain.com http://domain.com/ | tee wget.out 2>&1
Got it from: http://www.shallowsky.com/blog/tech/web/webstats.html

September 16, 2006

菜品走味怎么办

放有辣椒的菜太辣时或炒辣椒时加点醋,辣味大减。

烹调时,放酱油若错倒了食醋,可撒放少许小苏打,醋味即可消除。

菜太酸,将一只松花蛋捣烂放入。

菜太辣,放一只鸡蛋同炒。

菜太苦,滴入少许白醋。

汤太咸又不宜兑水时,可放几块豆腐或土豆或几片蕃茄到汤中;也可将一把米或面粉用布包起来放入汤中。

汤太腻,将少量紫菜在火上烤一下,然后撒入汤中。

September 2, 2006

John Reed Website, worth a read for investment

http://www.johntreed.com/

The BusinessWeek Best-Sellers of 2003

The top-selling business books as reported January through December, 2003

Paperback Business Books


1. FAST FOOD NATION By Eric Schlosser (HarperCollins -- $13.95) The bad news on burgers and fries, by an Atlantic Monthly writer.

2. NICKEL AND DIMED By Barbara Ehrenreich (Owl Books -- $13) How the working poor struggle to make ends meet.

3. THE TIPPING POINT By Malcolm Gladwell (Back Bay -- $14.95) What turns an idea into a hot trend, by a New Yorker writer.

4. THE E-MYTH REVISITED By Michael E. Gerber (HarperBusiness -- $16) A systems approach for small-business success.

5. WHAT COLOR IS YOUR PARACHUTE? By Richard Nelson Bolles (Ten Speed Press -- $17.95) The 2003 edition of the enduring job-search bible.

6. REAL ESTATE RICHES By Dolf de Roos (Warner -- $17.95) Why buildings are better investments than stocks.

7. REAL ESTATE LOOPHOLES By Diane Kennedy, C.P.A., and Garrett Sutton, Esq. (Warner -- $16.95) Tax and legal knowhow.

8. THE RICHEST MAN IN BABYLON By George S. Clason (Signet -- $6.99) Save 10% of what you earn, then pay your bills.

9. RICH DAD'S RETIRE YOUNG, RETIRE RICH By Robert T. Kiyosaki, with Sharon L. Lechter, C.P.A. (Warner -- $17.95) Plan ahead.

10. THE ERNST & YOUNG TAX GUIDE 2003 By Ernst & Young, LLP (Wiley -- $16.95) Dig out those W2s and 1099s.

11. EFFECTIVE PHRASES FOR PERFORMANCE APPRAISALS By James E. Neal Jr. (Neal Publications -- $10.95) How about: ``attaboy''?

12. THE INTELLIGENT INVESTOR, REVISED EDITION By Benjamin Graham, with Jason Zweig (HarperBusiness -- $19.95) The classic explanation of ``value investing.''

13. J.K. LASSER'S YOUR INCOME TAX 2003 By the J.K. Lasser Institute (Wiley -- $16.95) Home-office deductions, the alternative minimum tax, and other traps for the unwary.

14. IT'S NOT HOW GOOD YOU ARE, IT'S HOW GOOD YOU WANT TO BE By Paul Arden (Phaidon -- $7.95) An ad man's aphorisms on success.

15. THE POWER OF FOCUS By Jack Canfield, Mark Victor Hansen, and Les Hewitt (Health Communications -- $12.95) Zoning in on achievement.

Hardcover Business Books


1. GOOD TO GREAT By Jim Collins (HarperBusiness -- $27.50) How run-of-the-mill companies make the leap to excellence.

2. EXECUTION By Larry Bossidy and Ram Charan (Crown Business -- $27.50) Translating business strategies into results.

3. WHAT SHOULD I DO WITH MY LIFE? By Po Bronson (Random House -- $24.95) Overcoming confusion to find your true calling.

4. THE GREAT UNRAVELING By Paul Krugman (Norton -- $25.95) A Princeton University economist takes on the Bush Administration.

5. LEADERSHIP By Rudolph W. Giuliani (Talk Miramax -- $25.95) Hizzoner speaks.

6. THE PRESENT By Spencer Johnson (Doubleday -- $19.95) The pursuit of happiness and success, as described in a fable.

7. THE ONE MINUTE MILLIONAIRE By Mark Victor Hansen and Robert G. Allen (Harmony Books -- $21) Chicken soup for the investor.

8. REEFER MADNESS By Eric Schlosser (Houghton Mifflin -- $23) The underground economy in porn, pot, and migrant labor.

9. THE FIVE DYSFUNCTIONS OF A TEAM By Patrick Lencioni (Jossey-Bass -- $22) Overcoming behavior that obstructs teamwork.

10. TOTAL MONEY MAKEOVER By Dave Ramsey (Thomas Nelson -- $24.99) Getting rid of debt and building up your reserves.

11. THE POWER OF FULL ENGAGEMENT By Jim Loehr and Tony Schwartz (Free Press -- $26) Apply the rigors of the gym to your work and personal life.

12. THE LAWS OF MONEY, THE LESSONS OF LIFE By Suze Orman (Free Press -- $26) More maxims for prospering, from the queen of personal finance.

13. IN AN UNCERTAIN WORLD By Robert E. Rubin and Jacob Weisberg (Random House -- $35) The former Treasury Secretary tells why Clintonomics worked.

14. PURPLE COW By Seth Godin (Portfolio -- $19.95) Emulate Krispy Kreme and Dutch Boy paints, and astonish your customers.

15. RE-IMAGINE! By Tom Peters (DK Publishing -- $30) ``Think beautiful...think weird'' and other instructions, in an eye-popping format.

BusinessWeek's Best-Seller List is based on a survey of chain and independent booksellers that carry a broad selection of books on economics, management, sales and marketing, small business, investing, personal finance, and careers. Well over 1,000 retail outlets nationwide are represented. Current rankings are based on a weighted analysis of unit sales.

August 20, 2006

腌黄瓜

独门秘技腌黄瓜——酸,甜,辣,爽!

偶经常去前门的都一处烧麦馆,到不是贪恋里边的烧麦,而是钟情于里边的酸辣黄瓜条,自认为虽然价格贵了点(五快钱一小碟,直径不超过十厘米的那种碟子,而 且也只是码放着寥寥数根而已),但是味道超级爽口,舔中带酸,还伴着淡淡的辣味,冰冰凉凉清爽的不得了,在别的饭店从未吃到过。于是今天就琢磨着在家里自 己做了一盘,没想到效果非常好,和店里的味道一模一样,晚餐的时候摆上桌得到了老公的强烈支持!

现在就把方法介绍给大家,有兴趣的朋友不妨一试。

主料:鲜嫩黄瓜两至三根(一定要顶花带刺的做出来才爽口哦)

调料:A.盐一勺

B.白醋五勺,冰糖一块(核桃大小),蜂蜜一勺,味精小半勺,干红辣椒四个(掰成小段)

制作方法:1,将黄瓜洗净,切成大约八厘米长手指头粗细的小段,放在干净容器内用盐搅拌均匀,待用。

2,将B里面的材料一起放进一个小碗里边,搅拌均匀,并放置到冰糖完全溶解。

3,黄瓜用盐腌二十分钟左右会渗出很多水分,把这些水分倒掉,并将刚才调好的B调味料到入黄瓜中搅拌均匀。

4,放置冰箱中冷藏一天,即可。

特点:酸中带甜,辣味适中,冰爽可口,开胃消食。

需要注意的几点:1,一定要用冰糖和白醋,不要用白糖和米醋/陈醋,否则作不出那种清爽的味道。

2,用盐腌黄瓜的时候,盐的用量切勿过多,否则会影响最后清甜的口感。

3,此道凉菜放置在冰箱中腌制,所以特别适合夏天食用,根据我在都一处的经验,在保证卫生的情况下腌至的时间如果超过两天效果会更好。如果只腌一天,也已经很不错了。

4,和别的凉菜不同的是,这道凉菜不需要放香油,以强调不带一滴油脂的清爽感。


August 9, 2006

GVIM 7.0 and netrw

Recently I installed gvim7.0 on my windows XP and I am happy with the new spell check feature. However, I can not remotely write to a ftp file anymore by using "e ftp://www.abc.com/file.txt". I used to be able to do that with gvim6.3. The issues lies in the netrw vim plugin.

To solve this problem, I had to get to the netrw.vim plugin webpage (http://www.vim.org/scripts/script.php?script_id=1075) and install the latest version (version 102).

To install it, you have to gunzip it first. This is insane because gunzip is not native to Windows. Although I do have it on my machine through unixutils, I don't think many other Windows XPs have it.

Anyway, after installing the latest version, the problem seems to be fixed. I miss the old good days that vim (and gvim) just works out of box.

August 1, 2006

mail-to-blog for wordpress

http://www.economysizegeek.com/?page_id=395 has a good plugin (named postie) for wordpress to support mail-to-blog, supporting picture and attachment.

June 29, 2006

Pregnancy

My wife might be pregnant yesterday. At least according to the testing tick we bought. How wonderful that is! We are being cautious and will wait for a few more days to confirm it before announcing it. :-) Sweet. Shu......, Keept it a serect!

iLBC Codec

iLBC is the Best Codec over IP network.

It is the codec used by Skype and many other IM clients. It is open, FREE, and a standard.

Find more about it, incuding the the source code of a reference implementation at:

http://www.ilbcfreeware.org/index.html

June 28, 2006

CSS, List and Float

Very nice tutorial and examples on CSS, List and Float
http://css.maxdesign.com.au/floatutorial/

June 27, 2006

SNMP On Linux

To Add a mib file to the tool, do the following:
       
mkdir $HOME/.snmp
mkdir $HOME/.snmp/mibs
cp MY-MIB.txt $HOME/.snmp/mibs

And Then Add the following lines to the file ~/.snmp/snmp.conf

 mibs +MY-MIB 


Note MY-MIB here is the MIB Module name, not the Mib file name.

You can use the following command to list your new mib

 snmptranslate -Tp -IR ModuleName

June 26, 2006

Fix for VIM7 HTML Syntax File

A minor bug in the default VIM7 html syntax file is that spell check is not applied to paragraphs. Adding the following line to line 157 fixes that:


Write using HTML and CSS

Using HTML and CSS to write a book or an article
http://www.alistapart.com/articles/boom

June 22, 2006

Power Talk

Watched a video seminar on how to communicate better by George Walther.

I found the summary sheeet on his website very useful:
http://www.georgewalther.com/images/50WaystoSayWhatYouMean.pdf

June 16, 2006