Explanation of Cisco VPN Authentication mode

1. XAUTH, which really is PSK + XAUTH
2. mutual group authentication, also known as hybrid
3. certificate based authentication

so, to be more specific on the cisco side, there are three types of
phase 1/1.5 that work with the cisco road warrior ``vpndialer''
program.  You can tell which one your VPN is using by right-clicking
on its row in Connection Entries, picking Modify, and noting which of
the following three radio buttons is checked in the Authentication
tab:

 Group Authentication -- this is pre-shared key + XAUTH, where any
                         roadwarrior VPN client has enough
                         passphrases loaded into it to impersonate
                         the head-end.  The PSK is obfuscated in the
                         config file, but if you can un-rot13 it, you
                         can set up a spoof head-end and MITM nearby
                         wireless coworkers' passwords, not only
                         hijaaking your way into the VPN without a
                         password but probably also getting their
                         Master Windows Password to Everything, too,
                         thus imagineably making them LESS secure
                         than if they'd had no VPN at all.


 Mutual Group Authentication -- This uses a certificate on the
                                head-end, but the road warrior
                                presents no certificate.  Road
                                warriors validate the cert against a
                                CA certificate pubkey which you must
                                load into roadwarriors and use to
                                issue the head-end's cert, to stop
                                the MITM attack above.  It seems to
                                be un-confusing, so a lot of sites
                                probably use it.  It only works in
                                aggressive mode, though, because the
                                ``client has no identity,'' or some
                                other weird IPsec standards-ism.

                                This is probably the 'hybrid' you are
                                talking about, also known as 'hybrid
                                XAUTH'.  I understood once but am now
                                a bit rusty on how all Cisco's messy
                                configuration stanzas reference each
                                other, but have this in my notes (for
                                requesting it on PIX7.x/ASA head-end):

tunnel-group RoadWarrior ipsec-attributes
isakmp ikev1-user-authentication hybrid


 Certificate Authentication -- This uses certificates on both clients
                               and servers, and can work in main mode
                               instead of aggressive mode.  It's
                               possible to load a different cert into
                               each client and not use XAuth at all,
                               like in a site-to-site VPN.  The VPN
                               dialer supports this, but almost
                               everyone uses XAuth.

                               But some shops load all their road
                               warriors with the same cert, same
                               private key, and then use XAuth to
                               distinguish one client from another.
                               Sometimes the VPN client .zip with the
                               client cert, private key and all, is
                               available for download on some open
                               external web page.  Even with the
                               common client cert so freely
                               distributed, this behaves the same as
                               Mutual Group Authentication.  It's
                               older, and it's probably better than
                               mutual group auth / hybrid xauth.

                               upside: works in Main Mode, not as
                               cisco-proprietary.  downside: confuses
                               netadmins, fails-open on
                               misconfiguration (if you don't add
                               XAuth).  And the configuration is a
                               tangled mess.

I don't think you have to configure XAuth in their VPN dialer at all.
It pops up a box if asked.  That's it.

I don't know racoon well, but it's more likely to support Certificate
Authentication and PSK, less likely to support Mutual Group
Authentication.

There is also MTU fun.  Two IOS devices supposedly will to PMTU-D on
various kinds of tunnels including gre and ipsec.  I'm not sure PIXen
or the Windows/Mac VPNClient _ever_ do PMTU-D---in some packet dumps
they seem to punt by quietly defaulting to a small MTU like 1200 -
1300.  and I think BSD/Linux doesn't do PMTU-D either but might
confuse you by having a larger default.

source: http://mail-index.netbsd.org/current-users/2009/01/27/msg007643.html