December 18, 2014

Install a minimal X on Linux

apt-get install xinit i3 rxvt

i3 is the lightweight window manager
rxvt is the terminal
xinit gives you X windows and the famous "startx" command

To make i3 use your full screen resolution, create the file ~/.xinitrc and put the following there

xrandr --output Virtual1 --mode 1680x1050
exec i3


You may need to change the name "Virtual1" to something else. Use xrandr to list all the known windows (you first need to have X windows  running though)

December 9, 2014

p11tool, gnutls and PIV CAC card

p11tool --list-all-certs

p11tool --login --export "pkcs11:model=PKCS%2315%20emulated;manufacturer=piv_II;serial=36889385781093f6;token=PIV_II%20%28PIV%20Card%20Holder%20pin%29;id=%02;object=Certificate%20for%20Digital%20Signature;object-type=cert" > /tmp/02.cert

p11tool --list-all-privkeys --login



x509 certificate subject name and OID

In a X509 certificate, there is always a subject name like the following:

$ openssl x509 -in user-cert.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1373122324 (0x51d82f14)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=CA
        Validity
            Not Before: Jul  6 14:52:05 2013 GMT
            Not After : May 15 14:52:05 2023 GMT
        Subject: UID=test,CN=A user

Inside the subject line, there can be multiple subparts, such as CN=xxx, DC=xxxx, UID=xxx, OU=xxx, C=xxx, ... Each subpart is represented in the certificate as an OID that is globally unique and registred with IETF. For example, the OID of CN is 2.5.4.3, and the OID of UID is 0.9.2342.19200300.100.1.1. How are we supposed to find out the OID? Openssl provides a command option for just.  

openssl x509 -in user-cert.pem -text -noout -nameopt RFC2253,oid

This command will print out the cert with the OID=xxx instead of CN=xxx.

December 8, 2014

cross compile openconnect

Openconnect is a nice open source SSL VPN client for Cisco AnyConnect, and also for the open source SSL vpn server ocserv (hosted on the same website as openconnect). Below are some tips on how to cross compile openconnect for ARM, with GnuTLS

Openconnect works with both Openssl and GnuTLS. However, to use hardware token (smart card, etc), you will need GnuTLS.

Dependencies:

Openconnect depends on GnuTLS (3.3.9)
GnuTLS depends on libnettle, and libhogweed 2.7.1 (both in the nettle package)
libnettle depends on gnu GMP (libgmp, version 6.0.0)

To use hardware token, GnuTLS also depends on p11-kit (version 0.22.1) and pcsc-lite (version 1.8.11), and opensc (0.14.0), which depends on pcsc-lite.

All of these packages support autoconfig so that one can run "configure" to generate the makefile(s).  We use the --prefix "/opt/ncs-install" to install all packages. Below are the customized "configure" scripts for each package:

p11-kit-0.22.1:
CC=arm-none-linux-gnueabi-gcc CXX=arm-none-linux-gnueabi-g++ ./configure --host=arm-linux --prefix=/opt/ncs-install \
        --without-libffi --without-libtasn1

gmp-6.0.0:
CC=arm-none-linux-gnueabi-gcc CXX=arm-none-linux-gnueabi-g++ ./configure --host=arm-linux --prefix=/opt/ncs-install

nettle-2.7.1:
CFLAGS=-I/opt/ncs-install/include LDFLAGS=-L/opt/ncs-install/lib CC=arm-none-linux-gnueabi-gcc CXX=arm-none-linux-gnueabi-g++ ./configure --host=arm-linux --prefix=/opt/ncs-install

pcsc-lite-1.8.11:
CFLAGS="-I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include/"
CC=arm-none-linux-gnueabi-gcc ./configure -host=arm-linux  --disable-libudev --enable-libusb \
        LIBUSB_CFLAGS="-I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include/libusb-1.0/ -L/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/lib"  \
        LIBUSB_LIBS="-lusb-1.0"

/opensc-0.14.0:
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
export LTLIB_LIBS="-L/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/lib/ -lltdl"
export PCSC_CFLAGS="-I$DIR/../pcsc-lite-1.8.11/src/PCSC"
export LIBTOOL_SYSROOT_PATH=/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/
export CFLAGS="-I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include -L/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/lib"
export LDFLAGS="-lcrypto"
export CC=arm-none-linux-gnueabi-gcc
./configure -host=arm-linux  -v

gnutls-3.3.9:
CFLAGS=-I/opt/ncs-install/include LDFLAGS=-L/opt/ncs-install/lib \
ZLIB_CFLAGS=-I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include/ \
ZLIB_LIBS="-L/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/lib -lz" \
CC=arm-none-linux-gnueabi-gcc CXX=arm-none-linux-gnueabi-g++ ./configure --host=arm-linux --prefix=/opt/ncs-install \
 --with-nettle-mini --disable-crywrap \
 --with-p11-kit \
 --with-default-trust-store-file=/etc/ssl/certs/ca-certificates.crt


openconnect-7.00:
LIBPCSCLITE_CFLAGS=-I/opt/ncs-install/include/PCSC/ \
LIBPCSCLITE_LIBS="-L/opt/ncs-install/lib -lpcsclite" \
LIBXML2_CFLAGS=-I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include/libxml2/ \
LIBXML2_LIBS="-L/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/lib -lxml2" \
CFLAGS=-I/opt/ncs-install/include LDFLAGS=-L/opt/ncs-install/lib  \
LDFLAGS="-L/opt/ncs-install/lib -lp11-kit -lnettle -lhogweed -lgmp" \
ZLIB_CFLAGS=-I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include/ \
ZLIB_LIBS="-L/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/lib -lz" \
CC=arm-none-linux-gnueabi-gcc ./configure --prefix=/opt/install --disable-nls --host=arm-linux --without-openssl --with-gnutls


December 4, 2014

how to verify certificate signed by intermediate CA

 openssl verify -untrusted intermediate-ca.pem your-cert.pem

Put the list of intermediate CA (in PEM format, concatenated ) in intermediate-ca.pem, and use the "-untrusted" option. That name tricked me initially, and that's the one to use. 

The above command is to use the system CA list to verify the cert. If you have your own CA, just use the option "-CAfile your-ca.pem".

November 25, 2014

How to make IE not cache your golang web app responses

var w http.ResponseWriter
w.Header().Set("cache-control", "priviate, max-age=0, no-cache")
w.Header().Set("pragma", "no-cache")
w.Header().Set("expires", "-1")

November 24, 2014

Clear windows event log

Run this following line in PowerShell as Admin:

wevtutil el | Foreach-Object {Write-Host "Clearing $_"; wevtutil cl "$_"}

November 19, 2014

Read godoc in vim using Shift K

Add the following to your ~/.vim/ftplugin/go.vim (the key here is to use <cfile> not <cword>)

fun! ReadMan()
  " Assign current file under cursor to a script variable:
  let s:man_word = expand('<cfile>')
  :exe ":Godoc " . s:man_word
endfun
" Map the K key to the ReadMan function:
map K :call ReadMan()<CR>

=======UPDATE========
Had a better way for this:

1. Create a file in your ~/bin called godoc.sh with the following content:
#!/bin/sh
fullpkg=`gawk -v "cmd=$1" '
{ FS="/" }
{
        if ($NF==cmd || $0 ==cmd){
                printf("%s",$0);
                exit;
        }
}' ~/bin/gopkg-list.txt`
echo -n $fullpkg

2. Create a text file in ~/bin/gopkg-list.txt with the following contents:
log/syslog
unicode/utf8
unicode/utf16
testing
hash/crc64
hash/adler32
hash/fnv
hash/crc32
archive/zip
archive/tar
expvar
mime
reflect
crypto/des
crypto/sha1
crypto/subtle
crypto/aes
crypto/md5
crypto/cipher
crypto/rand
crypto/ecdsa
crypto/tls
crypto/sha512
crypto/x509
crypto/sha256
crypto/elliptic
crypto/hmac
crypto/rsa
crypto/x509/pkix
crypto/rc4
crypto/dsa
flag
strings
sync
os/user
os/signal
os/exec
os
fmt
syscall
time
runtime/pprof
runtime/cgo
runtime/debug
runtime/race
net/http/httputil
net/http/cookiejar
net/http/fcgi
net/http/pprof
net/http/cgi
net/http/httptest
net/rpc/jsonrpc
net/mail
net/http
net/textproto
net/url
net/rpc
net/smtp
io/ioutil
math
database/sql/driver
database/sql
mime/multipart
bytes
errors
sort
bufio
encoding
log
path/filepath
index/suffixarray
net
crypto
sync/atomic
go/printer
go/doc
go/scanner
go/ast
go/parser
go/token
go/build
go/format
unicode
image
io
runtime
html/template
regexp
html
container/ring
container/list
container/heap
hash
path
regexp/syntax
text/template/parse
text/tabwriter
text/template
text/scanner
strconv
testing/quick
testing/iotest
compress/flate
compress/zlib
compress/lzw
compress/bzip2
compress/gzip
image/color
image/draw
image/color/palette
image/png
image/gif
image/jpeg
encoding/base64
encoding/base32
encoding/xml
encoding/asn1
encoding/pem
encoding/ascii85
encoding/csv
encoding/hex
encoding/json
encoding/binary
encoding/gob
debug/pe
debug/plan9obj
debug/dwarf
debug/elf
debug/gosym
debug/macho
math/rand
math/big
math/cmplx

3. In your ~/.vim/ftplugin/go.vim, add the following:
fun! ReadMan()
  " Assign current file under cursor to a script variable:
  let s:man_word = expand('<cword>')
  let s:full_word = system('~/bin/godoc.sh '. s:man_word)
  :exe ":Godoc " . s:full_word
endfun
" Map the K key to the ReadMan function:
map K :call ReadMan()<CR>

This will take the keyword under cursor, expanded using the shell script, and call :Godoc on it. This takes care of subpackages

November 14, 2014

ios validate self signed certificate

http://stackoverflow.com/questions/10979922/ios-and-ssl-unable-to-validate-self-signed-server-certificate

Quote:


I did figure out how to resolve this issue.
I ended up comparing the client and server trust certificates, byte-by-byte. Although there could be another way to resolve such issues of self-signed certificate, but for this solution did work. Here is how I'm doing comparison of the client and server certificates, byte-by-byte, using their CFData objects(you can also reference 'AdvancedURLConnections' example code provided by Apple):
success = NO;
        pServerCert = SecTrustGetLeafCertificate(trust);
        if (clientCert != NULL) {
            CFDataRef       clientCertData;
            CFDataRef       serverCertData;

            clientCertData = SecCertificateCopyData(clientCert);
            serverCertData   = SecCertificateCopyData(pServerCert);

            assert(clientCertData != NULL);
            assert(serverCertData   != NULL);

            success = CFEqual(clientCertData, serverCertData);

            CFRelease(clientCertData);
            CFRelease(serverCertData);
        }
        if (success) {
            [[challenge sender] useCredential:credential forAuthenticationChallenge:challenge];
            [self printLogToConsole:@"Success! Trust validation successful."];
        } else {
            [self printLogToConsole:@"Failed! Trust evaluation failed for service root certificate.\n"];
            [[challenge sender] cancelAuthenticationChallenge:challenge];
        }
Hope this will help someone, who is looking for solution of similar issue,

November 12, 2014

the ffmpeg command to explode video into still images

ffmpeg.exe -i IMG_0278.MOV -ss 0  -t 1 -f image2 -sameq -vcodec mjpeg "img-%03d.jpg"

-ss: start time, 0 means starting at 0 second mark
-t : number of seconds to explode


November 6, 2014

vim and error list

If you have text file that contains the list of errors with line positions, you can tell vim to use that to help you navigate through the list.

1. Edit the list to the following format (suppose it is named "errors.txt")
    xxxx.c:123: Warning/Error/Info rest-of-the-text
    where
    xxxx.c is the source code file name
    123 is the line number

2. vim -q errors.txt

3. now you can type ":cw" to open the "quickfix" window which list the errors, and use ":cn" and ":cp" to go through them. Mapping these to a short cut key will make life easier. I map mine to Ctrl-j and Ctrl-k for down and up (copy and the paste the following to your ~/.vimrc file)

    map <C-J> :cn<CR>
    map <C-K> :cp<CR>

November 5, 2014

check ip address owner

For an U.S IP address, you can go to the following site:

http://whois.arin.net

and put the IP address in the search box of top right corner, and you can get more info on it.

Once you find the owner, you can click on "Related Networks" to find all the IP addresses that owner owns.

November 3, 2014

How to dump Mac OS X dns cache

DNS cache information is not managed by Directory Services, contrary to popular opinion on this site. It's managed by mDNSResponder, and the man page contains the answer to your question:


A SIGINFO signal will dump a snapshot summary of the internal state to /var/log/system.log:

   % sudo killall -INFO mDNSResponder

October 31, 2014

Windows Active Directory (AD) LDAP binding account

The common way to bind to AD as an LDAP server is to use "Distinguished Name(DN)" and Password. The DN usually is in the form of:  CN=username,CN=Users,DC=yourdomain,DC=com

To find out DN, you can use Sysinternal's Active Directory Explorer to connect to AD and browse to the user to find out.

According to this post on stackoverflow, http://serverfault.com/questions/497368/ldap-activedirectory-binddn-syntax, you can also use UPN, which typically has the value of:

<sAMAccountName>@<domain FQDN>


Linux Frame buffer screen capture

1. Calculate the screen size. For example 1920 x 1080. Make sure that the width is a multiple of 4. For example, for HD screen resolution of 1366x768,  use 1368x768

2. dd if=/dev/fb1 bs=1368 count=3072 | gzip -c > screen.bgra.gz
   Each pixel is 4 byte, B-G-R-A. So 1368x768 resolution takes 1368*768*4=1368*3072 bytes

3. transfer the file to a Linux computer with imagemagick installed (needs a version released after 2010), and then convert it:
   gunzip screen.bgra.gz
   convert -size 1368x768 -depth 8 screen.bgra screen.png

Kerberos simplified

The 6 steps of Kerberos

Players:
  User
  Service (Can be any service that user wants to access, such as mail service)
  Kerberos Authentication Service (AS)
  Kerberos Ticket Granting Service (TGS)
  

                                      +--------------------+----------------------+
                                      |                    |                      |
                                      |     Kerberos AS    |     Kerberos TGS     |
                                      |                    |                      |
                                      +---------+----------+--------+-------------+
                                         ^      |              ^    |
                                         |      |              |    |
                                       1 |      |2           3 |    | 4
                                         |      |              |    |
                                         |      |              |    |
      +-------------------+              |      |              |    |                       +-----------------------+
      |                   |--------------+      |              |    |                       |                       |
      |                   |<--------------------+              |    |                       |                       |
      |                   +------------------------------------+    |                       |                       |
      |      User         |<----------------------------------------+         5             |     Mail Service      |
      |                   +---------------------------------------------------------------->|                       |
      |                   |<----------------------------------------------------------------+                       |
      +-------------------+                                                   6             +-----------------------+



Step 1: User          ---- Username, Timestamp                                               ----> Kerberos AS
Step 2: Kerberos AS   ---- TGT=[K(user,tgs)<-P(tgs)], K(user,tgs)<-P(user)                   ----> User
Step 3: User          ---- TGT<-P(tgs), user_name, service_name, authenticator<-K(user,tgs)  ----> Kerberos TGS
Step 4: Kerberos TGS  ---- ST=[K(user,service)<-P(service)], K(user,service)<-K(user,tgs)    ----> User
Step 5: User          ---- ST<-P(service), user_name, authenticator<-K(user,service)         ----> Service
Step 6: Service       ---- OK, authenticator<-K(user,service)                                ----> User


Authenticator = (sender_name, sender_address, timestamp, lifespan) <- SessionKey

K(user,tgs) : a session key, randomly generaged by Kerberos, shared betwee user and TGS
P(tgs)      : a key based on the password of the tgs service. It's a password known only by the tgs service
P(user)     : a key based on the password of the user
K(user,tgs)<-P(user) : meaning that K(user,tgs) is encrypted with the key of P(user)
TGT         : Ticket Granting Ticket. It's just a token that needed for the user to talk to TGS. It contains
              a session key known only by user and TGS.

Keytab files
In kerberos step 1, users can enter password to obtain the TGT from Kerberos TGS, what about services/devices? Kerberos allow the value P(user) to be exported and saved to a file, usually named keytab. This allows the service to authenticate users without talking to Kerberos server.

A really helpful story telling about Kerberos the authentication mechanism

http://web.mit.edu/kerberos/dialogue.html

I have read quite a few articles about Kerberos. Among them the above dialog is the most helpful. It does not hurt that it came from the original authors who help designed Kerberos.

October 17, 2014

Windows Server LDAP MaxPwdAge, MinPwdAge

When using AD Explorer software to query Windows Server, on a particular domain, you can see the maximum and minimum days required for password to change. These values show up as HEX values such as:

Max: 0xFFFFDEFF0AA68000
Min: 0xFFFFFF36D5964000

This is how to convert them to actual days:

( 0xFFFFFF36D5964000 - <64-bit value> ) / 0xC92A69C000 + 1 = answer ( in days)

Basically, the time unit here is 100ns (See http://en.wikipedia.org/wiki/System_time to see how Windows use 100ns as their system time unit). So
   1 day is 24*3600*10,000,000 = 864000000000 = 0xC92A69C000

Starting from an unsigned 64-bit 0 value,
1 day = 0 - 0xC92A69C000 = 0xFFFFFF36D5964000
2 day2 = 0xFFFFFF36D5964000   - 0xC92A69C000  = 0xFFFFFE6DAB2C8000
3 days = 0xFFFFFDA480C2C000
...
41 days = 0xFFFFDFC835104000
42 days = 0xFFFFDEFF0AA68000
43 days = 0xFFFFDE35E03CC000
44 days = 0xFFFFDD6CB5D30000
...
89 days = 0xFFFFBA10413C4000
90 days = 0xFFFFB94716D28000

So for the above example:

Max: 42 days
Min: 1 day

October 16, 2014

If you have a computer that joins to a domain, you can use LDAP to bind to the domain name controller and find a lot of information. Here is how you do it:

1. open a "cmd" window, and do: echo %logonserver% . This tells you the domain controller's name
2. download sysinternals AD explorer software, and connect to the domain controller
3. On the left tree, suppose your company's domain is abc.com, click on "DC=abc,DC=com"
4. In the subtree, click on "OU=...", and keep click on "OU=..." in the subtrees, until your find all the users with "CN=...". There you can see all the users of the domain, and all other information.

September 11, 2014

SNMP v3 trap and engineID

When sending an SNMP v3 trap from the sender to the trap receiver, the authoritative engineID is the trap sender's engine ID.   Below is a method to remotely discover the engine ID of  the remote SNMP sender (likely SNMP agent):

use the net-snmp snmpwalk command, and add the command option: "-Dsnmp_sess_open" and do a snmpget v3 with your credentials, and you should get something like this:

snmp_sess_open:   probe found engineID:  80001f888016fb784553d902ac
.1.3.6.1.4.1.3432.8.1.1.7.0 = Gauge32: 0

The agent's engineID is right there: 80001f888016fb784553d902ac

You can also try "-Dlcd_get_enginetime", which gives you local engineID

On the other note, I have written the SNMP trap receiver that can detect the incoming engineID automatically. Check it out at github: https://github.com/tiebingzhang/WapSNMP

On a local Linux machine running snmpd, you can find out its engine ID by:

cat /var/net-snmp/snmpd.conf

If you have any SNMP v3 user configured, you should see one or more lines like this:

usmUser 1 3 0x80001f888016fb784553d902ac 0x68636d2e736e6d70763300 0x......

EngineID is the 4th element: 0x80001f888016fb784553d902ac 



September 10, 2014

On a Linux box find out actual serial port being used

1. 
cat /proc/tty/driver/serial

and 

cat /proc/tty/driver/usbserial


2. dmesg | grep tty

3. setserial -g /dev/tty*


September 5, 2014

cross compile wpa_supplicant for arm

cd wpa_supplicant

edit the .config file, copy from a default one if you don't have one.

At the beginning of .config file, add:

CFLAGS += -I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include -O2 -I../libnl/include -D_GNU_SOURCE
LIBS += -L/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/lib/ -L../libnl/lib
CC=arm-none-linux-gnueabi-gcc


Then make

Use "make V=1" to get verbose out.

September 4, 2014

cross compile nginx for ARM

nginx is a nice web server for embedded systems, but it is frustrating to cross compile because it does not use the standard autoconf structure, and the author provides no documentation to help make this process easier.

This is just my way of cross compiling it, feel fee to comment on your way.

1. compile it for your host. In my case, that's x86_64 Linux. I do the following:

  ./configure --without-pcre --without-http_rewrite_module
  make

2. Now, hand tweak the generated Makefie

   vi objs/Makefile

replace the top few lines with the following (change it to your environment):

TOP := $(dir $(lastword $(MAKEFILE_LIST)))
CC =    arm-none-linux-gnueabi-gcc
CFLAGS =  -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Wunused-function -Wunused-variable -Wunused-value
CFLAGS += -I${TOP}/../openssl-1.0.1i/openssl-1.0.1i/include  -I/opt/ezsdk/linux-devkit/arm-none-linux-gnueabi/usr/include
LINK =  $(CC) -L${TOP}/../openssl-1.0.1i/

3. You may need to change the file objs/ngx_auto_config.h because some functions available on host are not available on your target. In my case, I had to remove lines containing ACCEPT4

4. clean the old obj files:
   find . -name "*.o" | xargs rm -f

5. make. The final binary is located objs/nginx

Note: make clean will remove the objs/Makefile you just changed. Make sure you have a backup.

cross compile openssl 1.0.1i for ARM

export CROSS_COMPILE=arm-none-linux-gnueabi-
export ARCH=arm
./Configure shared linux-armv4
CC=${CROSS_COMPILE}gcc RANLIB=${CROSS_COMPILE}ranlib make

August 29, 2014

GO SNMP implementation

This is a open source SNMP trap receiver written in GO:

https://github.com/tiebingzhang/WapSNMP

It supports SNMP v2c and V3 traps.

It also includes utilities to perform SNMP Get and SNMP walk for both SNMP v2c and V3.

Since it is written in GO, it compiles on both Linux and Windows (and in theory also Mac OS, but not tested).

my .gitconfig

[user]
        name = Tiebing Zhang
        email = tzhang@advistatech.com
[core]
        editor = vim
[merge]
        tool = vimdiff
[alias]
        d = difftool
        st = status -s
        ci = commit -a
        br = branch
        co = checkout
        df = diff
        dc = diff --cached
        last = log -1 HEAD --stat
        lg = log --stat
        who = shortlog -s --
[diff]
        renames = copy
        tool = gvimdiff
[difftool "gvimdiff"]
        cmd = "gvim -d" "$LOCAL" "$REMOTE"
[difftool]
        prompt = false
[push]
        default = matching
[sendpack]
        sideband = false

GO cross compile Windows binary on Linux


# set up cross compilation to windows_amd64
# you only need to do this once
cd $GOROOT/src
GOOS=windows GOARCH=amd64 ./make.bash --no-clean

# whenever you want to cross compile
cd $YOUR_APP
GOOS=windows GOARCH=amd64 go build

More at
https://inconshreveable.com/04-30-2014/cross-compiling-golang-programs-with-native-libraries/

August 12, 2014

netcat send file with progress indicator

The following C code compiles to a file nc-send-file, and it will send a file to a raw tcp socket (such as netcat with the command: nc -l -p 8000 > myfile ), with progress indicator for every 64K bytes sent. On Windows, compile it using MinGW the following Makefile:
nc-send-file:nc-send-file.c
        gcc -Wall -O2 -o $@ $^ -s -lws2_32
clean:
        rm -f nc-send-file.exe
C code:
#ifdef WIN32
#include "winsock2.h"
#else
#include "sys/socket.h"
#include "netinet/in.h"
#include "unistd.h"
#endif
#include "stdio.h"

int main(int argc, char**argv)
{
   FILE * fd;
   int sockfd,ret,n;
   struct sockaddr_in servaddr;
   char sendline[1024*64];
   unsigned int totalbytes=0;

   if (argc != 4)
   {
      printf("usage:  client [IP address] [PORT] [File]\n");
      exit(1);
   }
#ifdef WIN32
   WSADATA wsaData;
   if (WSAStartup(MAKEWORD(2, 0), &wsaData) != 0){
    fprintf(stderr, "WSAStartup() failed");
    exit(1);
   }
#endif

   sockfd=socket(AF_INET,SOCK_STREAM,0);

   memset(&servaddr,0,sizeof(servaddr));
   servaddr.sin_family = AF_INET;
   servaddr.sin_addr.s_addr=inet_addr(argv[1]);
   servaddr.sin_port=htons(atoi(argv[2]));

   if (connect(sockfd, (struct sockaddr *)&servaddr, sizeof(servaddr))<0){
    fprintf(stderr,"Error connecting to target.\n");
    return -1;
   }

   fd=fopen(argv[3],"rb");
   if (fd==NULL){
    fprintf(stderr,"Opening file %s error\n",argv[3]);
    return -1;
   }

   while (1){
   ret=fread(sendline, 1, sizeof(sendline),fd);
   if (ret==0){
    if (feof(fd)){
     printf("Finished. Total bytes sent:%u\n",totalbytes);
    }else{
     printf("Error.\n");
    }
    break;
   }
      n=send(sockfd,sendline,ret,0);
   if (n!=ret){
    printf("did not send all in the buffer. expecting %d sent %d exit.\n", ret, n);
    return -1;
   }
   totalbytes+=ret;
   printf("sending %d bytes, total %d bytes\n",ret, totalbytes);
   }
   fclose(fd);
#ifdef WIN32
   closesocket(sockfd);
   WSACleanup();  /* Cleanup Winsock */
#else
   close(sockfd);
#endif
   return 0;
}

August 8, 2014

xrdp reconnect to the same session

cat xrdp/xrdp.ini

[globals]
bitmap_cache=yes
bitmap_compression=yes
port=3389
crypt_level=high
channel_code=1

[xrdp1]
name=sesman-Xvnc
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
port=-1

[xrdp2]
name=Reconnect
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
#port=-1
port=5910


Use stunnel to access https

stunnel -c -d localhost:8000 -r foo:443 -P /tmp/stunnel.pid -f

Now you can access local 8000 using plaintext so that you can debug and sniff packets.

July 6, 2014

ipsec local routing bypass

Suppose that you have a ipsec site-to-site network as follows:

On local: from 10.1.0.0/16 to 0.0.0.0/0  goes to remote ipsec
On remote: from 0.0.0.0/0 to 10.1.0.0/16 goes to ipsec

Now on local you have a subnet 10.1.1.0/24 that you want to talk to. For example, you local interface eth1 has 10.1.1.1, and there is another computer on your network 10.1.1.2. Now if you want to ping 10.1.1.2 it would not work, because the traffic is captured by the ipsec policy (use "ip xfrm policy" to show it) and directed to ipsec tunnel.

To add local route bypass, use this:
ip xfrm policy add dir in src 10.1.1.0/24 dst 10.1.1.0/24
ip xfrm policy add dir out src 10.1.1.0/24 dst 10.1.1.0/24

run this after your ipsec tunnel is up. Then you should be able to ping 10.1.1.2.

Note when I add polices directly use "setkey" it did not work for me.

Looks like StrongSwan can do this using config file section called "shunt" policies. Needs to try this.

July 3, 2014

sfdisk tips

sfdisk can be used to list partition information and alter partition tables

cat /proc/partitions to see the list of kernel recognized drivers and partitions

sfdisk -l : list information

sfdisk -l -u M : list information using MiB as unit. 

sfdisk -l -u S : List information using sector (512 bytes) as unit.

Example:
 sudo sfdisk -l /dev/sdc -u M

Disk /dev/sdc: 1044 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = mebibytes of 1048576 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start   End    MiB    #blocks   Id  System
/dev/sdc1   *     1   7806   7806    7993344   83  Linux
/dev/sdc2      7807+  8190    384-    392193    5  Extended
/dev/sdc3         0      -      0          0    0  Empty
/dev/sdc4         0      -      0          0    0  Empty
/dev/sdc5      7808   8190    383     392192   82  Linux swap / Solaris

Notice "7807+" and "384-" means that the numbers are not a multiple of MiB. Therefore that partition is not MiB aligned. For Extended partition, that is OK, but for real data-holding partitions, you may want them to be MiB aligned.

This is the list by sector results:
sudo sfdisk -l /dev/sdc -u S

Disk /dev/sdc: 1044 cylinders, 255 heads, 63 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdc1   *      2048  15988735   15986688  83  Linux
/dev/sdc2      15990782  16775167     784386   5  Extended
/dev/sdc3             0         -          0   0  Empty
/dev/sdc4             0         -          0   0  Empty
/dev/sdc5      15990784  16775167     784384  82  Linux swap / Solaris

Notice now there is no +/- anymore, and start/end are in sectors. This will give you exact layout of the partitions.

Write partitions:
sfdisk takes two kinds of formats as input to write partitions: the "manual" format as documented in the man page, and the "dump" format.

Use "-d" to dump the current partition table.

 sudo sfdisk -d /dev/sdc
# partition table of /dev/sdc
unit: sectors

/dev/sdc1 : start=     2048, size= 15986688, Id=83, bootable
/dev/sdc2 : start= 15990782, size=   784386, Id= 5
/dev/sdc3 : start=        0, size=        0, Id= 0
/dev/sdc4 : start=        0, size=        0, Id= 0
/dev/sdc5 : start= 15990784, size=   784384, Id=82

Now you can edit this file, and then feed it back to change partitions using command "sfdisk -d /dev/sdc < part.txt", supposing that you wrote the above text to "part.txt" file.

Now some explanation of the "dump" format:

1. Empty lines are ignored
2. # and things after it is ignored (for that line)
3. "unit: sectors" is recognized as a valid command to set unit to sectors
4. Each partition line starts with the partition name, which is ignored. Then ":", then  multiplle "variable-name=value" separated by ",". "bootable" is special because it is a boolean value.
5. All values have to be present. No auto calculation as in "manual" mode.

Partition Alignment
When I tried to start partitioning on the 2nd cylinder instead of the first,  I ran into issues. The command was:

sfdisk /dev/sdb -H 16 -S 63 <<EOF
1,159,83
,4096,82
,,83
;

EOF

My 80GB disk does not have a multiple of 2048 sectors, because the total number of sectors is a multiple of 63, which is the typical sectors per track. So to make my partition aligned at least to 4K, I used head of 16, which gives me the cylinder size of 63*16*512bytes=63*8KiB, so my cylinder is 4KiB aligned.

The above command will produce the following error:
sfdisk: Warning: given size (4096) exceeds max allowable size (1)
sfdisk: bad input

I was very puzzled. After compiling sfdisk from source code and adding debug statements in it, I finally found out the reason: sfdisk is trying to fit the 2nd partition in the first free cylinder because it think it is free! Well it is free except for the MBR in the first sector, but I don't want my 2nd partition to there. How do we tell sfdisk that? Here comes the "undocumented dangerous option" "--in-order". If you specify the "--in-order", sfdisk will work as you intended it to.

OR, you could manually specify the start address of each partition.

So the final working command is:
sfdisk /dev/sdb --in-order -H 16 -S 63 <<EOF
1,159,83
,4096,82
,,83
;
EOF


At last, you can use "hdparm -i" to list harddrive info, including number of LBA sectors. However, this may not work on some disks. 

July 2, 2014

Understand fdisk output


Disk /dev/hdb: 64 heads, 63 sectors, 621 cylinders
Units = cylinders of 4032 * 512 bytes
 
   Device Boot    Start       End    Blocks   Id  System
/dev/hdb1   *         1       184    370912+  83  Linux
/dev/hdb2           185       368    370944   83  Linux
/dev/hdb3           369       552    370944   83  Linux
/dev/hdb4           553       621    139104   82  Linux swap

Sector size: 512 bytes

64 Heads: there are 64/2=32 platters in the disk, each platter has 2 heads. Think head as a surface.

63 sectors: Each head is divided in to 63 sectors

621 cylinders: There are 621 of concentric tracks/cylinders on each head. Each track has 63 sectors as indicated above. 

A cylinder is a vertical slice of a track on all heads. Cylinder size = #Heads * #sectors * 512 = 64*63*512 = 4032*512 = 2064384 bytes

Disk are allocated from low cylinders to high cylinders. 

Start/End unit is cylinder. 

Block size is 1024 bytes. 

Blocks per cylinder is 2064384/1024 = 2016

The "+" sign in "370912+" means the number 3780912 is not accurate. The actual blocks should be a little more. To calculate the actual one, just do 370912/2016=183.98, and then rounded it up to get 184. So the first partition has 184 cylinders.

Another example:
Disk /dev/sda: 250.0 GB, 250059350016 bytes
64 heads, 32 sectors/track, 238475 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          81   82928   83  Linux
/dev/sda2              82        2082   2049024    83  Linux
/dev/sda3            2083      238475   242066432   83  Linux

Cylinder size = 64*32*512=1048576 (1MB)
Blocks per cylinder = 1MB/1024=1024


July 1, 2014

Linux USB sysfs device naming scheme

In Linux dmesg, you may see something like this:

usb 1-1.1: new high speed USB device using musb-hdrc and address 8
usb 1-1.1: New USB device found, idVendor=05e3, idProduct=0608
usb 1-1.1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
usb 1-1.1: Product: USB2.0 Hub
hub 1-1.1:1.0: USB hub found

The naming scheme is like this:

root_hub-hub_port.hub_port.hub_port(...):config.interface

So 1-1.1:1.0 means:

Device connected to Root Hub 1, Port 1 which has another hub connected to it, Port1. Then Config 1, and interface 0.

June 30, 2014

php md5sum

#!/usr/bin/php -f
<?php
if (count($argv)<2) die("no file specified\n");

if (!file_exists($argv[1])) die ("file does not exist\n");
echo md5_file($argv[1])."\n";

June 25, 2014

git windows push hang 100%

Solution:

1. Download the latest git for windows http://git-scm.com/download/win
2. git config --global sendpack.sideband false

You can also hack the git binary (either on Server or on Client) to change the string "side-band-64k" to something different such as "side-bond-64k". This essentially disables git side-band.

http://billauer.co.il/blog/2012/10/git-pull-windows-freeze-receive-pack/

install gitweb on ubuntu/debian


you have to install the package gitweb sudo apt-get install gitweb

Then you have to edit the apache gitweb config file
$EDITOR /etc/apache2/conf.d/gitweb
change the line Alias /gitweb /usr/share/gitweb to Alias /git /usr/share/gitweb

open the /etc/gitweb.conf file:
you have to change the line $projectroot ".." to $projectroot "/code/git"
and change any other line containing /gitweb to /git for example
$stylesheet = "/gitweb/gitweb.css";
to $stylesheet = "/git/gitweb.css";


then reload you apache webserver with sudo /etc/init.d/apache2 force-reload


[Source:http://askubuntu.com/questions/10452/setting-up-gitweb-apache2]

June 13, 2014

Kernel bypassing networking

Here is a selection of the many kernel-bypass solutions that are available:
These products each take their own design approaches and it’s interesting to consider choices that they make.

  • Customized kernel device driver. netmap and DNA both fork standard Intel drivers with extensions to map I/O memory into userspace.
  • Custom hardware. Myricom and Napatech both distribute bespoke device drivers for their own custom hardware (ASIC for Myricom and FPGA for Napatech).
  • Userspace library. These solutions each provide unique libraries to access their extensions. The scope varies tremendously: Ethernet I/O, libpcap compatibility, hardware-assisted traffic dispatching for multiprocessing, buffer memory management, all the way up to entire TCP/IP socket layers.
  • Licensing. netmap is open-source, DNA requires a modest license for its userspace library, Napatech requires an NDA and depends on very expensive hardware.

June 11, 2014

golang http client Server Sent Event receiver

package main

import (
 "crypto/tls"
 "log"
 "bufio"
 "net/http"
 "time"
)

func main() {
 var client *http.Client
 tr := &http.Transport{
  TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
 }

 client = &http.Client{tr, nil, nil, 0 * time.Second}

 /* open a request, can't use httpclient.Get because we need the http.Request so we can close the connection later */
 req, err := http.NewRequest("GET", "https://dev1.advistatech.com/sse2.php", nil)
 if err != nil {
  log.Fatal(err);
 }

 resp, err := client.Do(req)
 if err != nil {
  log.Fatal(err);
 }
 log.Printf("connected\n");

 reader:=bufio.NewReader(resp.Body);
 for {
  line,err:=reader.ReadString('\n');
  if err != nil {
   log.Fatal(err);
  }else{
   log.Printf("resp:%s\n",line);
  }
 }

 //respbytes, _ := ioutil.ReadAll(resp.Body)
 //log.Printf("resp: %s\n", respbytes)

 resp.Body.Close()
}

June 10, 2014

California FTB live person

Service section at 800.852.5711.
The hours of operation are 7:00 a.m. - 5:00 p.m., Monday through Friday, except state holidays.
Choose the Business prompt 2 to then 4 to speak to a live representative.  

Openssl AES-NI Test

For OpenSSL versions after 1.0.1, AES-NI should be buit-in. This is how to test it:

Command A (with AES-NI) = openssl speed -elapsed -evp aes-128-cbc
Command B (Without AES-NI) = OPENSSL_ia32cap="~0x200000200000000" openssl speed -elapsed -evp aes-128-cbc

bit #33 denoting availability of PCLMULQDQ instruction (for AES-GCM computation);
bit #57 denoting AES-NI instruction set extension;

Results:
Command   16 bytes     64 bytes     256 bytes    1024 bytes   8192 bytes
------------------------------------------------------------------------
A         796435.32k   845155.61k   852750.59k   860752.55k   865828.86k
B         393740.06k   431465.71k   438168.23k   443452.42k   446458.54k

http://www.openssl.org/docs/crypto/OPENSSL_ia32cap.html

June 4, 2014

How to add a schema to OpenLDAP server

vim /tmp/borrame.conf
(this is what goes in the file)

include /etc/ldap/schema/core.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/nis.schema
include /etc/ldap/schema/inetorgperson.schema
include /etc/ldap/schema/misc.schema
include /etc/ldap/schema/mypersonalschema.schema

mkdir /tmp/borrame.d
slaptest -f /tmp/borrame.conf -F /tmp/borrame.d

Edit the generated file
vim /tmp/borrame.d/cn\=config/cn\=schema/cn\=\{5\}mypersonalschema.ldif

I changed the three head lines to this:
dn: cn=mypersonalschema,cn=schema,cn=config
objectClass: olcSchemaConfig
cn: mypersonalschema

Then I deleted these lines from the bottom of the file:

structuralObjectClass:
entryUUID:
creatorsName:
createTimestamp:
entryCSN:
modifiersName:
modifyTimestamp:

And at last I inserted the new schema to the ldap tree:
ldapadd -Y EXTERNAL -H ldapi:/// -f /tmp/borrame.d/cn\=config/cn\=schema/cn\=\{5\}mypersonalschema.ldif

Add LDAP user authentication to YellowDog Linux

/etc/ldap.conf
*************************************
host ldap-server-ip-address
base ou=Users,dc=advistatech,dc=com
ssl no
pam_password md5
*************************************

/etc/openldap/ldap.conf
*************************************
HOST ldap-server-ip-address
BASE ou=Users,dc=advistatech,dc=com
*************************************

/etc/pam.d/system-auth
*************************************
auth required /lib/security/$ISA/pam_env.so
auth sufficient /lib/security/$ISA/pam_unix.so likeauth nullok
auth sufficient /lib/security/$ISA/pam_ldap.so use_first_pass
auth required /lib/security/$ISA/pam_deny.so

account sufficient /lib/security/$ISA/pam_succeed_if.so uid < 100
account required /lib/security/$ISA/pam_unix.so
account [default=bad success=ok user_unknown=ignore] /lib/security/$ISA/pam_ldap.so

password requisite /lib/security/$ISA/pam_cracklib.so retry=3
password sufficient /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow
password sufficient /lib/security/$ISA/pam_ldap.so use_authtok
password required /lib/security/$ISA/pam_deny.so

session required /lib/security/$ISA/pam_limits.so
session required /lib/security/$ISA/pam_unix.so
session optional /lib/security/$ISA/pam_ldap.so
*************************************

nsswitch.conf
**********************
passwd: files ldap
shadow: files ldap
group: files ldap
**********************

You can use "getent passwd" to  list all the users in the ldap server.

To make sshd work, restart the sshd service.

May 31, 2014

Newbie’s OpenLDAP tips

  1. Newbie’s OpenLDAP tips

    1. Following https://help.ubuntu.com/12.10/serverguide/openldap-server.html to get OpenLDAP server and tools installed and configured on your server.
    2. The binary is called slapd. The configuration file is not a file anymore, but a directory structure in LDAP file format. The files used in LDAP is using the LDIF format, which is plaintext. The configuration is stored at /etc/ldap/sldap.d/, in file “cn=config.ldif” and inside directory “cn=config”.
    3. The ldap server access credential is stored in  file “cn=config/olcDatabase={1}hdb.ldif”
      1. olcRootDN:
      2. olcRootPW:
    4. User LDAP data is stored in /var/lib/ldap using a binary DB format (Berkely DB or some other DB format). User LDAP data can only be viewed by using LDAP tools such as ldapsearch, and can be edited by ldapadd, ldapdelete, ldapmodify,etc
    5. Command to list all entries
      1. ldapsearch -x -LLL -b dc=advistatech,dc=com
    6. To authenticate to ldapserver when running command like ldapsearch, you can use “-x”, which is amount to local authentication. Or you can “bind” to the server using the credential listed above in “olcRootDN” and “olcRootPW” as follows. Only binding will print out user password information stored in LDAP DB.  See more at http://blogs.splunk.com/2009/07/30/ldapsearch-is-your-friend/
      1. -D "cn=admin,dc=advistatech,dc=com"
        -W will prompt you for your password
    7. PHP can act as a LDAP client. It has dedicated function to connect, bind, and query LDAP Servers.
    8. In PosixAccount object (like Unix user account), the password field name is “userPassword”. It is usually hashed with LDAP special seeded SHA1 hash function. If you query it when bound to the server, you will see something like {SSHA}….
    9. ldapcompare does not automatically hash clear password when comparing. So you would need to hash the password first (probably using ldappasswd) and then do the compare (to be validated)
    10. The usually way to testing an user’s account credential is actually try to bind (login) to the ldap server using that credential.
    11.  A simple PHP script to test a user credential:
    <?php
    $ds=ldap_connect("localhost");
    if (!$ds) {
        die ("Unable to connect to LDAP server.");
    }

    ldap_set_option($ds, LDAP_OPT_PROTOCOL_VERSION, 3);
    ldap_set_option($ds, LDAP_OPT_REFERRALS, 0);

    $dn = "uid=tony,ou=Users,dc=advistatech,dc=com";
    $pass= "mypass";

    // bind
    if (ldap_bind($ds,$dn,$pass))
    {
        echo ("bound successfully");
    } else {
        echo "Unable to bind to LDAP server.";
    }

    ldap_close($ds);

May 30, 2014

ldapscript not working fixed

Make sure your /etc/ldapscripts/ldapscripts.passwd does not contain a trailing newline. (This is the default behaviour of many editors, including vim)

samba server symbolic links

To enable symbolic links, edit smb.conf with the following changes:

1. Add the following lines in the [global] section
[global]
    follow symlinks = yes
    wide links = yes
    unix extensions = no

2. Add the following lines to your shared directory section
[home]
    follow symlinks = yes
    wide links = yes

3. Restart smbd. On Fedora/Redhat, use  
         sudo systemctl restart smb.service
   on Debian/Ubuntu, use
        sudo /etc/init.d/smbd restart

May 2, 2014

The Secret to 10 Million Concurrent Connections -The Kernel is the Problem, Not the Solution

http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html

Netmap

Intel DPDK

PF_RING

April 22, 2014

ultravnc has better performance than tightvnc on Windows 7

Use ultravnc (Windows 7 X64) and you should see better performance.

Prolific USB to COM driver can not start error code 10 on Windows 7 64-bit

There are a couple of prolific chipsets i've seen, the older chips sets don't seem to work with the new driver on the prolific website.

Which version: Rev_0300 or Rev_0400
1. Plug in the adapter and open the device manager.
  • If the driver is loaded, you'll see the device under ports, probably with a Yellow !
  • If the driver isn't loaded it will likely appear under Other (or unknown?) Devices.
2. Double click the adapter to open its properties.  
3. Go to the details tab in the top right.
4. Click the drop down box and choose Harware IDs.
5. Note if it says Rev_0300 or Rev_0400

Which Driver Version
  • Rev_0400 works with the lasest drivers from the Prolific Website (Xp, Win7 32 or 64.) 
  • Rev_0300 needs older drivers XP: driver version 2.0.2.24.  Windows 7 (32 and 64) version 3.0.1.0.  Download it HERE or HERE
Loading the Drivers:

Win7

1. Run the application VISTA.EXE to install the drivers.  Open the device manager and the adapter should be present under Port and may be showing a yellow ! exclamation mark showing in the device manager and reporting Code 10.  (If it looks normal, it's already good to go)

2. Right click the device (in the device manager) and choose update driver. Don't let windows update automatically - you need to pick from a list. 

After clicking update drvier choose:

  • Browse My Computer for Drvier Sowftware
  • Let me pick from a list of device drivers on my computer
  • Prolific USB-to-Serial Comm Port version [version] > Next

April 8, 2014

build a golang cross-compile toolchain for ARM on Linux

1. Download the latest golang source: wget https://go.googlecode.com/files/go1.2.1.src.tar.gz
2. export GOOS=linux
3. export GOARCH=arm
4. export GOARM=7 (find out your ARM version based on the ARM processor). This step can be skipped as long as you know your ARM version is not 5.
5. export GOROOT=/home/me/go (set this to where your untarred go directory)
6. cd go/src; ./make.bash

There you have it.