August 23, 2019

Instant file sharing without logging in

1. https://sharedrop.io Web-RTC based; Local LAN transfer doesn't leave LAN

2. https://file.pizza HTTP streaming and Web-RTC based.

3. https://ppng.ml HTTP streaming, supporting curl based command line.

Source code at: https://github.com/nwtgck/piping-server, and a minimal golang version at https://github.com/nwtgck/go-piping-server

Blog at https://dev.to/nwtgck

4. https://xkcd949.com HTTP Streaming


WebRTC Demo using peer.js:
https://jmcker.github.io/Peer-to-Peer-Cue-System/

July 30, 2019

Windows 10 spotlight image files location

%userprofile%\AppData\Local\Packages\Microsoft.Windows.ContentDeliveryManager_cw5n1h2txyewy\LocalState\Assets

https://www.howtogeek.com/247643/how-to-save-windows-10s-lock-screen-spotlight-images-to-your-hard-drive/

July 18, 2019

Search Linux Kernel to find out when a feature was added

Use the LKDDB (Linux Kernel Driver DataBase):

https://cateee.net/lkddb/web-lkddb/IP_SET_HASH_IPMAC.html

Search a CONFIG_xxx and it will tell you since what version of Linux kernel it was added.


May 15, 2019

TCP socket send buffer deep dive




A typical TCP socket send buffer is composed of three parts: unacked-bytes, unsent-bytes, and free-buffer.
                               +----------------+
                               |                |
                               |                |
                               |  FREE BUFFER   |
                               |                |
                               |                |
                               +----------------+
                               |                |
                               |  UNSENT BYTES  |
                               |                |
                               +----------------+
                               |                |
                               |  UNACKED BYTES |
                               |                |
                               +----------------+

Total send buffer size

Total send buffer size = unacked-bytes + unsent-bytes + free-buffer.  
It can be obtained using the SO_SNDBUF socket option. The buffer size could dynamically change its size as seen needed by the OS. This works for both Linux and macOS.
        slen = sizeof(sndbufsiz);
        err = getsockopt(sd, SOL_SOCKET, SO_SNDBUF, &sndbufsiz, &slen);

Total in-flight bytes

Total inflight bytes = unacked-bytes + unsent-bytes. 
It can be obtained using SO_NWRITE socket option on macOS, and SIOCOUTQ ioctl on Linux.
int get_socket_used(int sd){
    int err;
    int used;
#ifdef __APPLE__
    socklen_t slen = sizeof(used);
    err = getsockopt(sd, SOL_SOCKET, SO_NWRITE, &used,&slen);
    if(err < 0) {
        perror("getsockopt 2");
        exit(1);
    }
#else
    err = ioctl(sd, SIOCOUTQ, &used);
    if(err < 0) {
        perror("ioctl SIOCOUTQ");
        exit(1);
    }
#endif
    return used;
}
On macOS, it seems that this can also be obtained using the TCP_INFO struct, but it is a private API.
u_int32_t       tcpi_snd_sbbytes;       /* bytes in snd buffer including data inflight */

unacked-bytes

On Linux, unacked-bytes can be obtained from the TCP_INFO structure, but the result is number of segments, or bytes. On macOS, TCP_INFO seems to contain this infomration (private API).
int get_socket_unacked(int sd){
    struct tcp_info tcp_info;
    socklen_t tcp_info_length = sizeof(tcp_info);
    if ( getsockopt(sd, IPPROTO_TCP, TCP_INFO, (void *)&tcp_info, &tcp_info_length ) == 0 ) {
        return tcp_info.tcpi_unacked;
    }
    return 0;
}

//For macOS, use TCP_INFO
    u_int64_t       tcpi_txunacked __attribute__((aligned(8)));    /* current number of bytes not acknowledged */
macOS tcp_info definition

unsent-bytes (not including un-acked bytes)

On Linux, unsent-bytes can be obtained from the tcpi_notsent_bytes field of the TCP_INFO structure. NOTE that this requires kernel version to be 4.6 or newer. For Android that means Android 8 or newer. On Linux, it can also be obtained using SIOCOUTQNSD ioctl. It’s not clear how to do this on macOS.
//defined in /usr/include/linux/sockios.h 
int get_socket_unsent(int sd){
    int err;
    int unsent;
    err = ioctl(sd, SIOCOUTQNSD, &unsent);
    if(err < 0) {
        perror("ioctl SIOCOUTQNSD");
        exit(1);
    }
    return unsent;
}

//OR 

int get_socket_unsent(int sd){
    struct tcp_info tcp_info;
    socklen_t tcp_info_length = sizeof(tcp_info);
    if ( getsockopt(sd, IPPROTO_TCP, TCP_INFO, (void *)&tcp_info, &tcp_info_length ) == 0 ) {
        return tcp_info.tcpi_notsent_bytes;
    }
    return 0;
}

Stackoverflow discussion on getting unsent bytes

epoll and kevent

epoll on Linux, and kevent on macOS, get triggered by the unsent-bytes, when the TCP option TCP_NOTSENT_LOWAT is set. On macOS, kevent() doesn’t report socket as writable until the unsent TCP data drops below specified threshold (typically 8 kilobytes).

May 9, 2019

Enable user-id based packet routing on Mac OS


If you would like to route all socket (TCP/UDP) traffic from processes running by a particular user on a Mac OS to be routed differently, you can do that.

1. Add the user to your Mac OS if not already done. In this example, I will add an user named "test1"
2. run the command:
        sudo vi /private/etc/pf.conf
    and add the following line before ' anchor "com.apple/*"
         pass out quick on en0 route-to { utun4 192.168.15.2 } user test1

   Note:
   a) change en0 to your default network interface name on Mac
   b) change utun4 to the network interface you would these packets to be routed to

3. restart pf by doing:
    sudo pfctl -d; sudo pfctl -e -f /etc/pf.conf

Now all processes running by user test1 should be routed to the new interface as specified.

February 26, 2019

keep ssh running in background for tunneling

1. write the following script to a file, named "tunnel.sh" and make it executable (make sure user has public auth enabled on remote host):

while true; do ssh -t -n -R 127.0.0.1:2233:127.0.0.1:22 user@remote.host.com "while true; do ps -ef; sleep 1; done" ; sleep 1; done

2.  Run the the above script in a detached screen session:
    screen -S tunnel -d -m /path/to/tunnel.sh


 That's all. This creates a background screen session, which runs the tunnel.sh script, which loops an ssh command to keep it up and running.

September 28, 2018

July 28, 2018

How to use Netlink to get ipv6 neighbors

1. open netlink socket
2. send request to get neighbors
3. parse the response


The response is in the following format
[ netlink msg ] ... [ netlink msg ]


use the Macros defined in "man 3 netlink" to iterate through the messages. Each netlink message has a header "struct nlmsghdr *", defined in "man 7 netlink".

The netlink message for getting ipv6 neighbors is defined in "man 7 rtnetlink"

for ipv6 neighbors message, each netlink msg block has the following format:
  netlink-msg-header | neighbor-discover-msg-header (struct ndmsg) | route-attributes (struct rtattr) | more route-attributes...


Each route-attributes has the following format:
  header (struct rtattr) | data


The types of rtattr are defined in /usr/include/linux/neighbour.h, with the commons are:
  NDA_DST: data is IP address
  NDA_LLADDR: data is MAC address
  NDA_CACHEINFO: data is struct nda_cacheinfo



Example NetLink payload parsing (not showing nlmsghdr):

0a 00 00 00  :  inet family
02 00 00 00  : ifindex
04 00  : state
00 : flags
01 : type


14 00 : rtattr len
01 00 : type
fe 80 00 00 00 00 00 00 b6 ef fa ff fe d0 fe 76 : Ipv6 address


0a 00 : rtattr len
02 00 : type
b4 ef fa d0 fe 76 : mac


00 00  //padding

08 00  //probe ?
04 00
00 00 00 00


14 00 //rtattr len
03 00 //cache info
6b a8 e8 01 fb 90 e8 01
6b b9 69 00 00 00 00 00

March 31, 2018

CAVEAT: golang uint to string conversion

In golang, you can cast a "uint8" value  to "string".

Be aware that
  if the uint8 x <= 127, string(x) is the ASCII char of the value x,
  if the uint8 x >128, string(x) is 2 bytes long, because of UTF-8

November 28, 2017

Thoughts on cloud server testing

Cloud service can be treated like a control system, which takes input and produces output.

Control system has two characteristics: Controllability and Observability

Control system is a tested by applying an impulse to it and observe the response of the system.

From this we get the inspiration of cloud server testing:

1. Increase observability: create observation points and record their values. Performance counters, Instrumentation, etc are the tools for this. This can be used for monitoring, but also to observe the internal state of the service for debugging and trouble-shooting.

2.  Apply different level of impulse input to observe the output of the system, and the observable states of the system.

3. Find the break point of the system and the corresponding impulse input.

September 27, 2017

run X program headless in Linux

On remote server:
1. run "xvfb-run xterm" (replace xterm with your program). The runs a virtual frame-buffer/X server. The default display is :99. You can change that.
2. find out where the X auth file is written. Default iat /tmp/xvfb-run.XXXX/Xauthority.
3. x11vnc -display :99 -nopw -auth /tmp/xvfb-run.o3K0jQ/Xauthority

 on your local desktop, run vnc viewer to connect to the remote server. You can set up password in the server if you want to.

September 25, 2017

Linux L2tp client setup for Mac OS X vpn server

Instructions below are based on the work at https://github.com/hwdsl2/setup-ipsec-vpn/blob/master/docs/clients.md#linux
Commands must be run as root on your VPN client.
To set up the VPN client, first install the following packages:
# Ubuntu & Debian
apt-get update
apt-get -y install strongswan xl2tpd

# CentOS & RHEL
yum -y install epel-release
yum -y install strongswan xl2tpd

# Fedora
yum -y install strongswan xl2tpd
Create VPN variables (replace with actual values):
VPN_SERVER_IP='your_vpn_server_ip'
VPN_IPSEC_PSK='your_ipsec_pre_shared_key'
VPN_USER='your_vpn_username'
VPN_PASSWORD='your_vpn_password'
Configure strongSwan:
cat > /etc/ipsec.conf <<EOF
# ipsec.conf - strongSwan IPsec configuration file

# basic configuration

config setup
  # strictcrlpolicy=yes
  # uniqueids = no

# Add connections here.

# Sample VPN connections

conn %default
  ikelifetime=60m
  keylife=20m
  rekeymargin=3m
  keyingtries=1
  keyexchange=ikev1
  authby=secret
  ike=aes128-sha1-modp1024,3des-sha1-modp1024!
  esp=aes128-sha1-modp1024,3des-sha1-modp1024!

conn myvpn
  keyexchange=ikev1
  left=%defaultroute
  auto=add
  authby=secret
  type=transport
  leftprotoport=17/1701
  rightprotoport=17/1701
  right=$VPN_SERVER_IP
  rightid=%any
EOF

cat > /etc/ipsec.secrets <<EOF
: PSK "$VPN_IPSEC_PSK"
EOF

chmod 600 /etc/ipsec.secrets

# For CentOS/RHEL & Fedora ONLY
mv /etc/strongswan/ipsec.conf /etc/strongswan/ipsec.conf.old 2>/dev/null
mv /etc/strongswan/ipsec.secrets /etc/strongswan/ipsec.secrets.old 2>/dev/null
ln -s /etc/ipsec.conf /etc/strongswan/ipsec.conf
ln -s /etc/ipsec.secrets /etc/strongswan/ipsec.secrets
Configure xl2tpd:
cat > /etc/xl2tpd/xl2tpd.conf <<EOF
[lac myvpn]
lns = $VPN_SERVER_IP
ppp debug = yes
pppoptfile = /etc/ppp/options.l2tpd.client
length bit = yes
EOF

cat > /etc/ppp/options.l2tpd.client <<EOF
ipcp-accept-local
ipcp-accept-remote
refuse-eap
require-chap
noccp
noauth
mtu 1280
mru 1280
noipdefault
defaultroute
usepeerdns
connect-delay 5000
name $VPN_USER
password $VPN_PASSWORD
EOF

chmod 600 /etc/ppp/options.l2tpd.client
The VPN client setup is now complete. Follow the steps below to connect.

Note: You must repeat all steps below every time you try to connect to the VPN.
Create xl2tpd control file:
mkdir -p /var/run/xl2tpd
touch /var/run/xl2tpd/l2tp-control
Restart services:
service strongswan restart
service xl2tpd restart
Start the IPsec connection:
# Ubuntu & Debian
ipsec up myvpn

# CentOS/RHEL & Fedora
strongswan up myvpn
Start the L2TP connection:
echo "c myvpn" > /var/run/xl2tpd/l2tp-control
Run ifconfig and check the output. You should now see a new interface ppp0.
Check your existing default route:
ip route

September 12, 2017

Compile GoLang for AR9341

Golang support for MIPS 32 has been added since version 1.8. However, Soft FPU is not added, making chipsets like AR9341 not able to run Go program. "vstafanovic" has submitted the patch but
it has not been accepted yet in 1.9.0. Hopefully it will make to 1.10

At the same time, you can apply the patch yourself to version 1.8.3:

1. Download the patch
2. Download Golang 1.8.3 source code and apply the patch
3. cd src; ./bash.all

If everything goes well, you will have a compiled go toolchain.

To compile your application to MIPS, do:
GOOS=linux GOARCH=misp GOMIPS=softfloat go build

July 31, 2017

automatically set gnu screen window title

http://scie.nti.st/2008/8/19/1-minute-post-hostname-as-screen-window-title/

In short, add this line to the remote host's .bashrc:

[ "$TERM" = "screen" ] && PROMPT_COMMAND='echo -ne "\033k$HOSTNAME\033\\"'

July 20, 2017

Setting ssh server to an user to only SFTP to the user's home directory

Here is a guide for setting up SFTP users who’s access is restricted to their home directory.

Add the following to the end of the /etc/ssh/sshd_config file:
Subsystem sftp internal-sftp

# This section must be placed at the very end of sshd_config
Match Group sftponly
    ChrootDirectory %h
    ForceCommand internal-sftp
    AllowTcpForwarding no

This means that all users in the ‘sftponly’ group will be chroot’d to their home directory, where they only will be able to run internal SFTP processes.

Now you can create the group sftponly by running the following command:
$ groupadd sftponly
Set a user’s group:
$ usermod steve -g sftponly
To deny SSH shell access, run the following command:
$ usermod steve -s /bin/false
And set the user’s home directory:
$ sudo chown root /home/steve
$ sudo chmod go-w /home/steve
$ sudo mkdir /home/steve/writable
$ sudo chown steve:sftponly /home/steve/writable
$ sudo chmod ug+rwX /home/steve/writable


Finally, you probably need to restart SSH
$ service ssh restart

The SSH part should now be in order, but you should make sure that file permissions also are correct. If the chroot environment is in a user’s home directory both /home and /home/username must be owned by root and should have permissions along the lines of 755 or 750.
In other words, every folder leading up to and including the home folder must be owned by root, otherwise you will get the following error after logging in:
Write failed: Broken pipe
Couldn't read packet: Connection reset by peer

June 15, 2017

xxd reverse with an offset

When using xxd to reverse a hex dump file, if you hexdump file has a non-0 offset like this:

bc000000: 01 02 03 04 05 06 07 08  ........

You would need to use the "-s offset" option of the xxd. However, there is a bug in the code that makes this options only works as the FIRST option. Otherwise, it wouldn't work.

You want to do this:

xxd -s -0xbc000000 -r -g 1 test.dump test.bin

Basically the xxd is hardcoded to look for the offset at argv[2].

Another alternative:
https://github.com/pheehs/hexdump2bin/blob/master/hexdump2bin.py