December 31, 2015

php code to normalize US phone number

This is the power of regex
Source: http://stackoverflow.com/questions/4708248/formatting-phone-numbers-in-php

This is a US phone formatter that works on more versions of numbers than any of the current answers.
$numbers = explode("\n", '(111) 222-3333
((111) 222-3333
1112223333
111 222-3333
111-222-3333
(111)2223333
+11234567890
    1-8002353551
    123-456-7890   -Hello!
+1 - 1234567890 
');


foreach($numbers as $number)
{
    print preg_replace('~.*(\d{3})[^\d]{0,7}(\d{3})[^\d]{0,7}(\d{4}).*~', '($1) $2-$3', $number). "\n";
}

And here is a breakdown of the regex:
Cell: +1 999-(555 0001)

.*          zero or more of anything "Cell: +1 "
(\d{3})     three digits "999"
[^\d]{0,7}  zero or up to 7 of something not a digit "-("
(\d{3})     three digits "555"
[^\d]{0,7}  zero or up to 7 of something not a digit " "
(\d{4})     four digits "0001"
.*          zero or more of anything ")"
Updated: March 11, 2015 to use {0,7} instead of {,7}

December 15, 2015

Linux routing based on IPtables MARK

http://www.linuxhorizon.ro/iproute2.html

Backup:

This page is a small HOWTO about the advanced linux routing...

First of all let me tell you where you can find the best source of information about the advanced routing under Linux. Most of you probably know or heard about the Linux Advanced Routing & Traffic Control site. There you can see a very comprehensive source of knowledge based not only on documentation but by easy to understand examples...
Credits: Linux Advanced Routing & Traffic Control, Thea
Ok, then...
This page will show you how to set a linux box to use 2 different ISPs on the same time...

First example:
Goal: To route packets that came from 4 network to different ISPs

Let's presume that you have two ISPs. In the following examples I'll use RDS and ASTRAL (two large ISPs from my country)
For the ASCII art and lynx console browser fans I'll use this kind of chart:
                                                                   ________
                                           +-------------+        /
                                           |    ISP 1    |       /
                             +-------------+    (RDS)    +------+
                             |             | gw 10.1.1.1 |     /
                      +------+-------+     +-------------+    / 
+----------------+    |     eth1     |                       /
|                |    |              |                      |
| Local networks +----+ Linux router |                      |  Internet cloud
|                |    |              |                      |
+----------------+    |     eth2     |                       \
                      +------+-------+     +-------------+    \
                             |             |    ISP 2    |     \
                             +-------------+  (ASTRAL)   +------+
                                           | gw 10.8.8.1 |       \
                                           +-------------+        \________
We will work only on Linux router box. From the root prompter do:
echo 1 RDS >> /etc/iproute2/rt_tables
echo 2 ASTRAL >> /etc/iproute2/rt_tables
The /etc/iproute2/rt_tables content after previous commands:
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
1 RDS
2 ASTRAL
Now we have three routing tables as follows: RDS table, ASTRAL table and the main table...
Let's fill up every table with the defaults routes:

The next step is to have some routing rules and routes:

For the RDS table:
ip route add default via 10.1.1.1 dev eth1 table RDS
ip rule add from 10.11.11.0/24 table RDS
ip rule add from 10.12.12.0/24 table RDS 
For the ASTRAL table:
ip route add default via 10.8.8.1 dev eth2 table ASTRAL
ip rule add from 10.22.22.0/24 table ASTRAL
ip rule add from 10.33.33.0/24 table ASTRAL
To see the routing tables:
ip route show table ASTRAL
ip route show table RDS
ip route show table main  # it's the same as "route -n" but in different format...
To see the routing tables:
ip rule show   # all the rule list
ip rule show | grep ASTRAL # only for ASRAL
ip rule show | grep RDS  # only for RDS
Let me explain the above rules.
The packets that came from the 10.11.11.0/24 and 10.12.12.0/24 networks will go to the RDS routing table and then (because we have a default route) will be passed to the RDS gateway. And similar, the packets that came from the 10.22.22.0/24 and 10.33.33.0/24 network will go to the ASTRAL gateway...
What is happening with the packets that came from other networks that are not shown in the above rules? Well, they just simply go to main routing table and follow the routing rules that reside there... If you want to block them to go to internet just delete the default route from the main table... (of course, doing that your router can not longer go to interent).


Second example:
Goal: To route the packets having the destination port 22/tcp to the RDS and 80/tcp to the ASTRAL (no matter what network generates them).
This example it is almost the same as the first one except that we will use iptables to mark the packets.

Same chart...
                                                                   ________
                                           +-------------+        /
                                           |    ISP 1    |       /
                             +-------------+    (RDS)    +------+
                             |             | gw 10.1.1.1 |     /
                      +------+-------+     +-------------+    / 
+----------------+    |     eth1     |                       /
|                |    |              |                      |
| Local networks +----+ Linux router |                      |  Internet cloud
|                |    |              |                      |
+----------------+    |     eth2     |                       \
                      +------+-------+     +-------------+    \
                             |             |    ISP 2    |     \
                             +-------------+  (ASTRAL)   +------+
                                           | gw 10.8.8.1 |       \
                                           +-------------+        \________

Same /etc/iproute2/rt_tables content:
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
1 RDS
2 ASTRAL
Before you start check your iptables configuration. I strongly recommend to read about iptables if you are unsure about what you will doing next.
For more documentation go to iptables home page or you can download a good documentation from this site (Security & Privacy Section) or directly from here.

To mark the packets that have the 22 and 80 as destination port we will use the MANGLE table...
iptables -A PREROUTING -t mangle -i eth0 -p tcp --dport 22 -j MARK --set-mark 1
iptables -A PREROUTING -t mangle -i eth0 -p tcp --dprot 80 -j MARK --set-mark 2
For the RDS table:
ip route add default via 10.1.1.1 dev eth1 table RDS # the same like in the first example
For the ASTRAL table:
ip route add default via 10.8.8.1 dev eth2 table ASTRAL # the same like in the first example
The next step is to have some routing rules based by the marked packets:

For the RDS:
ip rule add from all fwmark 1 table RDS
For the ASTRAL:
ip rule add from all fwmark 2 table ASTRAL
You can use the same commands to see the routing tables and rule lists as in the first example.
Now you have a routing solution based by the destination port...

December 14, 2015

RE: How to receive a million packets per second

https://blog.cloudflare.com/how-to-receive-a-million-packets/

rsyslogd dynamically create log file based on msg content

This is tested on a Ubuntu 10.04 system (should work on newer Ubuntu/Debian system too)

1. Create file /etc/rsyslog.d/30test.conf, with the following content:

$template DynFile,"/tmp/test-%msg:7:18%.log"
:msg,startswith," ABCD-001122334455" ?DynFile
#:syslogtag,startswith,"test" ?DynFile
#:syslogtag,startswith,"test" /tmp/test.log

2. Open file /etc/rsyslog.conf, and make the following modification for the following line:
$FileCreateMode 0644
comment out the following lines:
#$PrivDropToUser syslog
#$PrivDropToGroup syslog

3. sudo service rsyslog restart
4. logger -t test "ABCD-001122334455sdfsfsdfsdfsdfsdfs-----------"
5. Now check file /tmp/test-*.log, and you should see file /tmp/test-001122334455.log

To Log message with a defined template
1. First define the template
      $template shortlog,"%msg:10:1000%"
2. Use the template by appending ";template-name" to the end of the output file

Notes:

a. Property msg starts with a space " ". 
b. Eventually we should try to figure out why dropPriviledge does causes issues here.

TIPS:

A good debugs tip:
1. add /etc/rsyslog.d/debug.conf
*.* /var/log/all.log;RSYSLOG_DebugFormat
#Note here ";RSYSLOG_DebugFormat" is the output template
2. restart rsyslog
3. watch the file /var/log/all.log to see all log messages with property names

Log with UNIX Timestamp
$template unixTS,"%timegenerated:::date-unixtimestamp%,%msg%\n"
:msg,contains,"[UFW " /var/log/ufw.log;unixTS

#Note here ";unixTS" is the output template

http://www.rsyslog.com/tag/use-a-template/
Also refer to the rsyslog PDF version of the manual (Chapter 1) for overview of how rsyslogd processes messages)

Syntax Check
rsyslogd -N1

===============UPDATE: New Syntax for Syslog version 7 and newer =============
rsyslog's new syntax is called "RainerScript". To generate dynamic files using output templates, generate a file in /etc/rsyslogd.conf/10-mytest.conf with the following content, and then restart service rsyslogd. The following example only logs message from the application "abc" and the message itself starts with "ABC" to a dynamic file name.

template(name="myfilename" type="string" string="/tmp/my-%programname%.log")
template(name="shortlog" type="string" string="%msg:10:$%\n")
if  $programname == 'abc' and $msg startswith " ABC"  then {
    action(type="omfile" dynaFile="myfilename" template="shortlog" )

}


=============UPDATE: More examples==================
1. variables are set using "set" and they start with "$!" in Rainer Scripts
2. variables only work inside the if block
3. Inside template the variables are surrounded by %%. e.g. "%$!devid%"
4. "stop" has replaced the "~" to discard a particular message.

module(load="omprog")
template(name="devid" type="string" string="%$!devid%")
if ($programname == 'charon') then {
    if ($msg contains "deleting IKE_SA")  then {
        set $!devid= re_extract($msg, "([A-Z0-9])+-([A-Z0-9])+.vpnclient.meetcircle.co", 0, 0, "unknown");
        #action(type="omfile" file="/var/log/delete.log" template="devid" )
        action(type="omfwd" Target="192.168.1.2" Port="15140" Protocol="tcp" template="devid" )
        #action(type="omprog" binary="/usr/bin/rsyslogpost.sh" template="devid")
    }
}

:programname,isequal,"charon" stop

December 2, 2015

dnsmasq with Ubuntu 14.04

To make it work, you may need to edit the file
/etc/default/dnsmasq.conf

and uncomment this line:
IGNORE_RESOLVCONF=yes