tag:blogger.com,1999:blog-97975902024-03-11T14:00:11.937-07:00Journey of LifeUnknownnoreply@blogger.comBlogger686125tag:blogger.com,1999:blog-9797590.post-49791058866576703382022-01-03T14:47:00.002-08:002022-01-03T14:47:55.019-08:00AWS Cost EC2-Other<p> It usually consists the following cost:</p><p>1. EBS volumes<br class="jive-newline" />
2. Nat Gateway<br class="jive-newline" />
3. EBS snapshots<br class="jive-newline" />
4. Idle Elastic IPs<br class="jive-newline" />
5. Data transfer </p><p> </p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-79157822319196625342021-12-08T11:56:00.003-08:002021-12-08T11:56:44.633-08:00macos brew install java11<p> brew install java11</p><p>sudo ln -sfn /usr/local/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk</p><p><br /></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-77318111883423969102020-11-21T07:47:00.000-08:002020-11-21T07:47:08.122-08:002021 Car Brand Reliability Report from Consumer Reports<p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5AVvWsOLkcQ_34tfRi3Yl_iQB6rozAVx_gC93tFUp5iY5v2ivZqK4yvT3ttOqcX4b9X2PgyBFIoj_9V6TOuV8oPXxBfMMQvGA9USrFe70d18O2tTgaxSAkuCfQ8J_9tZuk5V5/s1910/106799526-1605799169988-20201119_CR_brand_ratings.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1910" data-original-width="1910" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5AVvWsOLkcQ_34tfRi3Yl_iQB6rozAVx_gC93tFUp5iY5v2ivZqK4yvT3ttOqcX4b9X2PgyBFIoj_9V6TOuV8oPXxBfMMQvGA9USrFe70d18O2tTgaxSAkuCfQ8J_9tZuk5V5/w640-h640/106799526-1605799169988-20201119_CR_brand_ratings.png" width="640" /></a></div><br /><p></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-87480687184667530882020-08-31T11:15:00.006-07:002020-08-31T11:15:44.356-07:00 openssl ssl version number interpretation<p><br /></p><p>in file: usr/include/openssl/opensslv.h of openssl source code. </p><div style="text-align: left;"><span style="font-size: x-small;"><span style="font-family: courier;">/*-<br /></span><span style="font-family: courier;"> * Numeric release version identifier:<br /></span><span style="font-family: courier;"> * MNNFFPPS: major minor fix patch status<br /></span><span style="font-family: courier;"> * The status nibble has one of the values 0 for development, 1 to e for betas<br /></span><span style="font-family: courier;"> * 1 to 14, and f for release. The patch level is exactly that.<br /></span><span style="font-family: courier;"> * For example:<br /></span><span style="font-family: courier;"> * 0.9.3-dev 0x00903000<br /></span><span style="font-family: courier;"> * 0.9.3-beta1 0x00903001<br /></span><span style="font-family: courier;"> * 0.9.3-beta2-dev 0x00903002<br /></span><span style="font-family: courier;"> * 0.9.3-beta2 0x00903002 (same as ...beta2-dev)<br /></span><span style="font-family: courier;"> * 0.9.3 0x0090300f<br /></span><span style="font-family: courier;"> * 0.9.3a 0x0090301f<br /></span><span style="font-family: courier;"> * 0.9.4 0x0090400f<br /></span><span style="font-family: courier;"> * 1.2.3z 0x102031af</span></span></div><p><br /></p><div>To get the above version number, you can all the following function: </div><div><div><span style="font-family: courier; font-size: x-small;">#include <openssl/opensslv.h></span></div><div><span style="font-family: courier; font-size: x-small;">#include <openssl/crypto.h></span></div><div><span style="font-family: courier; font-size: x-small;">void ssl_get_version(void) {</span></div><div><span style="font-family: courier; font-size: x-small;"> printf ("Using OpenSSL version %u\n", OpenSSL_version_num());</span></div><div><span style="font-family: courier; font-size: x-small;">}</span></div></div><div><br /></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-59433161811872850062020-08-16T08:36:00.002-07:002020-08-16T08:36:27.177-07:00ffmpeg command for generating Amazon Ads video<p> <span style="font-family: courier;">ffmpeg -i videoSharingFinalOutput.MOV -profile:v main -r 25 -vf scale=1280:720 output.mp4</span></p><div class="yj6qo"></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-27838802339408412712020-07-21T15:18:00.001-07:002020-07-21T15:18:41.534-07:00Golang short variable declaration gotcha<p>Golang allows <a href="https://golang.org/ref/spec#Short_variable_declarations">short variable declaration</a> in the form of <code>i:=0</code>, which defines a integer variable <code>i</code>. Golang also allows multi-variable short declaration, e.g.</p>
<pre><code>i,j := 0,1</code></pre>
<p>In this case, as long as there is one new variable on the left side, for example, <code>i</code>, the compiler would be happy. However, if <code>j</code> is not defined in the SAME BLOCK SCOPE, a new <code>j</code> will be created! Most times this is not what you want. Got to be very careful with that. A full example:</p>
<pre><code>func test(){
j:=1;
if j==1 {
i,j:=2,3
}
fmt.Println(j) // j is still 1!!
}</code></pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-58257389554709690862020-06-26T10:24:00.001-07:002020-06-26T10:24:39.269-07:00Golang range gotcha<p>In golang, when you range through an array of elements as in the following:</p>
<pre><code>// THIS CODE IS WRONG; DO NOT USE THIS
for i,v := range elements {
go do_something(v)
}</code></pre>
<p>In the above example, the value of v may change while <code>do_something</code> tries to used it. This is a race condition and will definitely cause problem. To fix this, one should use <code>elements[i]</code> instead;</p>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-59021291876410552782020-06-12T22:34:00.001-07:002020-06-12T22:34:14.589-07:00Vim typescript jump-to-definition<ol type="1">
<li>Use vim 8.1 or newer (in older version of vim, <code>tagstack</code> doesn’t work with the solution below)</li>
<li>Install ALE <a href="https://github.com/dense-analysis/ale" class="uri">https://github.com/dense-analysis/ale</a></li>
<li>bind <code>Ctrl-]</code> to <code>ALEGoToDefinition</code> using the following snippet of code</li>
</ol>
<pre><code>function ALELSPMappings()
let l:lsp_found=0
for l:linter in ale#linter#Get(&filetype) | if !empty(l:linter.lsp) | let l:lsp_found=1 | endif | endfor
if (l:lsp_found)
nnoremap <buffer> <C-]> :ALEGoToDefinition<CR>
nnoremap <buffer> <C-^> :ALEFindReferences<CR>
else
silent! unmap <buffer> <C-]>
silent! unmap <buffer> <C-^>
endif
endfunction
autocmd BufRead,FileType * call ALELSPMappings()</code></pre>
<ol start="4" type="1">
<li>install <code>tsserver</code> if you have not done so using <code>npm</code></li>
<li>Done.</li>
</ol>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-55559587265319845762020-06-11T12:28:00.002-07:002020-06-11T12:30:44.121-07:00AWS API Gateway Wildcard Path<p>In API Gateway configuration, typically one specifies an complete API Path, such as <code>/api/v1/fruits/list</code>. Sometimes, it may be necessary to specify a wildcard path that includes multiple complete API paths. This is how you can do it.</p>
<ol type="1">
<li>Use curly brackets. Specify <code>/api/v1/fruits/{action}</code> to allow any action to be accepted. Action can be any valid URL path segment, but it only covers one depth. It does not cover deeper URL like <code>api/v1/fruits/list/all</code>.</li>
<li>To cover all sub-links, use <code>{proxy+}</code>. The <code>+</code> sign tells API Gateway to accept all sub-links.</li>
<li>Note that you need a separate API for the root path itself, in this case <code>/api/v1/fruits</code> needs its own API. It’s possible to use <font face="courier" size="2">/api/v1/fruits/{action?}</font> to cover the root path. Needs to be verified since it’s not in the documentation.</li>
<li>In GoLang, the wildcard value is passed to the Lambda handler in a string-to-string map named PathParameters. <code>PathParameters:map[string]string{"id":"234"}</code></li>
</ol>
<p>See more at <a class="uri" href="https://aws.amazon.com/blogs/aws/api-gateway-update-new-features-simplify-api-development/">https://aws.amazon.com/blogs/aws/api-gateway-update-new-features-simplify-api-development/</a></p>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-1934252968687028872020-05-04T10:38:00.001-07:002020-05-04T10:47:50.113-07:00gitconfig<pre><code>[user]
name = MY NAME
email = MY EMAIL
[core]
editor = vim
whitespace = cr-at-eol
[merge]
tool = vimdiff
[alias]
vimdiff = difftool
st = status -sb
ci = commit -a
br = branch
co = checkout
df = diff --ignore-space-at-eol -w -b
dc = diff --cached
dl = diff --name-only
last = log -1 HEAD --stat
lg = log --stat
lb = log --graph --oneline --decorate --date=short --pretty=format:'%C(yellow)%h %Cred%ad %Cblue%an%Cgreen%d %Creset%s' --date=relative master..
tr = log --graph --oneline --decorate --all --date=short --pretty=format:'%C(yellow)%h %Cred%ad %Cblue%an%Cgreen%d %Creset%s' --date=relative
tr1 = log --graph --oneline --decorate --date=short --pretty=format:'%C(yellow)%h %Cred%ad %Cblue%an%Cgreen%d %Creset%s'
h = log --color --date=short --pretty=format:\"%C(yellow)%h%C(reset) %s%C(bold red)%d%C(reset) %C(green)%ad%C(reset) %C(blue)[%an]%C(reset)\" --decorate
who = shortlog -s --
out = log origin/master..master
in = "!git fetch; git log master..origin/master"
unstage = reset HEAD --
root="rev-parse --show-toplevel"
[diff]
renames = copy
tool = vimdiff
[difftool "vimdiff"]
cmd = "vim -d" "$LOCAL" "$REMOTE"
prompt = false
[push]
default = simple
[sendpack]
sideband = false
[pager]
status = true
[fetch]
prune = true</code></pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-1699486724974745792019-11-06T11:29:00.001-08:002019-11-06T11:29:00.608-08:00AWS API Gateway can be expensive<p>As of 2019/11, AWS charges $0.25 per million connection minutes. That sounds very innocent, but is it?</p>
<p>If your business grows and you have up to 1 million devices that needs to keep connected to an AWS Gateway, the cost, every month, just for the connection fee would be:</p>
<pre><code>$0.25*1440*30*1M/1M = $10,800 per month</code></pre>
<p>In this case, a highly available cluster may be a lot more cost effective.</p>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-86618932140285604652019-10-30T13:53:00.001-07:002019-10-30T16:25:40.989-07:00AWS VPC: Private Subnet vs Public SubnetIn AWS, a private subnet is defined as a subnet that doesn’t have a direct route to the Internet. It is only accessible from within the subnet, e.g. web server accessing internal database servers. If instances inside the private subnet needs to go out to the Internet, for example, updating packages, it will need to hop through a machine inside the public subnet. This can be done by a NAT instance (a dedicated instance just for this purpose) or by a NAT gateway ( a managed service provided by AWS).<br />
<table>
<colgroup>
<col style="width: 50%;"></col>
<col style="width: 50%;"></col>
</colgroup>
<thead>
<tr class="header">
<th>Public Subnet</th>
<th>Private Subnet</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Instances have public IP addresses</td>
<td>Instances do not have public IP address (selectable during instance creation)</td>
</tr>
<tr class="even">
<td>Route table attached to the subnet has a default route</td>
<td>Route table attached to the subnet has no default route<br />
<br /></td>
</tr>
</tbody>
</table>
<h3>
Internet Gateway </h3>
AWS Internet Gateway is a one-to-one private IP to public IP NAT device/service, not a regular NAT device like a home router. Instances inside the VPC subnet needs to have a public IP address associated with it. If not, Internet Gateway would not be able to route traffic for it.
<br />
<h4>
Egress-only Internet Gateway </h4>
IPv6 Only
<br />
<h4>
NAT gateway or NAT Instance </h4>
IPv4 Only<br />
<br />
<b>Internet Access for Default and Nondefault VPCs</b><br />
<br />
The following table provides an overview of whether your VPC automatically comes with
the
components required for internet access over IPv4 or IPv6.
<br />
<br />
<div class="table">
<div class="table-contents">
<table id="w109aac21c11b9c19">
<tbody>
<tr>
<th>Component</th>
<th>Default VPC</th>
<th>Nondefault VPC</th>
</tr>
<tr>
<td>Internet gateway</td>
<td>Yes</td>
<td>Yes, if you created the VPC using the first or second option in the VPC wizard.
Otherwise, you must manually create and attach the internet
gateway.
</td>
</tr>
<tr>
<td>Route table with route to internet gateway for IPv4 traffic (0.0.0.0/0)</td>
<td>Yes</td>
<td>Yes, if you created the VPC using the first or second option in the VPC wizard.
Otherwise, you must manually create the route table and add the route.
</td>
</tr>
<tr>
<td>Route table with route to internet gateway for IPv6 traffic (::/0)</td>
<td>No</td>
<td>Yes, if you created the VPC using the first or second option in the VPC wizard,
and if you specified the option to associate an IPv6 CIDR block with the VPC.
Otherwise, you must manually create the route table and add the route.
</td>
</tr>
<tr>
<td>Public IPv4 address automatically assigned to instance launched into
subnet
</td>
<td>Yes (default subnet)</td>
<td>No (nondefault subnet)</td>
</tr>
<tr>
<td>IPv6 address automatically assigned to instance launched into subnet</td>
<td>No (default subnet)</td>
<td>No (nondefault subnet)</td>
</tr>
</tbody></table>
</div>
</div>
<br />
<br />
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-58115064065814580452019-10-17T15:46:00.001-07:002019-10-23T14:55:11.043-07:00R ggplot2 geom_bar orderIn R, to plot a pie chart using a dataframe as the data source, one would use <code>geom_bar</code> first to plot a bar graph, and then use <code>coord_polar</code> to convert it to a pie chart. It works, but there is a catch:<br />
<ul>
<li><code>geom_bar</code> sorts the <code>fill</code> value alphabetically and always put the lowest value one on the top of the bar graph. For example, “apple” would be on the top of “pear”. This becomes a problem if you intend to label the pie chart.</li>
<li>The solution is to sort the <code>dataframe</code> in decreasing order by the <code>fill</code> variable. This way all calculations will be right.</li>
</ul>
<h3 id="example-r-code-with-correct-labels.">
Example R code with correct labels.</h3>
<pre><code>df=data.frame(val=c("z","b","c"),c2=c("b","d","f"),Freq=c(5,3,4))
df=df[order(df$val,decreasing=T),]
df$Label <- paste(df$val, paste(round((df$Freq/sum(df$Freq))*100,0),"%",sep=""), sep="-")
p <- ggplot(df,aes(x=1,y=Freq,fill=val))+geom_bar(stat="identity", color = "black")
p1 <- p + coord_polar(theta='y') + theme(axis.ticks=element_blank(),
axis.text.y=element_blank(),
axis.text.x=element_text(colour='black'),
axis.title=element_blank())
p2 <- p1 + scale_y_continuous(labels= df$Label,breaks=cumsum(df$Freq) - df$Freq/ 2)
p3 <- p2 + labs(title=title,fill=legend_title)+theme(plot.title = element_text(hjust = 0.5))
print(p3)
<b>Update on 10/23/2019
</b>
This article explains the order issue very clearly:
<a href="https://sebastiansauer.github.io/ordering-bars/">https://sebastiansauer.github.io/ordering-bars/</a>
In short, the rule for the sort order is:
<ul>
<li>if factor, the order of factor levels is used
</li>
<li>if character, an alphabetical order is used</li>
</ul>
<div>
So one can change the factor order to change the plot order.
</div>
<b></b><i></i><u></u><sub></sub><sup></sup><strike></strike>
</code></pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-84254242716448115862019-10-15T15:54:00.001-07:002019-10-15T15:54:23.957-07:00jq<p><code>jq</code> is a very useful tool to process JSON.</p>
<p>The idea of jq is centered on the idea of <code>pipe</code> and <code>data stream</code>.</p>
<p>Note below that a JSON entity is a self-contained and complete JSON object/array.</p>
<h3 id="pipes">Pipes</h3>
<ul>
<li>When jq is invoked, the input JSON file is parsed as one or multiple JSON entities.
<ul>
<li>The input file <em>can</em> have multiple JSON entities, separated by new lines or spaces. One stream is created for each JSON entity.</li>
<li>If multiple files are supplied to jq, each file should contain one or more self-contained JSON entities.</li>
<li>To combine all the multiple input JSON entities into one big JSON entity, pass the “–slurp” flag to jq. It will automatically create a containing JSON array and contain all input JSON entities into it. This includes the case of multiple input JSON entities in one input file, or multiple input files.</li>
</ul></li>
<li>In short, jq is started with one or multiple data streams which are from the input JSON entities.</li>
<li>Each stream is then passed through the specified <code>filter</code> independently.</li>
<li>One can pass the <code>-c</code> (compact) flag to jq to print each stream as one line.</li>
<li>Some key concepts of the filter.
<ul>
<li><code>.</code> means the entire input data stream</li>
<li><code>.[]</code> moves into the JSON object/array one layer, and splits the data stream into multiple streams, one for each entry in the object or array. For example. <code>{"a":1,"b":2}</code> will broken into two data streams after passing this filter, becoming <code>1</code> and <code>2</code> respectively. Note that they are the values of the entires and are not valid JSON anymore, which is okay but needs to be noted. The same streaming-splitting will also happen for JSON arrays after passing this filter.</li>
<li><code>.["key"]</code> generates one stream from the input stream. The shorthand is <code>.key</code>;</li>
<li><code>,</code> can be used to manually generate multiple streams.</li>
<li>To collect multiple streams into one stream, one can use <code>[ ]</code> to generate an array, or use <code>{ }</code> to generate an object. Note that to generate an object, one has to manually specify the keys, e.g.<code>{"key1":.a, "key2":.b}</code>, or use the shorthand <code>{key1,key2}</code> which is the same as <code>{"key1":.key1,"key2":.key2}</code>;</li>
<li><code>( )</code> can be used to instruct jq which part of the filter to be processed first, but it does not remove/combine/split the data streams.</li>
<li><code>|</code> has lower priority than <code>,</code>. Therefore, <code>1,2,3|3</code> will produce three streams first 1,2,3, respectively, and then pass each stream to the second filter <code>3</code>, with the end result of 3 streams of <code>3</code>. If we use <code>()</code>, we can change that order. <code>1,2,(4|3)</code> will process <code>4|3</code> first, generating <code>3</code>, then processing <code>1,2,3</code>, generating exact that as 3 streams of outoput. <code>1|4,2,3</code> will process <code>1</code> first, then passing to <code>4,2,3</code> three streams, causing the ending result to be <code>4,2,3</code> 3 streams.</li>
<li>The filter is not a programming language. Usually stream data is not outputted in the middle of the filter processing. It usually is only outputted at the end of the filter processing, one stream at a time.</li>
<li>One can use variable assignment such as <code>length as $array_length | .</code> to have jq remember a interim value and used in later stages of the filter.</li>
<li>On the very basic level, data stream is passed to each filter unchanged. For example, <code>.|.|.|.|.</code> is the same as <code>.</code></li>
<li><code>map(x)</code> takes an array as input, and outputs a new array, after running that filter for each element of the input array.</li>
<li>The <code>select()</code> function can be used to remove undesired streams. <code>[1,2,3] | map(select(. >= 2))</code> will give you <code>[2,3]</code></li>
<li>One can use <code>if-then-else-end</code> in the filter. Note that the <code>else</code> clause must be there. Sometimes the <code>empty</code> function can help.</li>
<li><code>a//b</code> The alternative operator can also be useful. It outputs a, if a is not false or null, otherwise it outputs b.</li>
<li><code>input_filename</code> can be used to find the input file name.</li>
<li>One can use <code>numbers/objects/arrays/etc.</code> to select results from multiple streams</li>
</ul></li>
<li>The final out is the multiple streams, one by one.</li>
</ul>
<p>Tags: jq</p>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-24091500450227366922019-10-15T15:32:00.001-07:002019-10-15T15:32:44.795-07:00IPv6 DNS server<pre><code>2600::
2600::1
2600::2</code></pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-78715520423816961202019-09-25T12:09:00.003-07:002019-09-25T12:09:16.306-07:00SSH: How to identify which key to use when AgentFowarding is in use <a href="https://superuser.com/questions/273037/using-the-identityfile-directive-in-ssh-config-when-agentforwarding-is-in-use">https://superuser.com/questions/273037/using-the-identityfile-directive-in-ssh-config-when-agentforwarding-is-in-use </a><br />
<br />
<div class="votecell post-layout--left">
<div class="js-voting-container grid fd-column ai-stretch gs4 fc-black-200" data-post-id="273094">
<div aria-label="accepted" class="js-accepted-answer-indicator grid--item fc-green-500 ta-center p4" role="note" tabindex="0" title="loading when this answer was accepted...">
</div>
</div>
</div>
<br />
<div class="answercell post-layout--right">
<div class="post-text" itemprop="text">
You can use the public part of a key to to specify which private key
you want to use from the forwarded agent. This requires creating an
extra file (the public part of the key) on any “intermediate” machines
(machines to which you forward your local <em>ssh-agent</em>).<br />
<ol>
<li>Arrange for the intermediate machine to have a copy of the public part of the desired key in a convenient location (e.g. <code>~/.ssh/some_other_key.pub</code>).<br />
From any machine that already has the public part of the key:<br />
<pre><code>scp some_other_key.pub intermediate:.ssh/
</code></pre>
or, on the intermediate machine:<br />
<pre><code>ssh-add -L | grep something_unique > ~/.ssh/some_other_key.pub
</code></pre>
You may want to edit the trailing “comment” part of the public key to
better identify the key’s origin/owner/purpose (or attempt to hide the
same).</li>
<li>Use the pathname to the above public key file with <code>-i</code> or <code>IdentityFile</code>.</li>
<li>You may also need to use <code>IdentitiesOnly yes</code> (in <code>.ssh/config</code> or <code>-o</code>) to keep <em>ssh</em> from trying to offer any additional identities from your forwarded agent.</li>
</ol>
</div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-67643603912808101632019-09-08T17:23:00.001-07:002019-09-08T17:33:40.098-07:00Lego EV3 Sound file .rsf formatLego EV3 support play sound files, and the Lego programmer software comes with many built-in sound files. If you want to add new sound files, you can use the "Sound Editor" that came with the software. Or you can convert an existing sound file to it.<br />
<br />
The found file ends with .rsf extension (probably standing for Robotic Sound File). It has the following format:<br />
<br />
[ 8 bytes of header]<br />
[ raw sound data ]<br />
<br />
<b>The first 8 bytes of the file are meta data with the following meaning:</b><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">byte 0, byte 1: 0x01 0x00</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">byte 2, byte 3: length, in big-endian, of raw sound data.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">byte 4, byte 5: 0x1f, 0x40 (demical 8000, the sampling rate)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">byte 6, byte 7: 0x00, 0x00.</span><br />
<br />
<b>raw sound data</b><br />
The raw sound data is 8-bit of PCM data, with sampling rate of 8000 samples per second.<br />
<br />
<h3>
Example Script </h3>
<div>
On a Mac OS computer, one can use the following command to generate a audio file:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">say "hello world" -o hello.aiff</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">Then you can use "ffmpeg" (you need to use brew to install it) to convert it to raw audio</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">ffmpeg -i hello.aiff -acodec pcm_u8 -f u8 -ar 8000 hello.raw</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
Then you can use the above mentioned tool "raw2rsf" to conver the raw file to .rsf file</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">raw2rsf hello.raw > hello.rsf</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
Then you can copy the hello.rsf file to your Lego Programmer sound file directory and then use it from the programmer software!</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<br />
raw2rsf.c: a simple C program to convert a .raw file to .rsf file.<br />
<script src="https://gist.github.com/tiebingzhang/8a3f86e4b53f259aa521575fc5f13d12.js"></script>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-57965635422849297832019-08-23T16:44:00.001-07:002019-08-23T16:44:51.700-07:00Instant file sharing without logging in1. <a href="https://sharedrop.io/">https://sharedrop.io</a> Web-RTC based; Local LAN transfer doesn't leave LAN<br />
<br />
2. <a href="https://file.pizza/">https://file.pizza</a> HTTP streaming and Web-RTC based.<br />
<br />
3. <a href="https://ppng.ml/">https://ppng.ml</a> HTTP streaming, supporting curl based command line.<br />
<br />
Source code at: <a href="https://github.com/nwtgck/piping-server">https://github.com/nwtgck/piping-server</a>, and a minimal golang version at <a href="https://github.com/nwtgck/go-piping-server">https://github.com/nwtgck/go-piping-server</a><br />
<br />
Blog at <a href="https://dev.to/nwtgck">https://dev.to/nwtgck</a><br />
<br />
4. <a href="https://xkcd949.com/">https://xkcd949.com</a> HTTP Streaming<br />
<br />
<br />
WebRTC Demo using peer.js:<br />
<a href="https://jmcker.github.io/Peer-to-Peer-Cue-System/">https://jmcker.github.io/Peer-to-Peer-Cue-System/</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-64098418280797266332019-07-30T23:00:00.002-07:002019-07-30T23:00:09.452-07:00Windows 10 spotlight image files location<pre>%userprofile%\AppData\Local\Packages\Microsoft.Windows.ContentDeliveryManager_cw5n1h2txyewy\LocalState\Assets</pre>
<pre>
</pre>
<pre><a href="https://www.howtogeek.com/247643/how-to-save-windows-10s-lock-screen-spotlight-images-to-your-hard-drive/">https://www.howtogeek.com/247643/how-to-save-windows-10s-lock-screen-spotlight-images-to-your-hard-drive/</a></pre>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-60680276964785370512019-07-18T14:51:00.002-07:002019-07-18T14:52:08.411-07:00Search Linux Kernel to find out when a feature was addedUse the LKDDB (Linux Kernel Driver DataBase):<br />
<br />
<a href="https://cateee.net/lkddb/web-lkddb/IP_SET_HASH_IPMAC.html">https://cateee.net/lkddb/web-lkddb/IP_SET_HASH_IPMAC.html</a><br />
<br />
Search a CONFIG_xxx and it will tell you since what version of Linux kernel it was added.<br />
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-41077714918068121312019-05-15T13:13:00.001-07:002019-05-15T13:14:31.725-07:00TCP socket send buffer deep dive<br />
<br />
<article class="markdown-body"> <h2>
<br /></h2>
A typical TCP socket send buffer is composed of three parts: unacked-bytes, unsent-bytes, and free-buffer.<br />
<pre><code> +----------------+
| |
| |
| FREE BUFFER |
| |
| |
+----------------+
| |
| UNSENT BYTES |
| |
+----------------+
| |
| UNACKED BYTES |
| |
+----------------+
</code></pre>
<h3>
Total send buffer size</h3>
<pre><code>Total send buffer size = unacked-bytes + unsent-bytes + free-buffer.
</code></pre>
It can be obtained using the <code>SO_SNDBUF</code> socket option.
The buffer size could dynamically change its size as seen needed by the
OS. This works for both Linux and macOS.<br />
<pre><code class="language-C"> slen = sizeof(sndbufsiz);
err = getsockopt(sd, SOL_SOCKET, SO_SNDBUF, &sndbufsiz, &slen);
</code></pre>
<h3>
Total in-flight bytes</h3>
<pre><code>Total inflight bytes = unacked-bytes + unsent-bytes.
</code></pre>
It can be obtained using <code>SO_NWRITE</code> socket option on macOS, and <code>SIOCOUTQ</code> ioctl on Linux.<br />
<pre><code class="language-C">int get_socket_used(int sd){
int err;
int used;
#ifdef __APPLE__
socklen_t slen = sizeof(used);
err = getsockopt(sd, SOL_SOCKET, SO_NWRITE, &used,&slen);
if(err < 0) {
perror("getsockopt 2");
exit(1);
}
#else
err = ioctl(sd, SIOCOUTQ, &used);
if(err < 0) {
perror("ioctl SIOCOUTQ");
exit(1);
}
#endif
return used;
}
</code></pre>
<em>On macOS, it seems that this can also be obtained using the <code>TCP_INFO</code> struct, but it is a private API.</em><br />
<pre><code class="language-C">u_int32_t tcpi_snd_sbbytes; /* bytes in snd buffer including data inflight */
</code></pre>
<h3>
unacked-bytes</h3>
On Linux, unacked-bytes can be obtained from the <code>TCP_INFO</code> structure, but the result is number of segments, or bytes. <em>On macOS, <code>TCP_INFO</code> seems to contain this infomration (private API).</em><br />
<pre><code class="language-C">int get_socket_unacked(int sd){
struct tcp_info tcp_info;
socklen_t tcp_info_length = sizeof(tcp_info);
if ( getsockopt(sd, IPPROTO_TCP, TCP_INFO, (void *)&tcp_info, &tcp_info_length ) == 0 ) {
return tcp_info.tcpi_unacked;
}
return 0;
}
//For macOS, use TCP_INFO
u_int64_t tcpi_txunacked __attribute__((aligned(8))); /* current number of bytes not acknowledged */
</code></pre>
<a href="https://github.com/st3fan/osx-10.9/blob/master/xnu-2422.1.72/bsd/netinet/tcp.h">macOS tcp_info definition</a><br />
<h3>
unsent-bytes (not including un-acked bytes)</h3>
On Linux, unsent-bytes can be obtained from the <code>tcpi_notsent_bytes</code> field of the <code>TCP_INFO</code> structure. NOTE that this requires kernel version to be <code>4.6</code> or newer. For Android that means Android 8 or newer. On Linux, it can also be obtained using <code>SIOCOUTQNSD</code> ioctl. It’s not clear how to do this on macOS.<br />
<pre><code class="language-C">//defined in /usr/include/linux/sockios.h
int get_socket_unsent(int sd){
int err;
int unsent;
err = ioctl(sd, SIOCOUTQNSD, &unsent);
if(err < 0) {
perror("ioctl SIOCOUTQNSD");
exit(1);
}
return unsent;
}
//OR
int get_socket_unsent(int sd){
struct tcp_info tcp_info;
socklen_t tcp_info_length = sizeof(tcp_info);
if ( getsockopt(sd, IPPROTO_TCP, TCP_INFO, (void *)&tcp_info, &tcp_info_length ) == 0 ) {
return tcp_info.tcpi_notsent_bytes;
}
return 0;
}
</code></pre>
<a href="https://stackoverflow.com/questions/595426/how-to-get-amount-of-non-ack-ed-tcp-data-for-the-socket">Stackoverflow discussion on getting unsent bytes</a><br />
<h3>
epoll and kevent</h3>
epoll on Linux, and kevent on macOS, get triggered by the <code>unsent-bytes</code>, when the TCP option <code>TCP_NOTSENT_LOWAT</code> is set. On macOS, <code>kevent()</code> doesn’t report socket as writable until the unsent TCP data drops below speciï¬ed threshold (typically 8 kilobytes).<br />
<ul>
<li><a href="https://lwn.net/Articles/560082/">Linux TCP_NOTSENT_LOWAT documentation - Kernel 3.12 or newer</a></li>
<li><a href="https://github.com/dabeaz/curio/issues/83">Discussion of TCP_NOTSENT_LOWAT and select calls</a></li>
<li><a href="https://redmine.named-data.net/issues/4362/">More discussions</a></li>
<li><a href="https://devstreaming-cdn.apple.com/videos/wwdc/2015/719ui2k57m/719/719_your_app_and_next_generation_networks.pdf">Apple’s presentation on this</a></li>
</ul>
</article>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-56896197659030674272019-05-09T13:25:00.000-07:002019-05-09T13:25:41.518-07:00Enable user-id based packet routing on Mac OS<br />
If you would like to route all socket (TCP/UDP) traffic from processes running by a particular user on a Mac OS to be routed differently, you can do that.<br />
<br />
1. Add the user to your Mac OS if not already done. In this example, I will add an user named "test1"<br />
2. run the command:<br />
sudo vi /private/etc/pf.conf<br />
and add the following line before ' anchor "com.apple/*"<br />
pass out quick on en0 route-to { utun4 192.168.15.2 } user test1<br />
<br />
Note:<br />
a) change en0 to your default network interface name on Mac<br />
b) change utun4 to the network interface you would these packets to be routed to<br />
<br />
3. restart pf by doing:<br />
sudo pfctl -d; sudo pfctl -e -f /etc/pf.conf<br />
<br />
Now all processes running by user test1 should be routed to the new interface as specified.<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-33720165728076320292019-02-26T07:28:00.002-08:002019-02-26T07:28:27.276-08:00keep ssh running in background for tunneling1. write the following script to a file, named "tunnel.sh" and make it executable (make sure user has public auth enabled on remote host):<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">while true; do ssh -t -n -R 127.0.0.1:2233:127.0.0.1:22 user@remote.host.com "while true; do ps -ef; sleep 1; done" ; sleep 1; done</span><br />
<br />
2. Run the the above script in a detached screen session:<br />
<pre class="lang-bsh prettyprint prettyprinted"><code><span class="pln"> screen </span><span class="pun">-</span><span class="pln">S tunnel </span><span class="pun">-</span><span class="pln">d </span><span class="pun">-</span><span class="pln">m /path/to/tunnel.sh</span></code></pre>
<pre class="lang-bsh prettyprint prettyprinted"><code><span class="pln">
</span></code></pre>
<pre class="lang-bsh prettyprint prettyprinted">
</pre>
That's all. This creates a background screen session, which runs the tunnel.sh script, which loops an ssh command to keep it up and running.<br />
<div>
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-43033303590516284852018-09-28T14:57:00.000-07:002018-09-28T14:57:02.362-07:00A great tool to generate pretty postman doc from postman Json file<br />
<a href="https://github.com/aubm/postmanerator">https://github.com/aubm/postmanerator</a><br />
<br />
Golang-based, generates beautiful HTML api document!Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-9797590.post-56488809545030300352018-07-28T04:43:00.005-07:002018-07-28T04:46:50.236-07:00<span style="font-family: "courier new" , "courier" , monospace;">How to use Netlink to get ipv6 neighbors</span><br />
<span style="font-family: "courier new";"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">1. open netlink socket<br />2. send request to get neighbors<br />3. parse the response</span><br />
<span style="font-family: "courier new";"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">The response is in the following format<br />[ netlink msg ] ... [ netlink msg ]</span><br />
<span style="font-family: "courier new";"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">use the Macros defined in "man 3 netlink" to iterate through the messages. Each netlink message has a header "struct nlmsghdr *", defined in "man 7 netlink".</span><br />
<span style="font-family: "courier new";"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">The netlink message for getting ipv6 neighbors is defined in "man 7 rtnetlink"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">for ipv6 neighbors message, each netlink msg block has the following format:<br /> netlink-msg-header | neighbor-discover-msg-header (struct ndmsg) | route-attributes (struct rtattr) | more route-attributes...</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Each route-attributes has the following format:<br /> header (struct rtattr) | data</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">The types of rtattr are defined in /usr/include/linux/neighbour.h, with the commons are:<br /> NDA_DST: data is IP address <br /> NDA_LLADDR: data is MAC address <br /> NDA_CACHEINFO: data is struct nda_cacheinfo</span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;">Example NetLink payload parsing (not showing nlmsghdr):</span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;">0a 00 00 00 : inet family<br />02 00 00 00 : ifindex<br />04 00 : state<br />00 : flags<br />01 : type</span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;">14 00 : rtattr len<br />01 00 : type<br />fe 80 00 00 00 00 00 00 b6 ef fa ff fe d0 fe 76 : Ipv6 address</span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;">0a 00 : rtattr len<br />02 00 : type<br />b4 ef fa d0 fe 76 : mac</span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;">00 00 //padding</span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;">08 00 //probe ?<br />04 00<br />00 00 00 00</span><br />
<span style="font-family: courier new;"></span><br />
<span style="font-family: courier new;">14 00 //rtattr len<br />03 00 //cache info<br />6b a8 e8 01 fb 90 e8 01<br />6b b9 69 00 00 00 00 00</span>Unknownnoreply@blogger.com0