La Vita è Bella

In Go, nil, one of the "magic" literals, could actually have types. Although in most cases this won't bite you, in rare cases it does bite and causes bugs. I was bitten by it for the second time (as far as I can remember) earlier this week.

This was the commit that introduced the bug and this was the commit that fixed it. The key issue is on these 2 lines:

41   var r intner = args.R
42   if r == nil {

What does this code do?

args.R is an optional argument to the function. The lines here are trying to do the fallback to the default implementation when it's absent.

Why does the old code not work?

On L42, when doing the comparison between r and nil, nil is actually a typed nil to match the type of r. Since r has the type of intner, this line actually implies:

if r == intner(nil) {

The problem is that args.R actually have a different type: *math/rand.Rand. When it's nil, that's actually typed nil of (*math/rand.Rand)(nil). When assigning it into r on L41, the type of the typed nil doesn't change, despite that r has a different type (the assignment is allowed because *math/rand.Rand is a concrete type that implements intner interface, so it's "assignable" in Go's type system). As a result, the comparison is actually:

if (*math/rand.Rand)(nil) == intner(nil) {

Since they are different types of typed nils, the comparison would never be true.

Why does the fix work?

In the fix, we changed to compare args.R against nil directly:

42   if args.R != nil {

So it always have the correct type for the typed nil.

If you use glog package in Go as your logger, one thing you'll notice is that the only way it rotates is by size. There's MaxSize defined and exported, and glog will rotate the log file when the current file exceeding it (1,800 MiB). There's no way to rotate by time, and there's no way to do manually log rotating.

Actually that's not true. There's a trick.

Did you notice that I emphasized the word "exported" when describing MaxSize? That's how we could implement manual log rotating.

Here is an example:

// RotateGlogInfo rotates glog info log
func RotateGlogInfo(message string, size uint64) {
  prevMaxSize := glog.MaxSize
  glog.MaxSize = size
  glog.Info(message)
  glog.Flush()
  glog.MaxSize = prevMaxSize
}

The idea is simple: we change glog.MaxSize to a very small value, so that the next write will definitely makes it to rotate. After that, we just restore the default size value.

The key here is the size value passed in. This must be small enough to ensure that rotate happens, but it cannot be too small (e.g. 0 or 1), otherwise if another thread writes a log larger than size before we restore the old value, that log alone will occupy a single log file (we certainly do not want a lock to block all other logging threads while rotating). If you write logs often, a value like 1024 should work fine enough for you.

SmartThings

Ever since I became a homeowner last year, I started to explore the IoT/home automation world.

The first IoT product I decided to purchase was Schlage Connect deadbolt. The only problem with it is that to use something more than passcode, I need to pair it with some third-party home automation hub.

After some research, I chose SmartThings in the end. It worked great with the Schlage deadbolt.

After the lock and the hub, the next step was garage door. The only garage door controller SmartThings "supports" out of the box is GD00Z-4, so I just bought that.

Besides basic functions (e.g. check status and open/close from your phone), SmartThings also provided routines and "SmartApps". SmartApps are the real gem here: besides the ones published on SmartThings' "Marketplace", you can also just use the developer site to write your own SmartApps (the developer platform is based on Groovy) and publish to yourself. Since it's just so easy to get code running, a lot of developers just don't bother to get their apps reviewed and published in the "Marketplace". Instead, they just share the source code, and anyone can just copy the code to their own developer console.

At first I found an app that can close the garage door after N minutes, and did some modifications so that it can send me a notification at 3 minutes, close the garage door at 5 minutes, and did a check again at 6 minutes to see if it's still open (because regulations, the remote closing of garage door could fail because of obstacles and things).

That's useful, but not really "smart". What I want is auto open the garage door when I'm driving home, and auto close it when I'm leaving.

So I purchased an Arrival Sensor to put in my car, and starting to write my own SmartApp.

This should quite simple, right? Just open the garage door when the arrival sensor's status changed from leave to present, and close the garage door when the arrival sensor's status changed from present to leave. That's what I did initially, and that worked great. At least for a while.

Until one day, the arrival sensor just went haywire, starting to flipping status randomly, while it's physically in my garage, unmoved. That caused my garage door to open unexpected.

So I changed the code, added a condition that the auto open will only happen if it's more than N minutes after we last tried to auto close it. And that worked great. You know the catch, for a while, again.

One night it went haywire again, and this time it flips status on a longer interval -- longer than the 5 minutes I set on my version 2. My garage door opened several times that night.

So it's time for version 3. This time I changed the condition. Instead of only open it if it's N minutes after last close, I only auto open it after a true auto close -- defined as it really closed the garage door, or the garage door was already closed, but that's no longer than N minutes before the app tries to close it (for the case when I manually closed the garage door when leaving).

Version 3 worked great for several months now, I think this is the mature version. You can find the code on this gist.

MyQ

After I happily used my Presence and Garage Door SmartApp for a couple of months, one day the smart garage door controller just stopped working.

Looking at Amazon reviews, that seems to be a very common problem: A lot of users also had it suddenly stopped working after couple of months. The manufacturer didn't provide any warranty, instead they kicked the ball to the sellers. We contacted Amazon, and got full refund (as it's still within the first year).

Because of this being a widespread issue, I decided to buy something else. One of my friend once has his garage door motor broken, so he was forced to upgrade his garage door motor, and the new one he purchased has MyQ. He was happy with that, so I decided to buy MyQ this time (also it comes from the same manufacturer of my garage door motor, just my garage door motor doesn't have MyQ built-in).

I purchased the MyQ package, installed it, and installed their app. Wow that's really disappointing. Yes the app can check the status of the garage door and open/close it, but there's literally nothing more. They don't have any integration (they do have a Nest integration, but that means I can control Nest in MyQ app, not vice-versa. Like, what's the point of that?), and they don't even have any open API.

Digging around the internet, someone actually reverse engineered the closed API they use on their mobile apps, so they do "have" an API, just there's no guarantee when will it be broken.

At first I decided to write an Android app to do the automation. The idea was simple, I define an interface to detect that I'm on the car (it could be from bluetooth connection, or detected that Android Auto app is running, etc.), and an interface to detect my presence (it could be geofence, or from WiFi connection). Combining the two interfaces and the MyQ API, I could implement the automation I wanted.

On first step I implemented the MyQ API on Android, nothing special here. Then I implemented the bluetooth car detection.

The WiFi presence detection didn't work as I expected, and geofence will require some extra UI work, so as a temporary workaround, I implemented an Android Auto notification instead: the app send an Android Auto notification when it detected it's in the car, and when I replied the notification with keyword "open" or "close", it uses MyQ API to open or close the garage door. That way, although it's not automation I wanted, at least I can control my garage door by voice, which is supposed to be much safer than fiddling with the MyQ app.

But that didn't work as I expected either. I'm not sure if it's because of Nexus 5X's infamous low memory, but the Android Auto reply will have a very long latency before it actually works. Often I tried to reply the notification with "open" when I'm near home, and I still need to manually open my garage door when I'm actually home, and after I already parked the car and in the living room, the reply actually worked and the garage door is (finally) open. That's not useful at all.

So I gave up on the Android app, and looked back at SmartThings.

SmartApps are not the only thing you can write on SmartThings' developer platform. You can also implement custom Device Handlers, to bring other device into SmartThings platform. Someone actually tried to do the same thing a few years ago, but their device handler no longer works, probably because the MyQ API they used no longer works. As a result, I took a night to implement the new, working MyQ API in the old device handler, and it actually worked. I can finally use my garage automation SmartApp with MyQ garage controller again, at least before MyQ kills the API I use (I hope they'll provide real open APIs if they ever decided to do that).

Scala

As I'm using closed MyQ API, there's an insecure fear that my SmartThings device handler could break any day. The SmartApp will actually send a notification when it auto open or close the garage door, but as I'm using Android Auto now, Android Auto will only show its compatible notifications, and hide everything else (which makes sense, because you don't want to distract drivers). And SmartThings notifications belong to "everything else".

With my experience with that failed Android app attempt, I thought maybe I can write another Android app, this time just translate normal notifications into Android Auto compatible notifications, so I can know that it worked.

There's a catch, I don't want to use Java this time. Having a day job writing Go, writing Java code at night is quite a contextual switch makes my head aches. So I decided to use this project as an opportunity to learn Kotlin or Scala.

There's another catch, I don't want to use Gradle or Android Studio. I wrote my codes in vim, and I want to use some better building tools, preferably Bazel.

They both have Bazel support (Scala through Bazel offically and Kotlin through third-party Skylark rules), but neither supports Android.

There are some comparisons on the internet prefer Kotlin over Scala, also Scala has some bad reputation about its learning curve, so I slightly leaned towards Kotlin.

But Kotlin has no documentation about how to use their kotlinc compiler CLI to build Android apps. I have absolutely no clue how to make it work outside of Gradle. Scala, on the other hand, has sbt. It's not Bazel, but at least I like it much more than Gradle.

So I took one week to wrote the app in Scala. The result is on GitHub. This is quite a simple app so there's nothing fancy there. I didn't have a chance to use a lot of advanced Scala features. But I do like the features I used, like foreach over an Option to do null handling.

When I tried to publish it on Google Play last night, I got a problem: Android Auto only allow one type of notifications, and that's peer-to-peer messages. Because my app send notifications that's not peer-to-peer messages, it's rejected. I guess that's why SmartThings didn't make their notifications compatible with Android Auto in the first place.

If you find it useful and don't mind sideloading apps on your Android phone, you can find the apk on the GitHub release page. But you probably mind it, as do I (I don't sideload any app on my phone). So going forward I need to find another way, at least before Android Auto loosen their notification restrictions.

One idea is to make it a chat-bot with an chat app with existing Android Auto support, and the chat app I'm thinking about is Telegram. The reason is that it has both quite good Android Auto support, and bot API support.

So that is probably what I will do in the next few weeks. Please stay tuned. Thinking about it, maybe I'll actually make a poor man's PushBullet accidentally :)

As I'm new to Go, I think it's quite understandable that I'm not familiar with the standard library yet, and reinvented the wheel sometimes.

One day I need to compare two []byte's. Since I'm not aware of any implementation in standard library, and this is simple enough, I wrote my own comparing function. It's something like:

func equal(b1, b2 []byte) bool {
  if len(b1) != len(b2) {
    return false
  }
  for i, b := range b1 {
    if b != b2[i] {
      return false
    }
  }
  return true
}

A few days later, I saw my colleague used reflect.DeepEqual in our codebase. I thought "Oh that's cool! No need to reinvent the wheel!" So I removed my own implementation and used reflect.DeepEqual instead.

A few more days later, I accidentally discovered the existence of bytes.Equal. This is obviously faster than reflect.DeepEqual (because its feature is much simpler), so I'm going to replace them. But before that, I decided to do a simple benchmark test to see how much faster.

(In case you are not familiar with Go or Go's testing package, Go provided a very easy to use benchmark framework in it. Some might think the lack of features like Assume makes the testing code mouthful. I kind of agree with that, but I'm also mostly OK with that.)

The full benchmark test code can be found in this public gist. I'll just post the result here

BenchmarkEqual/reflect.DeepEqual-1024-4         	   20000	     91193 ns/op
BenchmarkEqual/equal-1024-4                     	 3000000	       544 ns/op
BenchmarkEqual/bytes.Equal-1024-4               	50000000	        22.6 ns/op
BenchmarkEqual/reflect.DeepEqual-1048576-4      	      20	  89556304 ns/op
BenchmarkEqual/equal-1048576-4                  	    3000	    536891 ns/op
BenchmarkEqual/bytes.Equal-1048576-4            	   30000	     44613 ns/op
BenchmarkEqual/reflect.DeepEqual-67108864-4     	       1	5801186044 ns/op
BenchmarkEqual/equal-67108864-4                 	      30	  37011544 ns/op
BenchmarkEqual/bytes.Equal-67108864-4           	     200	   8574768 ns/op
BenchmarkNonEqual/reflect.DeepEqual-1024-4      	 5000000	       280 ns/op
BenchmarkNonEqual/equal-1024-4                  	500000000	         3.46 ns/op
BenchmarkNonEqual/bytes.Equal-1024-4            	300000000	         4.56 ns/op
BenchmarkNonEqual/reflect.DeepEqual-1048576-4   	 5000000	       272 ns/op
BenchmarkNonEqual/equal-1048576-4               	500000000	         3.44 ns/op
BenchmarkNonEqual/bytes.Equal-1048576-4         	300000000	         4.52 ns/op
BenchmarkNonEqual/reflect.DeepEqual-67108864-4  	 5000000	       269 ns/op
BenchmarkNonEqual/equal-67108864-4              	500000000	         3.42 ns/op
BenchmarkNonEqual/bytes.Equal-67108864-4        	300000000	         4.50 ns/op

In short, bytes.Equal is usually 3 orders of magnitude faster than reflect.DeepEqual when the result is true, and 2 orders of magnitude faster when the result is false. (Also, my own implementation is only 1 order of magnitude slower than bytes.Equal when the result is true, and slightly faster when the result is false :)

And of course I replaced reflect.DeepEqual with bytes.Equal in our code base as appropriate.

I recently did some work on a backend server written in Go, and one of the work is to reduce the memory usage, as the old code will use ~110GB of memory steadily (this is not memory leak), mainly because of we used to strive for the speed of coding, not efficiency. With the help of Go's standard pprof tool (which is wonderful), we managed to lowered memory usage down to less than 15GB steadily. Here are some notes I learnt from the process:

You don't always want to read io.Reader into []byte

When you are passing some data around, they usually come in one of the two forms in Go: either an io.Reader (or sometimes io.ReadCloser, which is kind of compatible with io.Reader), or []byte.

[]byte is something more permanent: It just sits there, you can do whatever you want with it. io.Reader, on the other hand, is more fragile: You can only read it once, and it's gone after that (unless you saved what you just read).

Because of that, sometimes it's very tempting to read that io.Reader you just got from some function into []byte (using ioutil.ReadAll) to save it permanently.

But that comes with a price: you need memory to save that []byte you just read. Do you really need that?

Most of the functions you need to pass the data on will gladly accept io.Reader instead of []byte as the parameter, and there's a good reason for that: It's always trivial to create an io.Reader out of []byte (both bytes.NewReader and bytes.NewBuffer can achieve that goal, I did some benchmark test and there're no significant performance difference between them, but I still slightly prefer bytes.NewReader, because Occam's razor). If you only need to pass the data to a single function to process once, there's no need to read it into []byte and convert it back into io.Reader again.

Sometimes even if you need to process the data in two different functions, you still don't need to call ioutil.ReadAll. One common use case is that you have a content of a file in the form of io.Reader, and you want to get the content type first using http.DetectContentType before sending that file to your user. You don't need the whole content of the file for that. http.DetectContentType only needs the first 512 bytes. So you could just wrap your reader with bufio.Reader, then Peek 512 bytes for http.DetectContentType, after that you can just send your bufio.Reader to the function to your user.

Beware of []byte in JSON

In Go's JSON package, []byte is "encoded as a base64-encoded string". This is a very reasonable design choice, but if you are using JSON to pass large chunk of []byte between machines, you might want to think twice.

On sender's side, you first need to save the whole chunk of []byte into memory, then when doing the JSON encoding, you will need to save both the original chunk of []byte, and the whole encoded base64 string, which is larger than your original chunk by design. Congratulations, you just more than doubled your memory consumption.

It's the same on receiver's side. At first you need to save the whole JSON string, including the very large base64 string, into memory (because Go's JSON decoding is not really a streaming friendly operation, although Go's base64 decoding is streaming friendly, that doesn't help in this case). During the decoding, it will allocate another chunk of memory to save your original []byte. So in the end you'll spend the same or more memory as the sender side.

If you really need to pass big chunk []byte between machines, JSON is not your friend. You should use some binary and streaming protocol instead. Both gRPC and thrift will do that job nicely, and they both have good Go support.

Avoid "adding" strings directly

Imagine you have a function to join strings together like this (No you don't really want to implement this. strings.Join already implemented this feature for you. This is just a simple example):

func JoinStrings(parts []string) string {
  var s string
  for _, part := range parts {
    s += part
  }
  return s
}

Every time you do s +=, Go will actually allocate a totally new, bigger string, copy previous s over, then append new part at the end. This wastes both CPU (for copying) and memory.

Instead, you can use bytes.Buffer, which works like StringBuilder in Java:

func JoinStrings(parts []string) string {
  var buf bytes.Buffer
  for _, part := range parts {
    buf.WriteString(part)
  }
  return buf.String()
}

I wrote a vim plugin named projtags, initially for loading tags file for projects. But as I can "set tags+=tags;" in my vimrc, it's not that useful on the tags file way. I expanded its feature to support commands for per projects, and this is much more useful :P Here are some examples:

Case 1: different coding styles

I did some squid hacking before. My coding style is as below:

set cindent
set autoindent
set smartindent
set tabstop=8
set softtabstop=8
set shiftwidth=8
set noexpandtab
set smarttab
set cino=:0,g0,t0

But squid coding style is different. They use 4 for shiftwidth. As hacking some project, you should always follow the original coding style. But add vim modeline into every file is also unacceptable. So I use projtags.vim to do this job:

let g:ProjTags += [["~/work/squid", ":set sw=4"]]

So that every time I'm editing a file within squid, vim will use 4 instead of 8 for shiftwidth, now I'm happy with both my own codes and squid codes.

Case 2: use ant as makeprg for java projects

First, I use a autocmd to set ant as makeprg for java files:

autocmd BufNewFile,BufRead *.java :setlocal makeprg=ant\ -s\ build.xml

It works well for java files. But as I did some Android development these days, I also have lots of xml files to edit (and then make to see the result). I can't use such a autocmd for xml's, as not all xml's are in a java project. So projtags.vim is again the answer:

let g:ProjTags += [["~/work/pmats", ":set mp=ant\\ -s\\ build.xml"]]

So I can always use ":make" for ant under my Android project now, no matter it's .java or .xml.

My vimrc segment for tags and projtags

set tags+=/usr/local/include/tags
set tags+=/usr/include/tags
set tags+=/opt/local/include/tags
set tags+=tags;

let g:ProjTags = []
let g:ProjTags += [["~/work/squid", ":set sw=4"]]
let g:ProjTags += [["~/work/pmats", ":set mp=ant\\ -s\\ build.xml"]]

Patches for projtags.vim (e.g. Windows support, I haven't tested it but I guess it won't work under Windows now) are welcomed :D You can get the code from the git repo for my various scripts.

Columbus V-900 is so far the GPS that best fit my requirements: it can log, it can take waypoints while logging, and it (can) have a big storage for logging (via TF card). I got one from WC, a friend to give it a try.

It log GPS tracks to CSV format, WC wrote a script to convert it to the GPX format, but without waypoints. So I rewrote a Python script (as I'm not so familiar with perl), to add the waypoints to the GPX file.

Get the script here, it will use the voice record filename as the name of the waypoint if available, or otherwise just "Waypoint #N". You may want to edit the converted GPX file to rename the waypoints.

This script will convert a line contains start and end IP separated by space like:

221.192.0.0 221.199.207.255

Into IP mask format (with comment) like:

#221.192.0.0 - 221.199.207.255
221.192.0.0/14
221.196.0.0/15
221.198.0.0/16
221.199.0.0/17
221.199.128.0/18
221.199.192.0/20

The script is:

 1 #!/usr/bin/env python
 2
 3 import sys
 4 import re
 5
 6 def ip2int(ip) :
 7         ret = 0
 8         match = re.match("(\d*)\.(\d*)\.(\d*)\.(\d*)", ip)
 9         if not match : return 0
10         for i in xrange(4) : ret = (ret << 8) + int(match.groups()[i])
11         return ret
12
13 def int2ip(ipnum) :
14         ip1 = ipnum >> 24
15         ip2 = ipnum >> 16 & 0xFF
16         ip3 = ipnum >> 8 & 0xFF
17         ip4 = ipnum & 0xFF
18         return "%d.%d.%d.%d" % (ip1, ip2, ip3, ip4)
19
20 def printrange(startip, endip) :
21         bits = 1
22         mask = 1
23         while bits < 32 :
24                 newip = startip | mask
25                 if (newip>endip) or (((startip>>bits) << bits) != startip) :
26                         bits = bits - 1
27                         mask = mask >> 1
28                         break
29                 bits = bits + 1
30                 mask = (mask<<1) + 1
31         newip = startip | mask
32         bits = 32 - bits
33         print "%s/%d" % (int2ip(startip), bits)
34         if newip < endip :
35                 printrange(newip + 1, endip)
36
37 while 1 :
38         line = sys.stdin.readline().strip()
39         if not line : break
40         chars = line.split(" ")
41         print "#%s - %s" % (chars[0], chars[1])
42         ip1 = ip2int(chars[0])
43         ip2 = ip2int(chars[1])
44         printrange(ip1, ip2)
45

The emails sent by NucleusCMS (e.g. new comment notification) didn't encode the subject, but just put the raw text there. For english language file, that's OK. But for other languages such as Chinese, it's not that good. It will produce garbled text on the subject (but not always garbled, weird).

So I wrote this patch to resolve this problem. Google tell me that if I want to encode an email subject in PHP, I should use the mb_encode_mimeheader() function. But Dreamhost just didn't provide any mb_ functions in PHP. I dug more about email subject encoding, and found that a hardcoded base64 should just do the trick. As the original sending mail code in NucleusCMS hardcoded UTF-8 in the Content-Type, I assume that hardcode UTF-8 in the subject is fine, too.

My company uses svn, and a VPN without Mac client. So I can't commit from home. In order to manage my off-time works, I use git-svn on my MacBook Pro.

Today I'm going to sync my 2 commits in git to svn, using:

$ git svn dcommit

But after committed the first change, the svn server have no spaces left! So the second commit was failed:

End of file found: Can't read file '/tmp/report.tmp': End of file found at /usr/bin/git-svn line 3856

After svn admin resolved this problem, and I dcommit again, but it still fails:

Merge conflict during commit: File or directory '***.cpp' is out of date; try updating: The version resource does not correspond to the resource within the transaction. Either the requested version resource is out of date (needs to be updated), or the requested version resource is newer than the transaction root (restart the commit). at /usr/bin/git-svn line 461

And I've tried

$ git svn fetch

but the problem remains. So I decided to manually commit that change.

First, generate a patch from git ("HEAD^" means make a patch against the parent of HEAD, so it's my last change):

$ git format-patch HEAD^

And this command will generate a 0001--***.patch file. Then, use "svn co" to checkout a svn working copy, and apply that patch:

$ patch -i /path/to/0001--***.patch -p1

Then, get my last commit log from "git log", then use "svn ci" to commit it.

Now back to the git working copy, rebase svn:

$ git svn fetch
$ git svn rebase

Then try dcommit again, no conflicts and no commit made (as the svn repository is already up to date).

Hooray!