La Vita è Bella
Monday, July 11, 2022
This blog will stop updating and be read-only in the near future
My new blog is at https://b.yuxuan.org/. The RSS feed for the new blog is https://b.yuxuan.org/rss.xml. The reason of the move is explained in A new blog, the first blog entry of my new blog.
I'll try to keep the permalinks of the the old blog to be available as much as possible (but with static, read-only renders).
16:19:09 by fishy - General - Permanent Link
Friday, September 04, 2020
Typed nil in Go
In Go, nil
, one of the "magic" literals, could actually have types. Although in most cases this won't bite you, in rare cases it does bite and causes bugs. I was bitten by it for the second time (as far as I can remember) earlier this week.
This was the commit that introduced the bug and this was the commit that fixed it. The key issue is on these 2 lines:
41 var r intner = args.R
42 if r == nil {
What does this code do?
args.R
is an optional argument to the function. The lines here are trying to do the fallback to the default implementation when it's absent.
Why does the old code not work?
On L42, when doing the comparison between r
and nil
, nil
is actually a typed nil to match the type of r
. Since r
has the type of intner
, this line actually implies:
if r == intner(nil) {
The problem is that args.R
actually have a different type: *math/rand.Rand
. When it's nil
, that's actually typed nil of (*math/rand.Rand)(nil)
. When assigning it into r
on L41, the type of the typed nil doesn't change, despite that r
has a different type (the assignment is allowed because *math/rand.Rand
is a concrete type that implements intner
interface, so it's "assignable" in Go's type system). As a result, the comparison is actually:
if (*math/rand.Rand)(nil) == intner(nil) {
Since they are different types of typed nils, the comparison would never be true
.
Why does the fix work?
In the fix, we changed to compare args.R
against nil
directly:
42 if args.R != nil {
So it always have the correct type for the typed nil.
tags: go, golang
09:32:40 by fishy - dev - Permanent Link
Sunday, July 26, 2020
Setting up a Linux laptop on 2020
I'm approaching 3 year anniversary at my current employer, which means my employer issued laptop is up to refresh. I asked for a Linux laptop to replace my Macbook Pro, as I have been disappointed with Apple's negligence to macOS for a long time. Actually if someone told me 3 years ago, when I first joined the company, that I could choose a Linux laptop, I was likely to ask for a Linux laptop back then.
The new laptop issued to me is Dell Precision 5540. It arrived yesterday and I spent most of yesterday and today setting it up. I tried to ask for a Chromebook but was denied. I also tried to ask for a lighter one (like Dell XPS 13) but was told that this is the only choice for Linux laptops for the fleet right now.
The iMac at home I bought at 2013 recently also had its hard drive died. So after I shipped my old Macbook Pro back to my employer, I will be officially without any Mac/macOS devices for the first time after using them for 14+ years. (Although in recent years I already shifted my personal use to ChromeOS as much as possible.)
Although for the past 14+ years I almost always have a Linux machine in the household, I mostly only use it as a home server that's not attached to any keyboard, mouse, nor display most of the time and only ssh into it to use it. So the whole setting up a Linux laptop thing is still kinda new to me.
The new laptop come with Ubuntu pre-installed, but I used that to create a bootable USB stick with Debian testing (bullseye) installer and wiped it with Debian. The setup process is much more smoother than my prior experience (from early 2000s). There were only a few issues I need to correct manually.
Installer with firmwares
One thing I always forgot is that for "modern" hardwares I do need the Debian installer with non-free firmwares. At first I used the netinst image for standard Debian testing, and during the installation process it warned me with 10+ missing firmwares, mostly for the WiFi chip, so I went back to download and burn the correct one again. Even with that one the installation process still warned with 2 missing firmwares, and from the look of them they are both for the WiFi chip. But with the prior experience I went ahead with it and the WiFi worked during the installation process, so I guess the non-free firmwares in the installer image are good enough to make WiFi workable, but probably not to its max capacity?
Bluetooth headsets
I installed Debian testing with KDE. After installation most things just work right out of the box, which is really nice. One thing I noticed not working are bluetooth headsets.
I used the KDE's builtin bluetooth widget to pair my my bluetooth headsets (yes, I have multiple sets). First I tried to pair it with Google Pixel Buds 2. The pairing process failed multiple times, and after it's finally paired, it can never connect to it. Then I tried Bowers & Wilkins PX. The pairing went smoothly, and the laptop can connect to it, but after connected to it I can't seem to be able to get any audio to be played on the headset instead of the built-in speaker.
After some googling, I did the following:
- Instead of using KDE's bluetooth widget, I used the bluetoothctl command line tool to do the pairing instead. That have no problem pairing and connecting to Google Pixel Buds 2.
- Apparently the pulseaudio-module-bluetooth package was not installed by default. After installed it and restart pulseaudio, I can choose (and they are actually chosen by default after connected) my bluetooth headsets as the default output device and play music through them.
- I also added load-module module-switch-on-connect to the end of /etc/pulse/default.pa config file per suggestions from the articles I read, but I'm not sure how important this step actually is. (EDIT: This is actually important. Without this step also the bluetooth device might appear to be the default output device but the sound doesn't go through there.)
- I also installed bluez package, also not sure how important this step is. (EDIT: apparently bluez is a dependency of pulseaudio-module-bluetooth so this is required but you don't need to install bluez explicitly.)
With those steps, I have no problem using my bluetooth headsets as A2DP device for output, but I still cannot use them for input (microphone). Guess I'll just stick with the laptop's builtin microphone for meetings for a while before I figure that out.
Setting Chrome as the default browser from Konsole
Although after installed Chrome and start it for the first time, it offered to set itself as the default browser across the system, apparently KDE has a different setting for default browser across KDE applications (the most important one being Konsole), and Konqueror is still the default browser when I click an URL inside Konsole. I need to change an additional setting inside KDE's System Settings to set Chrome as the default browser from there.
Google Drive client
I wrote a toy/experimental fuse implementation of Google Drive in Go in godrive-fuse, but that's too experimental and immature for really daily use, so I still installed google-drive-ocamlfuse for the daily use one. (EDIT: switched to rclone for daily Google Drive use.)
U2F device
I also purchased a Yubikey 5 nano to leave permanently in one of the USB-A ports. It works great for U2F purpose out of the box, without any additional work. But one problem is that when I touch it without U2F challenge in progress, it start to send random keys to my active window.
The reason is that when there's no U2F challenge in progress, touching it activates OTP mode. I don't really use it for any OTP purpose, so I disabled OTP mode via ykman config usb --disable OTP command (ykman comes from yubikey-manager package).
13:16:03 by fishy - linux - Permanent Link
Tuesday, January 01, 2019
Play with LIFX Tile devices
Over the last few days, during the holiday season, I wrote some Go code to play with my LIFX Tile device I bought several months ago. The result is a library, and the following 2 videos:
The main reason I wrote yet another Go library for LIFX LAN Protocol is that although there are several Go libraries available, at the time of writing none of them support tiles, and tiles are the stuff I'd like to play with. As the result of that, the library is incomplete and focused on tile related APIs. It's incomplete to the extreme that it cannot control normal LIFX lights yet, at the time of writing this blog post. But while designing the API, I intentionally made it very extendible and export as many stuff as possible. As an example, the tile library is in its own package by using the base library as a library and extending it.
Another reason I did this is this is fun and exciting. The LIFX LAN Protocol is on UDP, and the last time I wrote UDP related code was 10+ years ago. I needed to relearn a lot of stuff in the past few days.
The code is on GitHub.
22:52:23 by fishy - opensource - Permanent Link
Sunday, September 10, 2017
Kotlin Android in Bazel
The past week I took some of my spare time to make building Kotlin Android in Bazel work. The result is merged into pubref/rules_kotlin, which is kind of the "official" Kotlin rules for Bazel at the moment, and in its 0.4 release. Here I'm going to talk about some backgrounds and explain some technical details of this work.
Kotlin actually added Android support long before Android's announcement of official Kotlin support, and I did some attempts to make it work in Bazel, but to no avail. That's why I turned to Scala and sbt-android for my NotifBot project.
Recently I did some googling and found aaronj1335/bazel-kotlin-android, which is actually very nice, and the main inspiration of my work. I learnt a few interesting points from this project:
- The android_library rule on top of resource files will make the R class in Kotlin code work.
- The java_import rule with neverlink = 1 will make the Android SDK classes in Kotlin code work.
So I tried to rewrite NotifBot using Kotlin (from Scala), and followed the example of that project, but hit a problem: AAR libraries imported.
AAR libraries are kind of like JAR libraries, but with additional Android things in it. (e.g. resource files, AndroidManifest.xml, etc.) Since they are different from JAR files, when we generating the JAR file out of the Kotlin sources, they'll pose a problem.
To resolve this problem, I created the kotlin_android_library rule/macro to do the following things:
- Auto generate implicated "_res" and "_sdk" rules, which are the android_library and java_import rules from the aaronj1335/bazel-kotlin-android example.
- Auto generate an implicated "_aar" rule, which is an android_library rule of all the AAR dependencies, but with neverlink = 1, so that we won't try to include them into the generated JAR file.
- After generated the JAR file out of the Kotlin sources, auto generate an android_library rule, which includes the JAR file, all the AAR dependencies and the resource dependency so that this rule can be used alone in the android_binary rule.
With these changes, the rewriting of NotifBot in Kotlin worked. As a next step, I tried to rewrite OneKey in Kotlin (this time from Java directly). This rewriting discovered a bug in depended kotlin_library rules, and I fixed that.
After that the rewriting of OneKey was also a success, so I think the work of getting Kotlin Android support in Bazel is done. Yes it's still possible (and trivial) to add a kotlin_android_binary kind of rule to include everything in a single rule, but I think that's not very useful or needed. If you really think that will be useful in some cases, or if you find any bugs in the implementation, please do let me know.
If you are looking for an example of building Kotlin Android in Bazel, please take a look at my BUILD files for NotifBot and OneKey.
21:33:00 by fishy - opensource - Permanent Link
Tuesday, July 18, 2017
Force rotating glog in Go
If you use glog package in Go as your logger, one thing you'll notice is that the only way it rotates is by size. There's MaxSize defined and exported, and glog will rotate the log file when the current file exceeding it (1,800 MiB). There's no way to rotate by time, and there's no way to do manually log rotating.
Actually that's not true. There's a trick.
Did you notice that I emphasized the word "exported" when describing MaxSize? That's how we could implement manual log rotating.
Here is an example:
// RotateGlogInfo rotates glog info log
func RotateGlogInfo(message string, size uint64) {
prevMaxSize := glog.MaxSize
glog.MaxSize = size
glog.Info(message)
glog.Flush()
glog.MaxSize = prevMaxSize
}
The idea is simple: we change glog.MaxSize to a very small value, so that the next write will definitely makes it to rotate. After that, we just restore the default size value.
The key here is the size value passed in. This must be small enough to ensure that rotate happens, but it cannot be too small (e.g. 0 or 1), otherwise if another thread writes a log larger than size before we restore the old value, that log alone will occupy a single log file (we certainly do not want a lock to block all other logging threads while rotating). If you write logs often, a value like 1024 should work fine enough for you.
19:10:05 by fishy - dev - Permanent Link
Sunday, May 28, 2017
NotifBot: Android app to forward notifications to Telegram bot
In my last blog post, I discussed why do I need Android Auto compatible SmartThings notifications, and why my attempt, AutoNotif, was rejected by Google Play because of Android Auto nofication policies.
After the rejection, I decided to use Telegram bot instead, because Telegram's Android app has very good Android Auto support, and Telegram itself has a very good bot platform.
The result is NotifBot: an Android app with a Telegram bot. It's an open source project released under BSD license
The Android app was written in Scala. It's mostly the same as the previous app, AutoNotif, just changed sending direct Android Auto notifications into sending Telegram messages.
The Telegram bot was written in Go, running on Google App Engine. It handles both Telegram's webhook and Android app's requests to forward the message. All connections are encrypted using HTTPS, and on the client side there are basic token authentication, and the token is revocable anytime with /stop Telegram command. I used Cloud Datastore for persistent storage (telegram token, user tokens).
In order to keep it under AppEngine's free tier for as long as possible, I chose the standard environment, which has some weird restrictions on Go (e.g. only has Go 1.6, has AppEngine specified APIs for Datastore, etc.). But in the end that's not a really big problem.
Please give it a try, and send me feedbacks.
tags: android, telegram, go, scala
11:48:59 by fishy - opensource - Permanent Link
Sunday, March 19, 2017
SmartThings, MyQ and Scala
SmartThings
Ever since I became a homeowner last year, I started to explore the IoT/home automation world.
The first IoT product I decided to purchase was Schlage Connect deadbolt. The only problem with it is that to use something more than passcode, I need to pair it with some third-party home automation hub.
After some research, I chose SmartThings in the end. It worked great with the Schlage deadbolt.
After the lock and the hub, the next step was garage door. The only garage door controller SmartThings "supports" out of the box is GD00Z-4, so I just bought that.
Besides basic functions (e.g. check status and open/close from your phone), SmartThings also provided routines and "SmartApps". SmartApps are the real gem here: besides the ones published on SmartThings' "Marketplace", you can also just use the developer site to write your own SmartApps (the developer platform is based on Groovy) and publish to yourself. Since it's just so easy to get code running, a lot of developers just don't bother to get their apps reviewed and published in the "Marketplace". Instead, they just share the source code, and anyone can just copy the code to their own developer console.
At first I found an app that can close the garage door after N minutes, and did some modifications so that it can send me a notification at 3 minutes, close the garage door at 5 minutes, and did a check again at 6 minutes to see if it's still open (because regulations, the remote closing of garage door could fail because of obstacles and things).
That's useful, but not really "smart". What I want is auto open the garage door when I'm driving home, and auto close it when I'm leaving.
So I purchased an Arrival Sensor to put in my car, and starting to write my own SmartApp.
This should quite simple, right? Just open the garage door when the arrival sensor's status changed from leave to present, and close the garage door when the arrival sensor's status changed from present to leave. That's what I did initially, and that worked great. At least for a while.
Until one day, the arrival sensor just went haywire, starting to flipping status randomly, while it's physically in my garage, unmoved. That caused my garage door to open unexpected.
So I changed the code, added a condition that the auto open will only happen if it's more than N minutes after we last tried to auto close it. And that worked great. You know the catch, for a while, again.
One night it went haywire again, and this time it flips status on a longer interval -- longer than the 5 minutes I set on my version 2. My garage door opened several times that night.
So it's time for version 3. This time I changed the condition. Instead of only open it if it's N minutes after last close, I only auto open it after a true auto close -- defined as it really closed the garage door, or the garage door was already closed, but that's no longer than N minutes before the app tries to close it (for the case when I manually closed the garage door when leaving).
Version 3 worked great for several months now, I think this is the mature version. You can find the code on this gist.
MyQ
After I happily used my Presence and Garage Door SmartApp for a couple of months, one day the smart garage door controller just stopped working.
Looking at Amazon reviews, that seems to be a very common problem: A lot of users also had it suddenly stopped working after couple of months. The manufacturer didn't provide any warranty, instead they kicked the ball to the sellers. We contacted Amazon, and got full refund (as it's still within the first year).
Because of this being a widespread issue, I decided to buy something else. One of my friend once has his garage door motor broken, so he was forced to upgrade his garage door motor, and the new one he purchased has MyQ. He was happy with that, so I decided to buy MyQ this time (also it comes from the same manufacturer of my garage door motor, just my garage door motor doesn't have MyQ built-in).
I purchased the MyQ package, installed it, and installed their app. Wow that's really disappointing. Yes the app can check the status of the garage door and open/close it, but there's literally nothing more. They don't have any integration (they do have a Nest integration, but that means I can control Nest in MyQ app, not vice-versa. Like, what's the point of that?), and they don't even have any open API.
Digging around the internet, someone actually reverse engineered the closed API they use on their mobile apps, so they do "have" an API, just there's no guarantee when will it be broken.
At first I decided to write an Android app to do the automation. The idea was simple, I define an interface to detect that I'm on the car (it could be from bluetooth connection, or detected that Android Auto app is running, etc.), and an interface to detect my presence (it could be geofence, or from WiFi connection). Combining the two interfaces and the MyQ API, I could implement the automation I wanted.
On first step I implemented the MyQ API on Android, nothing special here. Then I implemented the bluetooth car detection.
The WiFi presence detection didn't work as I expected, and geofence will require some extra UI work, so as a temporary workaround, I implemented an Android Auto notification instead: the app send an Android Auto notification when it detected it's in the car, and when I replied the notification with keyword "open" or "close", it uses MyQ API to open or close the garage door. That way, although it's not automation I wanted, at least I can control my garage door by voice, which is supposed to be much safer than fiddling with the MyQ app.
But that didn't work as I expected either. I'm not sure if it's because of Nexus 5X's infamous low memory, but the Android Auto reply will have a very long latency before it actually works. Often I tried to reply the notification with "open" when I'm near home, and I still need to manually open my garage door when I'm actually home, and after I already parked the car and in the living room, the reply actually worked and the garage door is (finally) open. That's not useful at all.
So I gave up on the Android app, and looked back at SmartThings.
SmartApps are not the only thing you can write on SmartThings' developer platform. You can also implement custom Device Handlers, to bring other device into SmartThings platform. Someone actually tried to do the same thing a few years ago, but their device handler no longer works, probably because the MyQ API they used no longer works. As a result, I took a night to implement the new, working MyQ API in the old device handler, and it actually worked. I can finally use my garage automation SmartApp with MyQ garage controller again, at least before MyQ kills the API I use (I hope they'll provide real open APIs if they ever decided to do that).
Scala
As I'm using closed MyQ API, there's an insecure fear that my SmartThings device handler could break any day. The SmartApp will actually send a notification when it auto open or close the garage door, but as I'm using Android Auto now, Android Auto will only show its compatible notifications, and hide everything else (which makes sense, because you don't want to distract drivers). And SmartThings notifications belong to "everything else".
With my experience with that failed Android app attempt, I thought maybe I can write another Android app, this time just translate normal notifications into Android Auto compatible notifications, so I can know that it worked.
There's a catch, I don't want to use Java this time. Having a day job writing Go, writing Java code at night is quite a contextual switch makes my head aches. So I decided to use this project as an opportunity to learn Kotlin or Scala.
There's another catch, I don't want to use Gradle or Android Studio. I wrote my codes in vim, and I want to use some better building tools, preferably Bazel.
They both have Bazel support (Scala through Bazel offically and Kotlin through third-party Skylark rules), but neither supports Android.
There are some comparisons on the internet prefer Kotlin over Scala, also Scala has some bad reputation about its learning curve, so I slightly leaned towards Kotlin.
But Kotlin has no documentation about how to use their kotlinc compiler CLI to build Android apps. I have absolutely no clue how to make it work outside of Gradle. Scala, on the other hand, has sbt. It's not Bazel, but at least I like it much more than Gradle.
So I took one week to wrote the app in Scala. The result is on GitHub. This is quite a simple app so there's nothing fancy there. I didn't have a chance to use a lot of advanced Scala features. But I do like the features I used, like foreach over an Option to do null handling.
When I tried to publish it on Google Play last night, I got a problem: Android Auto only allow one type of notifications, and that's peer-to-peer messages. Because my app send notifications that's not peer-to-peer messages, it's rejected. I guess that's why SmartThings didn't make their notifications compatible with Android Auto in the first place.
If you find it useful and don't mind sideloading apps on your Android phone, you can find the apk on the GitHub release page. But you probably mind it, as do I (I don't sideload any app on my phone). So going forward I need to find another way, at least before Android Auto loosen their notification restrictions.
One idea is to make it a chat-bot with an chat app with existing Android Auto support, and the chat app I'm thinking about is Telegram. The reason is that it has both quite good Android Auto support, and bot API support.
So that is probably what I will do in the next few weeks. Please stay tuned. Thinking about it, maybe I'll actually make a poor man's PushBullet accidentally :)
tags: dev, smartthings, myq, iot, scala
18:52:00 by fishy - dev - Permanent Link
Tuesday, February 28, 2017
Comparing []byte's in Go
As I'm new to Go, I think it's quite understandable that I'm not familiar with the standard library yet, and reinvented the wheel sometimes.
One day I need to compare two []byte's. Since I'm not aware of any implementation in standard library, and this is simple enough, I wrote my own comparing function. It's something like:
func equal(b1, b2 []byte) bool {
if len(b1) != len(b2) {
return false
}
for i, b := range b1 {
if b != b2[i] {
return false
}
}
return true
}
A few days later, I saw my colleague used reflect.DeepEqual in our codebase. I thought "Oh that's cool! No need to reinvent the wheel!" So I removed my own implementation and used reflect.DeepEqual instead.
A few more days later, I accidentally discovered the existence of bytes.Equal. This is obviously faster than reflect.DeepEqual (because its feature is much simpler), so I'm going to replace them. But before that, I decided to do a simple benchmark test to see how much faster.
(In case you are not familiar with Go or Go's testing package, Go provided a very easy to use benchmark framework in it. Some might think the lack of features like Assume makes the testing code mouthful. I kind of agree with that, but I'm also mostly OK with that.)
The full benchmark test code can be found in this public gist. I'll just post the result here
BenchmarkEqual/reflect.DeepEqual-1024-4 20000 91193 ns/op BenchmarkEqual/equal-1024-4 3000000 544 ns/op BenchmarkEqual/bytes.Equal-1024-4 50000000 22.6 ns/op BenchmarkEqual/reflect.DeepEqual-1048576-4 20 89556304 ns/op BenchmarkEqual/equal-1048576-4 3000 536891 ns/op BenchmarkEqual/bytes.Equal-1048576-4 30000 44613 ns/op BenchmarkEqual/reflect.DeepEqual-67108864-4 1 5801186044 ns/op BenchmarkEqual/equal-67108864-4 30 37011544 ns/op BenchmarkEqual/bytes.Equal-67108864-4 200 8574768 ns/op BenchmarkNonEqual/reflect.DeepEqual-1024-4 5000000 280 ns/op BenchmarkNonEqual/equal-1024-4 500000000 3.46 ns/op BenchmarkNonEqual/bytes.Equal-1024-4 300000000 4.56 ns/op BenchmarkNonEqual/reflect.DeepEqual-1048576-4 5000000 272 ns/op BenchmarkNonEqual/equal-1048576-4 500000000 3.44 ns/op BenchmarkNonEqual/bytes.Equal-1048576-4 300000000 4.52 ns/op BenchmarkNonEqual/reflect.DeepEqual-67108864-4 5000000 269 ns/op BenchmarkNonEqual/equal-67108864-4 500000000 3.42 ns/op BenchmarkNonEqual/bytes.Equal-67108864-4 300000000 4.50 ns/op
In short, bytes.Equal is usually 3 orders of magnitude faster than reflect.DeepEqual when the result is true, and 2 orders of magnitude faster when the result is false. (Also, my own implementation is only 1 order of magnitude slower than bytes.Equal when the result is true, and slightly faster when the result is false :)
And of course I replaced reflect.DeepEqual with bytes.Equal in our code base as appropriate.
tags: dev, code, golang, bytes, compare
23:33:17 by fishy - dev - Permanent Link
Saturday, February 11, 2017
Some Go memory notes
I recently did some work on a backend server written in Go, and one of the work is to reduce the memory usage, as the old code will use ~110GB of memory steadily (this is not memory leak), mainly because of we used to strive for the speed of coding, not efficiency. With the help of Go's standard pprof tool (which is wonderful), we managed to lowered memory usage down to less than 15GB steadily. Here are some notes I learnt from the process:
You don't always want to read io.Reader into []byte
When you are passing some data around, they usually come in one of the two forms in Go: either an io.Reader (or sometimes io.ReadCloser, which is kind of compatible with io.Reader), or []byte.
[]byte is something more permanent: It just sits there, you can do whatever you want with it. io.Reader, on the other hand, is more fragile: You can only read it once, and it's gone after that (unless you saved what you just read).
Because of that, sometimes it's very tempting to read that io.Reader you just got from some function into []byte (using ioutil.ReadAll) to save it permanently.
But that comes with a price: you need memory to save that []byte you just read. Do you really need that?
Most of the functions you need to pass the data on will gladly accept io.Reader instead of []byte as the parameter, and there's a good reason for that: It's always trivial to create an io.Reader out of []byte (both bytes.NewReader and bytes.NewBuffer can achieve that goal, I did some benchmark test and there're no significant performance difference between them, but I still slightly prefer bytes.NewReader, because Occam's razor). If you only need to pass the data to a single function to process once, there's no need to read it into []byte and convert it back into io.Reader again.
Sometimes even if you need to process the data in two different functions, you still don't need to call ioutil.ReadAll. One common use case is that you have a content of a file in the form of io.Reader, and you want to get the content type first using http.DetectContentType before sending that file to your user. You don't need the whole content of the file for that. http.DetectContentType only needs the first 512 bytes. So you could just wrap your reader with bufio.Reader, then Peek 512 bytes for http.DetectContentType, after that you can just send your bufio.Reader to the function to your user.
Beware of []byte in JSON
In Go's JSON package, []byte is "encoded as a base64-encoded string". This is a very reasonable design choice, but if you are using JSON to pass large chunk of []byte between machines, you might want to think twice.
On sender's side, you first need to save the whole chunk of []byte into memory, then when doing the JSON encoding, you will need to save both the original chunk of []byte, and the whole encoded base64 string, which is larger than your original chunk by design. Congratulations, you just more than doubled your memory consumption.
It's the same on receiver's side. At first you need to save the whole JSON string, including the very large base64 string, into memory (because Go's JSON decoding is not really a streaming friendly operation, although Go's base64 decoding is streaming friendly, that doesn't help in this case). During the decoding, it will allocate another chunk of memory to save your original []byte. So in the end you'll spend the same or more memory as the sender side.
If you really need to pass big chunk []byte between machines, JSON is not your friend. You should use some binary and streaming protocol instead. Both gRPC and thrift will do that job nicely, and they both have good Go support.
Avoid "adding" strings directly
Imagine you have a function to join strings together like this (No you don't really want to implement this. strings.Join already implemented this feature for you. This is just a simple example):
func JoinStrings(parts []string) string {
var s string
for _, part := range parts {
s += part
}
return s
}
Every time you do s +=, Go will actually allocate a totally new, bigger string, copy previous s over, then append new part at the end. This wastes both CPU (for copying) and memory.
Instead, you can use bytes.Buffer, which works like StringBuilder in Java:
func JoinStrings(parts []string) string {
var buf bytes.Buffer
for _, part := range parts {
buf.WriteString(part)
}
return buf.String()
}
tags: dev, code, golang, memory
02:52:29 by fishy - dev - Permanent Link