La Vita è Bella

We configured some Squid 2.6 servers that use external_acl_type to use some headers sent by client for access control. And the authentication isn't username/password routine, but use some tag to calculate hash. When running, the memory used by Squid just keep increasing from time to time, just like it have a memory leak. We tried to disable the acl on some server, and these servers runs just fine.

As external acl run in individual process, even if the acl program have memory leak, the memory used by the squid process shouldn't be growing.

We tried many ways to figure out the problem, but all fails. Finally someone noticed that in the external_acl_type documentation, there's a parameter named "cache", with this description:

result cache size, 0 is unbounded (default)

"unbounded"! So this is the problem. For username/password routine, cache is useful. the next time some user with the same username/password comes, Squid can get the result from cache without communicate with acl program. But for our authentication method, as the headers used to calculate hash is differ from every request, cache is totally useless.

I really hope that "0" means no cache and "-1" means unbounded. But anyway, set "cache" to 1 can do the trick. Now the Squids don't have memory problems anymore, although cache replace will slow them down a bit.

In my last blog, I use ndiswrapper for wireless driver and it can only use 802.11g, but not 802.11n. But today I've got the solution!

According to this article on EeeUser forum, the rt2860 chipset released the source code for Linux driver! Download them from the official website, and build it. You will need kernel header package to build the driver.

After successfully build, use "modprobe rt2860sta" to install this module, and you may also add the line "rt2860sta" to your "/etc/mmodules" file to load it automatically every time (but seems that modprobe have done this, you may don't need this step).

Now here's a problem: seems that wpa_supplicant didn't support this driver. So you need to set wireless parameters by iwpriv manually. And the biggest problem is WPAPSK, you can't just input your passphrase to do it. Luckily there's a webpage that can calculate the WPAPSK for us. I'm using WPA2 and it works. I'm not sure about WPA. But WPA is broken! Why don't you move to WPA2?

Save the below script to "/etc/restart_wireless.sh" and give it execute privilege:

iface=ra0
w="iwpriv $iface"

total_start=$(date +%s)

init_start=$(date +%s)
echo -en "iwpriv config..."
$w set NetworkType=Infra
$w set AuthMode=WPA2PSK
$w set EncrypType=AES
$w set SSID=Your SSID
# get WPAPSK from http://www.wireshark.org/tools/wpa-psk.html
$w set WPAPSK=Your WPAPSK
echo "done"
init_end=$(date +%s)
init_time=$(($init_end-$init_start))

assoc_start=$(date +%s)
assoc=0
assoc_report=10
assoc_loop=310
echo -n "Associating..."
for ((i=0; $i < $assoc_loop; i++)); do
        if [ "$(iwconfig ra0 2>/dev/null | head -1 | cut -f2 -d: | cut -f1 -d" ")" == "\"\"" ]; then
                if [[ $(( ($i+1) % $assoc_report )) == 0 ]]; then
                        echo -n .
                fi
        else
                echo done
                assoc=1
                break
        fi
        sleep 1
done

if [[ $assoc != 1 ]]; then
        echo failed
        exit 1
fi
assoc_end=$(date +%s)
assoc_time=$(($assoc_end-$assoc_start))

total_end=$(date +%s)
total_time=$(($total_end-$total_start))

echo -e "Time spent (sesconds)\n\tinit: $init_time\n\tassociation: $assoc_time\n\tTotal: $total_time"

Credit to a1l0a2k9, the above script is also from EeeUser forum, but I removed the DHCP part and modprobe part. If you are using DHCP, then you may need the DHCP part and modprobe part from the original script.

Now the "/etc/network/interfaces" part, add the following lines for the ra0 interface:

iface ra0 inet static
up /etc/restart_wireless.sh
address 10.0.14.14
netmask 255.255.255.0
gateway 10.0.14.1
auto ra0

(for DHCP users: change "static" to "manual" and remove the "address", "netmask" and "gateway" lines.)

And now, "ifup ra0", then you're done!

UPDATE: now we have 802.11n!

My old home server is dying these days, so I bought a new Asus Eee Box B202 to replace it. It uses Intel Atom N270 CPU, 1G memory, 80G harddisk, 10/100/1000 Ethernet and 802.11n wireless.

The first thing I do on it is install Debian Lenny, my favourite system for server.

Preparing USB flash for net install

As it didn't come with a cd-rom, I choose USB flash. I use the SD card from my camera and a USB card reader to combine a USB flash, and it can be used to boot successfully.

I prepared the USB flash according to the Debian Lenny documentation, but meet some problems:

1. Google for hd-media returned the hd-media link for Sarge as the first result, I used the boot.img.gz from Sarge and lenny-businesscard iso but the iso can't be found by the installer, so the installer (boot.img.gz) and the iso must match.
2. The hd-media from Lenny and the Lenny beta2 iso didn't match, neither. The installer can recognize the iso, but complain about mismatched kernel version, and prompt you that it need network update. But the installer from boot.img.gz didn't come with ethernet driver, so it will fail and can't continue.

So finally I have to use "the flexible way" and net install. I use the initrd.gz that have ethernet driver, and the vmlinuz from Lenny hd-media. None of the iso is needed (and you can't use them), all packages will be downloaded from one of the Debian mirrors.

After prepared the USB flash, DON'T FORGET to lock the write protection lock before boot. It will save your life later.

Install Debian

Boot from the prepared USB flash, and it contains only GRUB CLI, so you need to boot the installer manually:

root (hd0,0)
kernel /vmlinuz
initrd /initrd.gz
boot

Now you have a Debian Installer that can drive your ethernet card, so you're ready to install.

Install steps are normal, nothing more to say until the grub-install step.

grub-install failure

On the grub-install step, it will complain that grub-install (hd0) failed. Why? cause (hd0) is your USB flash and (hd1) is your harddisk! That's why lock the write-protection lock is important, or otherwise it may succeed without write your harddisk mbr. That's really stupid. Manually install grub on (hd1) and it will continue.

And the installation completes.

The X problem

After installation, you may find that your X didn't work. This is because it uses a Intel Graphic chipset for lap-top, but it's not lap-top. Ubuntu wiki have the solution, and it works.

Wireless driver

UPDATE: follow my next blog article for official driver and 802.11n!

After installation, the first important thing to do is to drive the wireless card. It uses AzureWare card which uses a rt2860 chipset. And luckily, ndiswrapper can do it.

Follow the instructions on Debian wiki to install ndiswrapper. The Windows driver is on the CD (you have another computer to read the CD and copy the driver to the Eee Box, do you?), I used the WIN2KXP one. After install ndiswrapper, it works.

But there are some problem in the /etc/network/interfaces, if you use WPA (I didn't get thie problem when using WEP):

auto wlan0
iface wlan0 inet static
address 10.0.14.14
netmask 255.255.255.0
gateway 10.0.14.1
wpa-conf /etc/wpa_supplicant.conf

The configure above can be used to connect router (that I can see it from router admin), but the IP 10.0.14.14 can't be reached by another computer in the subnet. But if you execute an extra:

# ifconfig wlan0 10.0.14.14

Then it will be OK. I don't know why but if I move the "auto wlan0" line after the "wpa-conf" line, it works fine. Maybe it must wait wpa-supplicant to do something first?

Another problem is that it can only use 802.11g wireless, if anyone knows how to drive it to 802.11n, please tell me

The end and photos

Finally, I have a new home server now.

My company uses svn, and a VPN without Mac client. So I can't commit from home. In order to manage my off-time works, I use git-svn on my MacBook Pro.

Today I'm going to sync my 2 commits in git to svn, using:

$ git svn dcommit

But after committed the first change, the svn server have no spaces left! So the second commit was failed:

End of file found: Can't read file '/tmp/report.tmp': End of file found at /usr/bin/git-svn line 3856

After svn admin resolved this problem, and I dcommit again, but it still fails:

Merge conflict during commit: File or directory '***.cpp' is out of date; try updating: The version resource does not correspond to the resource within the transaction. Either the requested version resource is out of date (needs to be updated), or the requested version resource is newer than the transaction root (restart the commit). at /usr/bin/git-svn line 461

And I've tried

$ git svn fetch

but the problem remains. So I decided to manually commit that change.

First, generate a patch from git ("HEAD^" means make a patch against the parent of HEAD, so it's my last change):

$ git format-patch HEAD^

And this command will generate a 0001--***.patch file. Then, use "svn co" to checkout a svn working copy, and apply that patch:

$ patch -i /path/to/0001--***.patch -p1

Then, get my last commit log from "git log", then use "svn ci" to commit it.

Now back to the git working copy, rebase svn:

$ git svn fetch
$ git svn rebase

Then try dcommit again, no conflicts and no commit made (as the svn repository is already up to date).

Hooray!

As Bug #428274, I can't accept any meeting requests sent to my work mailbox, and have to add the meetings into Lightning manually. This is really annoying. As the bug got fixed, I decided to upgrade Lightning to a 0.9pre nightly to get it working.

After upgrading, the following error dialog will greet me every time I start Thunderbird:

I googled this error message and found that it was caused by the incompatible extension ThunderBirthDay. OK, delete the calendar should fix it. No! The calendar was hide from Lightning's calendar list!

Fine, downgrade Lightning and then remove the calendar should work. But unfortunately, Lightning 0.9 have changed the storage format and can't downgrade to 0.8 anymore.

So how to delete that calendar? Maybe I should delete the calendar direct from Lightning's storage. According to the official FAQ, it's stored as an SQLite database file.

Under Thunderbird's profile directory, I got a "storage.sdb" file. I tried to use the command line sqlite to open it, and it works:

$ sqlite3 storage.sdb 
SQLite version 3.5.0
Enter ".help" for instructions
sqlite> .tables
cal_attachments              cal_metadata               
cal_attendees                cal_properties             
cal_calendar_schema_version  cal_recurrence             
cal_calendars                cal_relations              
cal_calendars_prefs          cal_todos                  
cal_calmgr_schema_version    cal_tz_version             
cal_events                 

The "cal_calendars" table seems to be the one I'm looking for:

sqlite> select * from cal_calendars;
1|storage|moz-profile-calendar://
2|thunderbirthday|moz-abmdbdirectory://abook.mab

Yes it is! so I need to know how to delete that record:

sqlite> .schema cal_calendars
CREATE TABLE cal_calendars (id INTEGER PRIMARY KEY, type TEXT, uri TEXT);

OK, then this should work:

sqlite> delete from cal_calendars where type = "thunderbirthday";
sqlite> select * from cal_calendars;
1|storage|moz-profile-calendar://

It DOES work!

Now my Thunderbird is quiet on starting.

But, the guys in Lightning really should give a chance to delete calendars that aren't compatible. I should file a bug for it. I've filed Bug #450121!

There's a macro in ACE to eliminate a compiler warning:

379 // Some compilers complain about "statement with no effect" with (a).
380 // This eliminates the warnings, and no code is generated for the null
381 // conditional statement.  @note that may only be true if -O is enabled,
382 // such as with GreenHills (ghs) 1.8.8.
383 # define ACE_UNUSED_ARG(a) do {/* null */} while (&a == 0)

But when I use this macro, gcc 4.3 will complain:

warning: the address of ‘a’ will never be NULL

I've started the Miao project one year ago, aim to build a smart pinyin input method for Mac. But I just got a new job 2 weeks after I started this project. This is quite a busy year, and I haven't really wrote much code for it.

A month ago Yong Sun from Sun begin to port SunPinyin to Mac, and now it's more than just works. I'm satisfied with the Mac port and use it everyday. As the Miao project is scratching my own itch, and I'm not itchy anymore, I should call it a day.

Sorry for anyone who waits for Miao, but I'm not going to reinvent the wheel.

On the Mac side, the initial backup takes me about 6 hours to backup 126GB data, by ethernet cable, so the speed is about 20GB/hr.

The writing speed via 802.11n wireless is as below:

fishy@Makelele:~/Video$ dd if=Pixar\ -\ Lifted.wmv of=/Volumes/Cacapa-1/foo.wmv
91582+1 records in
91582+1 records out
46890385 bytes transferred in 15.478993 secs (3029292 bytes/sec)

That's about 3MB/s. So I think the speed on Mac is acceptable, but it should be faster.

On the Linux side, I use Debian Lenny, with samba, afpfs-ng and a 100M ethernet cable (my Linux machine didn't come with a Gigabit ethernet port). The facts are:

Writing via AFP:

root@deBoer:~# dd if=prog03.mdb of=foo/foo.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 6.89061 s, 649 kB/s

Writing via SMB:

root@deBoer:~# dd if=prog03.mdb of=/media/cacapa/bar.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 6.57683 s, 680 kB/s

Reading via AFP:

root@deBoer:~# dd if=foo/bar.mdb of=bar.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 8.87068 s, 504 kB/s

Reading via SMB:

root@deBoer:~# dd if=/media/cacapa/foo.mdb of=foo.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 0.919922 s, 4.9 MB/s

So the conclusion is that, for writing, AFP and SMB are both slow. But for reading, SMB is much much faster than AFP, so you should use SMB to connect a Time Capsule on Linux.

After long waiting, I got one finally.

The box:

Unboxing:

And then working!

And here's a note, the command to limit my 1TB Time Capsule use only 500G for backup (so that I can use the other 500G for other stuffs), according to the comment of this hint:

hdiutil resize -size 500g -shrinkonly /Volumes/path/to/sparsebundle/

But the speed of backup is really really slow, that disappointed me. What a shame! It would be perfect if the backup speed up.

Finally, Video on Flickr!

I've uploaded a test video:

And here's the same video on youtube, to be compared:

So here's the comparison:

• Speed: youtube is much much faster than flickr, anyway, youtube consumed half of California's bandwidth, so it's no surprise.
• Who can upload: everybody, every free users can upload to youtube, but only pro users ($24.95/y) can upload videos to flickr now.
• Geotag: you can geotag your video on flickr, but seems that you can't do it on youtube.
• Original file: you can download the original file of your video in flickr's embed page, so in case you lost your original file, it's a way to get it back.
• Sizes: there's only 1 size on youtube, but many sizes on flickr to embed.
• Group: on flickr, you can group your videos along with your photos, so the photos/videos can be grouped together.
• Privacy control: you can filter viewers with your flickr contacts, as I already have a community on flickr, but not on youtube.

So, from now on, I may move from youtube to flickr, flickr rocks!