Case 1: different coding styles

I did some squid hacking before. My coding style is as below:

set cindent
set autoindent
set smartindent
set tabstop=8
set softtabstop=8
set shiftwidth=8
set noexpandtab
set smarttab
set cino=:0,g0,t0

But squid coding style is different. They use 4 for shiftwidth. As hacking some project, you should always follow the original coding style. But add vim modeline into every file is also unacceptable. So I use projtags.vim to do this job:

let g:ProjTags += [["~/work/squid", ":set sw=4"]]

So that every time I'm editing a file within squid, vim will use 4 instead of 8 for shiftwidth, now I'm happy with both my own codes and squid codes.

Case 2: use ant as makeprg for java projects

First, I use a autocmd to set ant as makeprg for java files:

autocmd BufNewFile,BufRead *.java :setlocal makeprg=ant\ -s\ build.xml

It works well for java files. But as I did some Android development these days, I also have lots of xml files to edit (and then make to see the result). I can't use such a autocmd for xml's, as not all xml's are in a java project. So projtags.vim is again the answer:

let g:ProjTags += [["~/work/pmats", ":set mp=ant\\ -s\\ build.xml"]]

So I can always use ":make" for ant under my Android project now, no matter it's .java or .xml.

My vimrc segment for tags and projtags

set tags+=/usr/local/include/tags
set tags+=/usr/include/tags
set tags+=/opt/local/include/tags
set tags+=tags;

let g:ProjTags = []
let g:ProjTags += [["~/work/squid", ":set sw=4"]]
let g:ProjTags += [["~/work/pmats", ":set mp=ant\\ -s\\ build.xml"]]

Patches for projtags.vim (e.g. Windows support, I haven't tested it but I guess it won't work under Windows now) are welcomed You can get the code from the git repo for my various scripts.

You can get my patch here or my build here.

There's no GUI for setting 3rd party twitter API address. Instead, you need to change the configuration file. You can use either Property List Editor or the Terminal "defaults write" way. Here's an example:

$ defaults write com.aki.Pwitter twitter_https -bool NO
$ defaults write com.aki.Pwitter twitter_base "teewt.yhsif.com"

To restore the default settings ( https and "twitter.com" ), you can just delete the custom settings by:

$ defaults delete com.aki.Pwitter twitter_https
$ defaults delete com.aki.Pwitter twitter_base

It's useful when you can't connect to twitter.com directly, like me in China. Enjoy it.

It log GPS tracks to CSV format, WC wrote a script to convert it to the GPX format, but without waypoints. So I rewrote a Python script (as I'm not so familiar with perl), to add the waypoints to the GPX file.

Get the script here, it will use the voice record filename as the name of the waypoint if available, or otherwise just "Waypoint #N". You may want to edit the converted GPX file to rename the waypoints.

221.192.0.0 221.199.207.255

Into IP mask format (with comment) like:

#221.192.0.0 - 221.199.207.255
221.192.0.0/14
221.196.0.0/15
221.198.0.0/16
221.199.0.0/17
221.199.128.0/18
221.199.192.0/20

The script is:

 1 #!/usr/bin/env python
 2
 3 import sys
 4 import re
 5
 6 def ip2int(ip) :
 7         ret = 0
 8         match = re.match("(\d*)\.(\d*)\.(\d*)\.(\d*)", ip)
 9         if not match : return 0
10         for i in xrange(4) : ret = (ret << 8) + int(match.groups()[i])
11         return ret
12
13 def int2ip(ipnum) :
14         ip1 = ipnum >> 24
15         ip2 = ipnum >> 16 & 0xFF
16         ip3 = ipnum >> 8 & 0xFF
17         ip4 = ipnum & 0xFF
18         return "%d.%d.%d.%d" % (ip1, ip2, ip3, ip4)
19
20 def printrange(startip, endip) :
21         bits = 1
22         mask = 1
23         while bits < 32 :
24                 newip = startip | mask
25                 if (newip>endip) or (((startip>>bits) << bits) != startip) :
26                         bits = bits - 1
27                         mask = mask >> 1
28                         break
29                 bits = bits + 1
30                 mask = (mask<<1) + 1
31         newip = startip | mask
32         bits = 32 - bits
33         print "%s/%d" % (int2ip(startip), bits)
34         if newip < endip :
35                 printrange(newip + 1, endip)
36
37 while 1 :
38         line = sys.stdin.readline().strip()
39         if not line : break
40         chars = line.split(" ")
41         print "#%s - %s" % (chars[0], chars[1])
42         ip1 = ip2int(chars[0])
43         ip2 = ip2int(chars[1])
44         printrange(ip1, ip2)
45

So I wrote this patch to resolve this problem. Google tell me that if I want to encode an email subject in PHP, I should use the mb_encode_mimeheader() function. But Dreamhost just didn't provide any mb_ functions in PHP. I dug more about email subject encoding, and found that a hardcoded base64 should just do the trick. As the original sending mail code in NucleusCMS hardcoded UTF-8 in the Content-Type, I assume that hardcode UTF-8 in the subject is fine, too.

But I have to set my lap-top's ethernet to use DHCP in office, to avoid 2 network configurations on my lap-top, I need my Eee Box to act as a dhcpd, that can automatically assign an IP to my lap-top, but don't harm the router/nameserver configurations on my lap-top, and the existing DHCP in the WiFi network.

I aptituded the dhcp3-server package, and look into the default configuration file, got this code:

subnet 192.168.1.0 netmask 255.255.255.0 {
  range 192.168.1.100 192.168.1.254;
}

And it works!

As external acl run in individual process, even if the acl program have memory leak, the memory used by the squid process shouldn't be growing.

We tried many ways to figure out the problem, but all fails. Finally someone noticed that in the external_acl_type documentation, there's a parameter named "cache", with this description:

result cache size, 0 is unbounded (default)

"unbounded"! So this is the problem. For username/password routine, cache is useful. the next time some user with the same username/password comes, Squid can get the result from cache without communicate with acl program. But for our authentication method, as the headers used to calculate hash is differ from every request, cache is totally useless.

I really hope that "0" means no cache and "-1" means unbounded. But anyway, set "cache" to 1 can do the trick. Now the Squids don't have memory problems anymore, although cache replace will slow them down a bit.

According to this article on EeeUser forum, the rt2860 chipset released the source code for Linux driver! Download them from the official website, and build it. You will need kernel header package to build the driver.

After successfully build, use "modprobe rt2860sta" to install this module, and you may also add the line "rt2860sta" to your "/etc/mmodules" file to load it automatically every time (but seems that modprobe have done this, you may don't need this step).

Now here's a problem: seems that wpa_supplicant didn't support this driver. So you need to set wireless parameters by iwpriv manually. And the biggest problem is WPAPSK, you can't just input your passphrase to do it. Luckily there's a webpage that can calculate the WPAPSK for us. I'm using WPA2 and it works. I'm not sure about WPA. But WPA is broken! Why don't you move to WPA2?

Save the below script to "/etc/restart_wireless.sh" and give it execute privilege:

iface=ra0
w="iwpriv $iface"

total_start=$(date +%s)

init_start=$(date +%s)
echo -en "iwpriv config..."
$w set NetworkType=Infra
$w set AuthMode=WPA2PSK
$w set EncrypType=AES
$w set SSID=Your SSID
# get WPAPSK from http://www.wireshark.org/tools/wpa-psk.html
$w set WPAPSK=Your WPAPSK
echo "done"
init_end=$(date +%s)
init_time=$(($init_end-$init_start))

assoc_start=$(date +%s)
assoc=0
assoc_report=10
assoc_loop=310
echo -n "Associating..."
for ((i=0; $i < $assoc_loop; i++)); do
        if [ "$(iwconfig ra0 2>/dev/null | head -1 | cut -f2 -d: | cut -f1 -d" ")" == "\"\"" ]; then
                if [[ $(( ($i+1) % $assoc_report )) == 0 ]]; then
                        echo -n .
                fi
        else
                echo done
                assoc=1
                break
        fi
        sleep 1
done

if [[ $assoc != 1 ]]; then
        echo failed
        exit 1
fi
assoc_end=$(date +%s)
assoc_time=$(($assoc_end-$assoc_start))

total_end=$(date +%s)
total_time=$(($total_end-$total_start))

echo -e "Time spent (sesconds)\n\tinit: $init_time\n\tassociation: $assoc_time\n\tTotal: $total_time"

Credit to a1l0a2k9, the above script is also from EeeUser forum, but I removed the DHCP part and modprobe part. If you are using DHCP, then you may need the DHCP part and modprobe part from the original script.

Now the "/etc/network/interfaces" part, add the following lines for the ra0 interface:

iface ra0 inet static
up /etc/restart_wireless.sh
address 10.0.14.14
netmask 255.255.255.0
gateway 10.0.14.1
auto ra0

(for DHCP users: change "static" to "manual" and remove the "address", "netmask" and "gateway" lines.)

And now, "ifup ra0", then you're done!

My old home server is dying these days, so I bought a new Asus Eee Box B202 to replace it. It uses Intel Atom N270 CPU, 1G memory, 80G harddisk, 10/100/1000 Ethernet and 802.11n wireless.

The first thing I do on it is install Debian Lenny, my favourite system for server.

Preparing USB flash for net install

As it didn't come with a cd-rom, I choose USB flash. I use the SD card from my camera and a USB card reader to combine a USB flash, and it can be used to boot successfully.

I prepared the USB flash according to the Debian Lenny documentation, but meet some problems:

1. Google for hd-media returned the hd-media link for Sarge as the first result, I used the boot.img.gz from Sarge and lenny-businesscard iso but the iso can't be found by the installer, so the installer (boot.img.gz) and the iso must match.
2. The hd-media from Lenny and the Lenny beta2 iso didn't match, neither. The installer can recognize the iso, but complain about mismatched kernel version, and prompt you that it need network update. But the installer from boot.img.gz didn't come with ethernet driver, so it will fail and can't continue.

So finally I have to use "the flexible way" and net install. I use the initrd.gz that have ethernet driver, and the vmlinuz from Lenny hd-media. None of the iso is needed (and you can't use them), all packages will be downloaded from one of the Debian mirrors.

After prepared the USB flash, DON'T FORGET to lock the write protection lock before boot. It will save your life later.

Install Debian

Boot from the prepared USB flash, and it contains only GRUB CLI, so you need to boot the installer manually:

root (hd0,0)
kernel /vmlinuz
initrd /initrd.gz
boot

Now you have a Debian Installer that can drive your ethernet card, so you're ready to install.

Install steps are normal, nothing more to say until the grub-install step.

grub-install failure

On the grub-install step, it will complain that grub-install (hd0) failed. Why? cause (hd0) is your USB flash and (hd1) is your harddisk! That's why lock the write-protection lock is important, or otherwise it may succeed without write your harddisk mbr. That's really stupid. Manually install grub on (hd1) and it will continue.

And the installation completes.

The X problem

After installation, you may find that your X didn't work. This is because it uses a Intel Graphic chipset for lap-top, but it's not lap-top. Ubuntu wiki have the solution, and it works.

Wireless driver

UPDATE: follow my next blog article for official driver and 802.11n!

After installation, the first important thing to do is to drive the wireless card. It uses AzureWare card which uses a rt2860 chipset. And luckily, ndiswrapper can do it.

Follow the instructions on Debian wiki to install ndiswrapper. The Windows driver is on the CD (you have another computer to read the CD and copy the driver to the Eee Box, do you?), I used the WIN2KXP one. After install ndiswrapper, it works.

But there are some problem in the /etc/network/interfaces, if you use WPA (I didn't get thie problem when using WEP):

auto wlan0
iface wlan0 inet static
address 10.0.14.14
netmask 255.255.255.0
gateway 10.0.14.1
wpa-conf /etc/wpa_supplicant.conf

The configure above can be used to connect router (that I can see it from router admin), but the IP 10.0.14.14 can't be reached by another computer in the subnet. But if you execute an extra:

# ifconfig wlan0 10.0.14.14

Then it will be OK. I don't know why but if I move the "auto wlan0" line after the "wpa-conf" line, it works fine. Maybe it must wait wpa-supplicant to do something first?

Another problem is that it can only use 802.11g wireless, if anyone knows how to drive it to 802.11n, please tell me

The end and photos

Finally, I have a new home server now.

Today I'm going to sync my 2 commits in git to svn, using:

$ git svn dcommit

But after committed the first change, the svn server have no spaces left! So the second commit was failed:

End of file found: Can't read file '/tmp/report.tmp': End of file found at /usr/bin/git-svn line 3856

After svn admin resolved this problem, and I dcommit again, but it still fails:

Merge conflict during commit: File or directory '***.cpp' is out of date; try updating: The version resource does not correspond to the resource within the transaction. Either the requested version resource is out of date (needs to be updated), or the requested version resource is newer than the transaction root (restart the commit). at /usr/bin/git-svn line 461

And I've tried

$ git svn fetch

but the problem remains. So I decided to manually commit that change.

First, generate a patch from git ("HEAD^" means make a patch against the parent of HEAD, so it's my last change):

$ git format-patch HEAD^

And this command will generate a 0001--***.patch file. Then, use "svn co" to checkout a svn working copy, and apply that patch:

$ patch -i /path/to/0001--***.patch -p1

Then, get my last commit log from "git log", then use "svn ci" to commit it.

Now back to the git working copy, rebase svn:

$ git svn fetch
$ git svn rebase

Then try dcommit again, no conflicts and no commit made (as the svn repository is already up to date).

Hooray!

After upgrading, the following error dialog will greet me every time I start Thunderbird:

I googled this error message and found that it was caused by the incompatible extension ThunderBirthDay. OK, delete the calendar should fix it. No! The calendar was hide from Lightning's calendar list!

Fine, downgrade Lightning and then remove the calendar should work. But unfortunately, Lightning 0.9 have changed the storage format and can't downgrade to 0.8 anymore.

So how to delete that calendar? Maybe I should delete the calendar direct from Lightning's storage. According to the official FAQ, it's stored as an SQLite database file.

Under Thunderbird's profile directory, I got a "storage.sdb" file. I tried to use the command line sqlite to open it, and it works:

$ sqlite3 storage.sdb 
SQLite version 3.5.0
Enter ".help" for instructions
sqlite> .tables
cal_attachments              cal_metadata               
cal_attendees                cal_properties             
cal_calendar_schema_version  cal_recurrence             
cal_calendars                cal_relations              
cal_calendars_prefs          cal_todos                  
cal_calmgr_schema_version    cal_tz_version             
cal_events                 

The "cal_calendars" table seems to be the one I'm looking for:

sqlite> select * from cal_calendars;
1|storage|moz-profile-calendar://
2|thunderbirthday|moz-abmdbdirectory://abook.mab

Yes it is! so I need to know how to delete that record:

sqlite> .schema cal_calendars
CREATE TABLE cal_calendars (id INTEGER PRIMARY KEY, type TEXT, uri TEXT);

OK, then this should work:

sqlite> delete from cal_calendars where type = "thunderbirthday";
sqlite> select * from cal_calendars;
1|storage|moz-profile-calendar://

It DOES work!

Now my Thunderbird is quiet on starting.

But, the guys in Lightning really should give a chance to delete calendars that aren't compatible. I should file a bug for it. I've filed Bug #450121!

379 // Some compilers complain about "statement with no effect" with (a).
380 // This eliminates the warnings, and no code is generated for the null
381 // conditional statement.  @note that may only be true if -O is enabled,
382 // such as with GreenHills (ghs) 1.8.8.
383 # define ACE_UNUSED_ARG(a) do {/* null */} while (&a == 0)

But when I use this macro, gcc 4.3 will complain:

warning: the address of ‘a’ will never be NULL

A month ago Yong Sun from Sun begin to port SunPinyin to Mac, and now it's more than just works. I'm satisfied with the Mac port and use it everyday. As the Miao project is scratching my own itch, and I'm not itchy anymore, I should call it a day.

Sorry for anyone who waits for Miao, but I'm not going to reinvent the wheel.

The writing speed via 802.11n wireless is as below:

fishy@Makelele:~/Video$ dd if=Pixar\ -\ Lifted.wmv of=/Volumes/Cacapa-1/foo.wmv
91582+1 records in
91582+1 records out
46890385 bytes transferred in 15.478993 secs (3029292 bytes/sec)

That's about 3MB/s. So I think the speed on Mac is acceptable, but it should be faster.

On the Linux side, I use Debian Lenny, with samba, afpfs-ng and a 100M ethernet cable (my Linux machine didn't come with a Gigabit ethernet port). The facts are:

Writing via AFP:

root@deBoer:~# dd if=prog03.mdb of=foo/foo.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 6.89061 s, 649 kB/s

Writing via SMB:

root@deBoer:~# dd if=prog03.mdb of=/media/cacapa/bar.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 6.57683 s, 680 kB/s

Reading via AFP:

root@deBoer:~# dd if=foo/bar.mdb of=bar.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 8.87068 s, 504 kB/s

Reading via SMB:

root@deBoer:~# dd if=/media/cacapa/foo.mdb of=foo.mdb
8736+0 records in
8736+0 records out
4472832 bytes (4.5 MB) copied, 0.919922 s, 4.9 MB/s

So the conclusion is that, for writing, AFP and SMB are both slow. But for reading, SMB is much much faster than AFP, so you should use SMB to connect a Time Capsule on Linux.

The box:

Unboxing:

And then working!

And here's a note, the command to limit my 1TB Time Capsule use only 500G for backup (so that I can use the other 500G for other stuffs), according to the comment of this hint:

hdiutil resize -size 500g -shrinkonly /Volumes/path/to/sparsebundle/

But the speed of backup is really really slow, that disappointed me. What a shame! It would be perfect if the backup speed up.

I've uploaded a test video:

And here's the same video on youtube, to be compared:

So here's the comparison:

• Speed: youtube is much much faster than flickr, anyway, youtube consumed half of California's bandwidth, so it's no surprise.
• Who can upload: everybody, every free users can upload to youtube, but only pro users ($24.95/y) can upload videos to flickr now.
• Geotag: you can geotag your video on flickr, but seems that you can't do it on youtube.
• Original file: you can download the original file of your video in flickr's embed page, so in case you lost your original file, it's a way to get it back.
• Sizes: there's only 1 size on youtube, but many sizes on flickr to embed.
• Group: on flickr, you can group your videos along with your photos, so the photos/videos can be grouped together.
• Privacy control: you can filter viewers with your flickr contacts, as I already have a community on flickr, but not on youtube.

So, from now on, I may move from youtube to flickr, flickr rocks!

But I haven't got any "real" trackbacks, so they are all spams, and more accurate, crazy spams, that simply too much to overload my DB.

So I have to wrote this patch to add an option to NP_Trackback: drop blocked trackbacks directly. After enabled, trackbacks that blocked by NP_Trackback (failed spam test or no link) will not be stored into the DB.

I use "apt-cache search afp" to find out that there's a package named "netatalk" can provide AFP file sharing, so install it. But I can only login use guest account, not my system user, from Leopard.

I googled it and found that the problem is: on the Debian side, as a license issue, the Debian package didn't come with SSL support; on the Leopard side, it didn't allow you exchange your password with AFP server without SSL. So the solution is build netatalk yourself, with SSL.

The building steps are described on this blog, and I also disabled atalkd as the author suggested, it caused netatalk to start-up much faster than before.

So finally I got a AFP server for my Mac (compare this icon to the famous BSOD icon for SMB servers in Leopard ):

And as expected, AFP is much much faster than SMB, here's the write test:

For AFP:

fishy@McManaman:~$ dd if=/dev/zero of=/Volumes/Home\ Directory/foo
^C57345+0 records in
57345+0 records out
29360640 bytes (29 MB) copied, 11.0833 s, 2.6 MB/s

And for SMB:

fishy@McManaman:~$ dd if=/dev/zero of=/Volumes/fishy/bar
^C4235+0 records in
4235+0 records out
2168320 bytes (2.2 MB) copied, 10.6889 s, 203 kB/s

I'm impressed!

Finally I've got my Gmail account with IMAP access. I use Thunderbird as the mail client, and find that the "date" of many mails are the time I retrieved them via IMAP, which is incorrect.

These mails including gtalk chat logs and Gmail invitation accepted responses, that means most of the mails are sent/generated by Gmail.

An incorrect date mail have a Date line in the headers like this:

Date: Fri, 1 Dec 2006 05:35:25 -0800 (PST)

And a normal mail have a Date line in the headers like this:

Date: Thu, 21 Dec 2006 19:16:34 +0800

Seems that the problem is, Thunderbird didn't recognize a Date line with a 3-letter timezone description(the "PST" in this example), so it use the date retrieved instead.

As most of mail clients (even Gmail itself) didn't add this timezone description in the Date line while sending a mail, I guess it's Gmail's fault to add it into the chat logs and invitation responses to make it unstandard? Anyone can tell me what's the definition in the RFC?

Although NucleusCMS itself have good support for multi-blog, as it's mainly used as a single blog system like WordPress, nearly none of the plugins considered the multi-blog situation.

So I made patches for the plugins I used, to make them multi-blog ready.

These patches generally move some option from global to blog, so that every blog can have its own setting. And for some plugins that have an admin area, I also filtered the things it can admin to avoid security risk.

• patch for NP_TrackBack 2.1.0:
  1. move notify options into blog settings (Notify, NotifyEmail and NoNotifyBlocked)
  2. show admin area when user is blog admin (copied from NP_Referer 1.0 code )
  3. in admin area, user can only see/manage the trackbacks in a blog he have admin privilege
• patch for NP_Referrer 1.0.1:
  1. roll back to 1.0 to show admin area for non-admin users
  2. in admin area, user can only see/modify the referrers in a blog he have admin privilege
• patch for NP_Calendar: move the locale option into blog option
• patch for NP_RSSItem: move lang, RSSLink and RSSImage option into blog option.
• patch for NP_WeatherReport: move city_name, city_id and unit option into blog option.

Also it's promised to have a syncing feature, it didn't have now. But I need the ability to sync my schedule with my Palm Treo 680. I can sync my Palm with iCal, so getting the linkage between Lightning and iCal is a good way to do this.

First, I installed the Provider for Google Calendar extension to get the bidirectional access to Google Calendar to Lightning.

Then, I use GCALDaemon to sync my iCal with Google Calendar. It can also sync Google Calendar with Lightning, but Provider for Google Calendar is much better at this.

Now if I add/modify/remove an event in Lightning, Provider for Google Calendar will update my Google Calendar immediately. Later, GCALDaemon will find the update, and update iCal. Now I sync my Palm with my computer, it will get the update.

If I update an event in my Palm, and sync it to iCal, GCALDaemon will update Google Calendar later, and then Lightning will update it later.

So it works.

Make a symbolic link with the ics file of Lightning and iCal may also work, but I can't find the ics file for Lightning

You can subscribe my busy/available status on Google Calendar now

This should also works for Mozilla Sunbird.

And this command can be used as the reversed diff:

cat file1 file2 | sort | uniq -d

In Debian, it's:

43 extern size_t iconv (iconv_t __cd, char **__restrict __inbuf,
44                      size_t *__restrict __inbytesleft,
45                      char **__restrict __outbuf,
46                      size_t *__restrict __outbytesleft);

And in other systems, such as Mac OS X, it's:

83 extern size_t iconv (iconv_t cd, const char* * inbuf, size_t *inbytesleft, char* * outbuf, size_t *outbytesleft);

So you can see that the second parameter, "inbuf", can be "const char **" or "char **"

It will make a big problem while writing code for both systems. One resolution is to use condition compile ("#ifdef" blah blah...), but RoachCock@newsmth give me another (and much better) resolution: use operator overload (so it's C++ only):

133 struct iconv_param_adapter {
134         iconv_param_adapter(const char**p) : p(p) {}
135         iconv_param_adapter(char**p) : p((const char**)p) {}
136         operator char**() const
137         {
138                 return (char**)p;
139         }
140         operator const char**() const
141         {
142                 return (const char**)p;
143         }
144         const char** p;
145 };

When you calling "iconv()", call it like this:

111         size_t res = iconv(data, iconv_param_adapter(&s), &inbytesleft, &outnew, &outbytesleft);

Thank you RoachCock!

I use libiconv to convert the GBK string to Unicode for expat.

First, implement a function to pass to XML_SetUnknownEncodingHandler:

int
XML_Stream_Parser::xml_unknown_encoding(void* data, const char* name, XML_Encoding* info) {
        iconv_t cd;
        if(strncasecmp(name, "GB", 2) != 0 || (cd = iconv_open("UCS-2BE", name)) == (iconv_t)-1) {     // not GB, unsupported
                fprintf(stderr, "can't convert %s\n", name);
                return 0;
        }
        for(size_t i=0; i<128; i++) info->map[i] = i;
        for(size_t i=128; i<256; i++) info->map[i] = -2;
        info->convert = XML_Stream_Parser::xml_convert_gb;
        info->release = XML_Stream_Parser::xml_convert_release;
        info->data = cd;
        return 1;
}

In this function, I tell expat that for GBK encoding, ASCII 0~127 is left as is, and ASCII 128~255 will need to be dealt together with the next byte.

Then implement the "convert" and "release" functions:

int
XML_Stream_Parser::xml_convert_gb(void* data, const char* s) {
        const size_t out_initial = 4;
        size_t inbytesleft = 2, outbytesleft = out_initial;
        char *out = new char[out_initial], *outnew = out;
        size_t res = iconv(data, &s, &inbytesleft, &outnew, &outbytesleft);
        int ret = 0;
        if(res == (size_t)-1) {
                fprintf(stderr, "error in conversion\n");
                delete []out;
                return '?';
        }
        for(size_t i = 0; i < out_initial - outbytesleft; i++)
                ret = (ret<<8) + (unsigned char)out[i];
        delete []out;
        return ret;
}

void
XML_Stream_Parser::xml_convert_release(void* data) {
        iconv_close(data);
}

In "convert", I use iconv to convert the string to unicode, and return the unicode to expat.

The limitation in this interface is that it can't deal with 4-byte GB18030 codes, as I can't judge whether it's a 4-byte code just by the first code.

Anyway, I suggest that all XML should be encoded to UTF-8, so that this is unneeded

It's purpose is to recover some image source URL that was f'ed by some firewall, with the URL with web proxy.

I used the web proxy code from gladder, credit to gneheix.

For Camino users that can't use GreaseMonkey directly, you can use GeekMonkey.

Enjoy it and f' back.

And there's something should be kept in mind: don't use "`"s.

As I used to use MySQL before, I always put "`" in SQL statements, for example:

create table `foo` (`bar` int)

It's OK in SQLite command line program, but not in the C library.

When I execute a SQL query with "`", I'll get this error message:

sql error #1: unrecognized token: "`"

So I deleted "`"s in the SQL statement, but still get this error message:

sql error #4: malformed database schema - unrecognized token: "`"

This really drive me crazy, as I don't know what's wrong. But finally I've got the reason: that's because there's "`" in the "create table" statement. So I recreated the table and rerun the program, it's finally OK now.

 1 #include <stdio.h>
 2 #include <stdlib.h>
 3 #include <sqlite3.h>
 4
 5 #define NAME "miao.db"
 6
 7 int main() {
 8         sqlite3 *db;
 9         sqlite3_stmt *stmt;
10         int rc;
11         char *errmsg = 0;
12
13         rc = sqlite3_open(NAME, &db);
14         if(rc) {
15                 fprintf(stderr, "can't open db %s: %s\n", NAME, sqlite3_errmsg(db));
16                 sqlite3_close(db);
17                 exit(1);
18         }
19         rc = sqlite3_prepare(db, "select * from words", 0, &stmt, 0);
20         if(rc!=SQLITE_OK) {
21                 fprintf(stderr, "sql error #%d: %s\n", rc, sqlite3_errmsg(db));
22         } else while((rc = sqlite3_step(stmt)) != SQLITE_DONE) {
23                 switch(rc) {
24                 case SQLITE_BUSY:
25                         fprintf(stderr, "busy, wait 1 seconds\n");
26                         sleep(1);
27                         break;
28                 case SQLITE_ERROR:
29                         fprintf(stderr, "step error: %s\n", sqlite3_errmsg(db));
30                         break;
31                 case SQLITE_ROW:
32                         {
33                                 int n = sqlite3_column_count(stmt);
34                                 int i;
35                                 for(i=0; i<n; i++) {
36                                         printf("%s = ", sqlite3_column_name(stmt, i));
37                                         switch(sqlite3_column_type(stmt, i)) {
38                                         case SQLITE_TEXT:
39                                                 printf("%s", sqlite3_column_text(stmt, i));
40                                                 break;
41                                         case SQLITE_INTEGER:
42                                                 printf("%d", sqlite3_column_int(stmt, i));
43                                                 break;
44                                         case SQLITE_FLOAT:
45                                                 printf("%f", sqlite3_column_double(stmt, i));
46                                                 break;
47                                         case SQLITE_BLOB:
48                                                 printf("(blob)");
49                                                 break;
50                                         case SQLITE_NULL:
51                                                 printf("(null)");
52                                                 break;
53                                         default:
54                                                 printf("(unknown: %d)", sqlite3_column_type(stmt, i));
55                                         }
56                                         printf("\n");
57                                 }
58                         }
59                         break;
60                 }
61         }
62         sqlite3_finalize(stmt);
63         sqlite3_close(db);
64 }
65

tags: sqlite, c, program]]> dev http://wang.yuxuan.org/blog/2007/5/22/simple_sqlite_test_c_program_without_callbacks Tue, 22 May 2007 00:38:33 +0800 fishy <nospam@yuxuan.org> http://wang.yuxuan.org/blog/2007/5/19/add_emacs_mode_line_support_to_vim_and_call_for_help http://wang.yuxuan.org/blog/2007/5/19/add_emacs_mode_line_support_to_vim_and_call_for_help Different editor configurations, especially tabstops are always pain for co-work. So if everyone uses vim, you can specify some vim instruction to override one's vim configuration in your file, just like:

// vim:tabstop=4:

And if everyone uses emacs, you can also specify emacs mode line like:

// -*- tab-width: 2 -*-

But what about make vim to read emacs mode line? I've wrote a script to do so.

As I didn't use emacs at all, I don't know which instructions can be specified in the emacs mode line. I just wrote a handler for "tab-width" as an example. If you are familiar with both emacs and vim, please help me to add more handlers into this script.

After more handlers added, I'll submit this script to vim.org.

The answer is simple: the current pinyin input methods on Mac doesn't satisfy me. I'm going to scratch my own itch

ITABC (comes with Mac) and Pinyin Module in OpenVanilla isn't a smart input method; QIM is smart but it's a shareware, and a expensive shareware (USD 19.99 or RMB 69.00, but only for one major version free update); FIT is smarter than ITABC and free (for charge), but is not smart enough and I don't like some of its feature but can't turn them off.

Miao will be a smart pinyin input method, designed for people who likes to input a whole sentence once. It will learn user's accustoming and finally become a essential tool in inputing (after some training).

The idea and feature

I won't focus on pre-train dictionaries, just like Sogou Pinyin or Google Pinyin. My focus will be on "learning"

For each sentence you input, it will record the words within the sentence, including continuos words and non-continuos words, and calculated for probability.

Continuos word probability will be used to auto-learn new words, that is, if there's a high probability that two known words will comes together, they will be formed as a new word.

Non-continuos word probability will be used for candidate word sorting. The word with the highest probability will get ordered first. As a instance, for the pinyin word "yaoming", if the sentence comes with "huojian"(火箭/Rockets), "maidi"(麦迪/McGrady), it's more likely to be "姚明"(Yao Ming the NBA player); If the sentence comes with "can"(惨/misery), it's more likely to be "要命"(Kill me!).

The license

It may be licensed under GPL or BSD license, it's not decided yet. But I can assume that it will be certainly free, as freedom!

About the name

"Miao" is the pinyin of the Chinese character "喵", which just like "meow" in English, is "the characteristic crying sound of a cat" (from Oxford American Dictionaries).

It's a pinyin input method, so I choose a pinyin word as its name

Other informations

I should write it as a module for OpenVanilla, instead of stand-alone. The reason is that I'm not going to reinvent the wheel. I want to focus on the learning algorithm, but not the system interfaces. And this may make it easier to port into other platforms, for example Linux.

The project homepage will be http://oaim.yhsif.com. I'll start it soon, any contribution, e.g. codes, ideas or artworks, are appreciated.

By default, vim will only use the "tags" file under your current working directory. You can use this command to see it:

:set tags?

Generally, your "tags" file under you current working directory won't contain informations about system libraries (glibc, stl, etc.). If you miss them, you can generate tags file for your system libraries, and ask your vim to load them:

set tags+=/usr/local/include/tags
set tags+=/usr/include/tags

If you are working on a big project, which have many subdirectories, the "tags" file under each working directory may be not enough, as you also need some information about the functions under other subdirectories. So you can generate a "tags" file under your project root, and ask vim to load it when editing a file within your project:

autocmd BufEnter ~/work/myproj/* :setlocal tags+=~/work/myproj/tags

Thanks for Ryan Phillips, you can also add "tags;" (notice the semicolon) to your tags so that vim will automatically look up "tags" file in the file tree (":help file-searching" for document):

set tags+=tags;

To make it easier to use, I've also made a script named "projtags.vim" so that you need only set your project directories in your vimrc:

let g:ProjTags = ["~/work/proj1"]
let g:ProjTags+= [["~/work/proj2", "~/work/proj2.tags"]]

Happy vimming

When I use Camino to visit live spaces pages, I always get a "XML Parsing Error", even the page I'm visiting is not the RSS but a html. When I use Firefox or Safari, I always get a HTTP 500 error.

The "XML Parsing Error" is really weird. It seems that the live spaces server send a wrong "Content-Type" so that Camino treat it as a XML document and try to parse it use a XML tree, and that failed (with "view source" I can see the html source corretly).

But Camino didn't have a extension like live HTTP headers to see what Content-Type do the server send, thanks to smorgan on MozillaZine Forums, I can see it using curl.

This is what curl get:

McManaman:~ fishy$ curl -v http://spaces.live.com -o /dev/null 2>&1 | grep Content-Type
< Content-Type: application/xhtml+xml; charset=utf-8

This is what Camino will get:

McManaman:~ fishy$ curl -v http://spaces.live.com -o /dev/null -A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en; rv:1.8.1.4pre) Gecko/20070408 Camino/1.1b+" 2>&1 | grep Content-Type
< Content-Type: application/xhtml+xml; charset=utf-8

This is what Firefox will get:

McManaman:~ fishy$ curl -v http://spaces.live.com -o /dev/null -A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0" 2>&1 | grep Content-Type
< Content-Type: text/html; charset=utf-8

And this is what Safari will get:

McManaman:~ fishy$ curl -v http://spaces.live.com -o /dev/null -A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/418.9.1 (KHTML, like Gecko) Safari/419.3" 2>&1 | grep Content-Type
< Content-Type: text/html; charset=utf-8

As you can see, for some common browsers, such as Firefox or Safari, live spaces server send the right Content-Type. For unknown browsers (including curl, Camino, etc.), it claims that it's "application/xhtml+xml". Oh my god, what did they think they are doing? What can they get from this Content-Type? I can't understand this at all.

So the only word I can say is, "live spaces suck, one more time"

1. Insert your disk into your driver, of course.
2. Unmount it but didn't eject it. To do so, you need to type this command in your Terminal: "sudo umount /dev/disk1". "disk1" may vary depends on your other drivers.
3. Use dd to create the image: "dd if=/dev/disk1 of=/path/to/foo.iso". Again, "disk1" may vary.
4. When it's done, you've got the ISO image file, and you may eject the disk now. But as a bug in Finder, you can't eject a volume which you unmounted in Terminal in normal ways. Instead, you need to type this command in Terminal to eject it: "sudo diskutil eject /dev/disk1"

The Finder bug is really annoying, even a logout can't eject the disk, you must reboot your machine. But fortunately I've found the workaround on macosxhints.

This bash script shows how to uses ImageMagick to batch resize your photos:

1 #!/bin/sh
2
3 for file in *.JPG; do
4         convert -resize 1024x768 $file ${file%.JPG}_resize.jpg
5 done

wget -Y -e "http_proxy=host:port" url

"How to set proxy for wget?" I've been asked this question for many times, but it seems didn't appears in the "-h" output nor man page, so I always forgot it.

That's why I'm making a note here

UPDATED: Troels Bay have use another way to do it, please wait for his official build

Google Reader Notifier by Troels Bay is a great open source application that provide a desktop notifier for Google Reader, the online feed reader, and also a feed subscriber. As I'm using Camino as my browser, which didn't have a build-in RSS support, I need it even more to subscribe feeds to Google Reader with one click. (not just "notifier" )

But when subscribe with preview, if there's a "?" in the URL, the problem is that Google Reader will ignore the part after "?". So that for example my photos on Flickr, "http://api.flickr.com/services/feeds/photos_public.gne?id=76236359@N00&format=rss_200" , will be recognised by Google as "http://api.flickr.com/services/feeds/photos_public.gne".

So I wrote this patch to escape URL before submit to Google.

 1 --- Google Reader copy/GRController.m   2007-03-11 18:45:56.000000000 +0800
 2 +++ Google Reader/GRController.m        2007-03-11 18:49:46.000000000 +0800
 3 @@ -1610,7 +1610,15 @@
 4  
 5         if ([[prefs valueForKey:@"dontVerifySubscription"] boolValue] != YES) {
 6  
 7 -               [[NSWorkspace sharedWorkspace] openURL:[NSURL URLWithString:[[NSString stringWithFormat:@"%@://www.google.com/reader/preview/*/feed/", [self getURLPrefix]] stringByAppendingString:url]]];                    
 8 +               NSMutableString * escapedUrl = [NSMutableString stringWithCapacity: ([url length]*3)];   // for the worst case the length will growth to 3 times
 9 +               [escapedUrl setString:url];
10 +               [escapedUrl replaceOccurrencesOfString:@"?" withString:@"%3F" options:0 range:NSMakeRange(0, [escapedUrl length])];
11 +               [escapedUrl replaceOccurrencesOfString:@"&" withString:@"%26" options:0 range:NSMakeRange(0, [escapedUrl length])];
12 +               [escapedUrl replaceOccurrencesOfString:@":" withString:@"%3A" options:0 range:NSMakeRange(0, [escapedUrl length])];
13 +               [escapedUrl replaceOccurrencesOfString:@"/" withString:@"%2F" options:0 range:NSMakeRange(0, [escapedUrl length])];
14 +               [escapedUrl replaceOccurrencesOfString:@"=" withString:@"%3D" options:0 range:NSMakeRange(0, [escapedUrl length])];
15 +
16 +               [[NSWorkspace sharedWorkspace] openURL:[NSURL URLWithString:[[NSString stringWithFormat:@"%@://www.google.com/reader/preview/*/feed/", [self getURLPrefix]] stringByAppendingString:escapedUrl]]];                      
17                         
18         } else {
19  

But if I set TERM to "vt100" before launch screen, it's OK, so this alias can resolve the problem:

alias screen='env TERM=vt100 screen'

For the poor network, I always need a multi-thread http downloader. Firefox has a extension DownThemAll!, but after I finish surfing websites (but didn't finish downloading yet), I used to press Command+Q to quit Firefox, and the download was interrupted. lftp has a built-in downloader: pget, so use pget is a good idea:

alias pget="lftp -c pget"

Put the codes into your bashrc file, and restart your terminal, it's done.

As from MySQL 4.1, if the database default charset is UTF8 but the default charset setting in my.cnf is not UTF8, you must do a "SET NAMES UTF8" query after connection, to make sure that in this mysql connection, all query will be handled as UTF8. otherwise seems that it will use latin1 as default.

If the "SET NAMES" isn't queried, the articles and comments won't be malformed, but the data that stored in MySQL database isn't actually in UTF8 encoding, so that for example mysqldump, or phpmyadmin, won't get the right data.

So what we need to do is check database's charset setting, and send a "SET NAME" query to ensure we're using the database's charset setting.

I've found that LifeType did this thing, so I copied its detecting code and made patches for NucleusCMS and Wikka Wiki (and also my custom homepage for LifeType, of course!).

But before downloading the patches, I should tell you that:

First, you MUST convert your database. Cause before you patch NucleusCMS or Wikka Wiki, your articles, comments, etc. that already posted, was stored in the old way in the db.

To do that, follow the following steps (this is in case the my.cnf indicate "latin1" but your database was in "utf8", if you're using other charsets, modify them):

1. mysqldump -h host -u username -ppassword database --default-character-set=latin1 > dump.sql
2. Edit dump.sql, on Line 10, replace "latin1" with "utf8"
3. mysql -h host -u username -ppassword database < dump.sql

Second, you must ensure that in your database, ALL charset settings are set to "utf8". The word "ALL" means databases, tables and fields.

Finally, here're the patches: for nucleus, for wikka, released under GPL. WARNING: Use it on your own risk, and make backup before patching.

For aqua buttons, that shown in Camino and some optimized 3rd-party builds of Firefox, the CJK buttons often be malformed and cause you can't see the text on the button well. For common buttons, they're just ugly.

I've found a hack through the internet, and I'm not so sure what did it actually do (seems to be some dirty tricks), but it works. So I cut off some unnecessary codes and made this patch. To use it, just execute the following command under terminal (Terminal.app or iTerm.app, replace "/Applications/Camino.app" to the path of the app you want to patch):

cd /Applications/Camino.app/Contents/MacOS/res/
patch -i /path/to/moz-forms.patch

And it's done.

Before:

After:

I didn't write this patch. I just found it and made it easier to use.

Compare to Firefox, it comes with some good system integrations, including Address Book integration, Keychain Access integration, etc. And compare to Safari, it's much closer to Firefox, it also have a "about:config" thing (though a little different from Firefox), and a powerful ad-blocking engine.

So I'm going to give it a week's trial, to see if I can be used to it.

Although it's close to Firefox, it didn't have extensions. So there's something I missing in Firefox:

1. It can't force links that open a new window to open in new tab. I can use about:config to set "browser.link.open_newwindow" to "1" to disable links that opens a new window, but can't use a new tab instead. Anyway, this is acceptable. At least they won't open lots of new windows to annoy me.
2. I can't type a site name (e.g. "google" ) and press cmd+enter to automatically surround it with "www." and ".com", this will open it in new tab and use "I'm feeling lucky" search. So I must type more words to input a address.
3. Keyboard short-cut is different from Firefox, it mixed some Firefox short-cuts and Safari short-cuts up. I don't like this. I prefer a configurable keyboard short-cuts. Maybe I can configure it in System Preferences? UPDATED: Oh yes, I can configure them in System Preferences. This makes me feel much better.
4. I can't specify the text encoding of the keyword in the search tool-box. Some search engines in China only accept GBK keywords but Camino can only encode keyword in UTF-8.
5. Lack of spell checkers. It can't use the system spell check for standard text editors (cause it didn't use the standard text editor?), nor the spell checker provided by Google Toolbar for Firefox or Firefox 2.

Anyway, a impressive thing is that the build-in ad-blocking engine is powerful, and the default black-list is very useful that it filtered out almost all the ad from the pages I visited. And if the black-list isn't powerful enough, I can edit it myself.

1. In the chat style tweak window, the chat example is from George Orwell's 1984. Maybe because of Mac begins from 1984?
   
2. In the newest beta (currently) version, the history events was brought on and off with a cool effect, similar to the water wave effect of dashboard.
   

But things are sometimes not so easy. If you've chosen to make symbolic links to your system path (/usr/bin) during the installation, like I had, then you have to unlink all the links TeXLive created.

It's hard to do that manually, so I wrote the following script. You just need to run it under your /usr/bin directory and it will unlink all links that pointed to /usr/local/texlive.

 1 #!/bin/sh
 2
 3 tlprefix="^/usr/local/texlive/"
 4
 5 for file in *; do
 6         if [ -L ${file} ]; then
 7                 link=`readlink ${file}`
 8                 tllink=`echo "${link}" | grep "${tlprefix}"`
 9                 if [ -n "${tllink}" ]; then
10                         echo "going to unlink \"${file}\" that links to \"${link}\"..."
11                         unlink ${file}
12                 fi
13         fi
14 done

It unlinked 240 symbolic links from my /usr/bin directory, horrible! I'll never choose the "make symbolic links to your system path" option again during the installation of TeXLive.

I choosed a Chinese dynamic DNS provider "Peanut Shell", they've provided a Linux client to commit your IP.

But the Linux client they provided is close-source and often crashes. After it crashed, my domain name can't be kept up-to-date.

So I have to write a script to make sure it's still running:

1 #!/bin/sh
2 pid=`pidof phlinux`
3 if [ -z "$pid" ]; then
4         echo "phlinux not running, start it as daemon now..." >&2
5         /usr/local/phlinux/phlinux -d
6 else
7         echo "phlinux is running"
8 fi

Save this script in some place, and make a link to your "/etc/cron.hourly", so it will be checked every hour.

UPDATED: or maybe you don't want to get mail report on every check, I've updated the script, and you should write a script in your "/etc/cron.hourly" instead of link the script, and writes "/path/to/your/script > /dev/null"

It's amazing because, it can track all your changes, just like a wiki (or CVS/svn)! It can show the old version, and show the diff's.

For instance, this is the diff between revision 1.0 and revision 1.1 of my last post.

But seems for the posts post before installation of this plugin, the time was wrong. I'll try to fix it later.

btw: I've changed the url here from Fancy URL to Fancier URL. So if you're using bloglines or google reader or some RSS reader that track changes, you'll get notify for old items. My apologize for that. Old stuff fancy URLs still works.

NP_TrackBack was good, especially on its anti-spam ability. It blocks nearly all my received trackback pings automatically (I haven't got any real trackback yet, all I got were spams).

NP_TrackBack can also send a notification to me when a trackback ping received. But the problem is, that it also send me notification about blocked trackback pings.

I have faith on its anti-spam ability, so I don't wanna get notifications about blocked trackbacks. But I do hope get a notification on real trackbacks. So finally, I've made this hack.

I've added a option to NP_TrackBack not to send notification on blocked pings (see the screenshot below), so that my mailbox won't be spammed instead of my blog. And its default value is "no", so that it won't change your default settings.

Here's the patch file and the plugin php file itself (don't forget to replace the ".txt" extension with ".php" ). Cause I've added a option, you can't simply replace your old file if you are currently using NP_TrackBack plugin. You may need to uninstall your old one and install your new one to make it work.

After that, you'll be able to use "ie6" to start-up ie 6 via Darwine. Remember to open X11.app first.

As you can see, the font is really, really ugly, and Chinese characters can't be displayed properly.

But finally I've found the way:

1. Open System Preferences
2. Open International
3. Click the Format tab
4. Click the Customize button beside "Times"
5. Hover your mouse over the hour indicator, and it will show a arrow. Click on the arrow you'll get a drop-down list to choose hour formats.

Here's my snapshot:

So I get my 24-hour format time back, in my iCal, Mail, anywhere.

 1 #include <stdio.h>
 2 #include <string>
 3 #include <vector>
 4
 5 class base_foo {
 6 public:
 7         virtual std::string name() { return "base_foo"; }
 8 };
 9
10 class foo: public base_foo {
11 public:
12         std::string name() { return "foo"; }
13 };
14
15 class bar: public base_foo {
16 public:
17         std::string name() { return "bar"; }
18 };
19
20 int main() {
21         std::vector<base_foo*> foo_list;
22         foo f;
23         bar b;
24         foo_list.push_back(&f);
25         foo_list.push_back(&b);
26         printf("%s\n%s\n",
27                         foo_list[0]->name().c_str(),
28                         foo_list[1]->name().c_str());
29         return 0;
30 }

The result of this program is:

foo
bar

I've made a static build of the latest release (0.4.0) on my MacBook. It's a static build, that means you don't need Qt installed, you need just this app to run stand alone. And it's not a Universal Binary, it can only run on Intel Macs

QTerm is designed to be primarily ran on Linux platforms, so that the keyboard shortcuts are in Linux-style. I've also made a modified version to change some of the keyboard shortcuts into Mac-style. The changes include:

1. alt+0~9 -> cmd+0~9
2. alt+up, alt+down, alt+left, alt+right -> cmd+up, cmd+down, cmd+left, cmd+right
3. copy: cmd+c, paste: cmd+v

Here are my builds: original version, modified version. They are bzipped tarballs of ".app", extract them and move qterm.app to your Applications folder.

This domain is in Chinese pinyin, which means:

Mom says that, whatever how long your domain name is, Google can also search it out

But, can google really search it out?

For my new MacBook, the name's McManaman, he's one of my favorite players

Based on this rule, I named my old lap-top with Debian to deBoer.

And finally, I decided to name my router Routledge, this name is perfect for a router, isn't it?