Hiro Nakamura is a level-3 programmer, from the television show “Heros” that one day discovers he can manipulate space and time. Hiro, however is not the best thing to come out of the Tokyo software industry.

Recently I’ve been spending the last couple of months trying to learn how to scale our databases for the massive flux of data we expect once our site goes live. To this end we’ve tried all sorts of different opporunities. They have all left us with a rotten feeling in our stomach that most of our past year’s work is indeed worthless if we can’t scale our site accordingly.

Enter TokyoCabinet. TokyoCabinet is an extremely powerful key-value hash-based database management system straight out of Japan where top engineers collaborate to build mixi, the famous Japanese social networking site ala Facebook.

How do I install this?
Installation is extremely easy. Let’s run through it real quick shall we?

First off let’s install bzip2 and their headers — you probably have these already but I’m
assuming you are starting off fresh somewhere.

1

sudo apt-get install libbz2-dev

Now let’s grab your files from sourceforge:

1
2

wget http://tokyocabinet.sourceforge.net/tokyocabinet-1.4.7.tar.gz
wget http://tokyocabinet.sourceforge.net/rubypkg/tokyocabinet-ruby-1.20.tar.gz

unpackage them:

1
2

tar xzf tokyocabinet-1.4.7.tar.gz
tar xzf tokyocabinet-ruby-1.20.tar.gz

install tokyocabinet:

1
2

cd tokyocabinet-1*
./configure && make && make install

install the tokyocabinet ruby gem

1
2
3
4
5

cd tokyocabinet-r*
ruby extconf.rb
make
sudo make install
sudo ldconfig

test to make sure everything went ok

1
2
3

/home/user$ irb
irb(main):001:0> require 'tokyocabinet'
=> true

Now, let’s bench your system!
Grab this file (http://gist.github.com/64391):

git clone git://gist.github.com/64391.git

NOTE: do NOT install the ffi version if you want speed. I installed the ffi version yesterday and only achieved around 5555 req/second. Today I installed the real version and achieved 28,000 inserts/second and 48,000 selects/second. This was for the table database — which if you are trying to emulate your mysql RDBMS, you’ll want to choose this.