Microblogging is something that is becoming popular in Japan (though I wouldn't say "big"), and at the same time an interesting application that involves many technical challenges. A micro blogging infrastructure must be able to scale both write and read requests and like any problems, there are various ways of solving these problems, some better than others.
Thanks to Kazuho Oku, who hosted a micro blogging conference at Cybozu Labs last night, web developers gained information on how real-world applications are solving these challenging problems. Speakers included folks from Wassr (one of the largest micro blogging service in Japan) and Mixi (the largest social networking service in Japan), who had recently began experimenting with micro blogging called "mixi Echo". If you're interested in the architecture of mixi Echo, you can take a look here (the actual number of servers are much more).
So what was I doing at the conference? I was there to speak about middleware development, namely the concept of modularizing the backend of memcached and how this feature could help reduce people from forking memcached. I also described how the plugin mechanism is achieved in the prototype developed by Trond Norbye from SUN and myself.
I would love to cover the details in this entry but unfortunately time is against me at the moment so I will save it until my next entry. For now I just wanted to get the word out that we're very serious about micro blogging infrastructure in Japan ;)
The last few days I've been working on updating the binary protocol test in the latest memcached development branch to comply with the latest binary protocol specification. Prior to this update, the test client was sending an invalid request to the server, which as a consequence made the test hang and never finish.
In brief, this is the big difference:
Previously, CAS (compare and swap) value was treated as part of the extra header that is appended/serialized behind the request/response header. In the latest specification, CAS value is a required 8 byte field in the 24 byte request/response header (header size in the previous version was 16 bytes). Other than that, the rest were minor differences in the packet format of extra fields in certain commands. Easy work :)
Here is the actual diff:
As you can see, the test suite currently used by memcached is Perl based. This was fortunate for me since Perl is the second language to C that I like. I also made the code style to be more "perl-like" by fixing the indents. Although heh, I can see a Perl programmer arguing that the use of if/else blocks in the test is not best practice :P
You know, fixing the test was pretty meaningful to me since it had forced me to study the binary protocol specification, which I knew almost nothing about at the hackathon in Santa Clara, CA back in april. Hopefully I can make productive suggestions at the upcoming hackathon in Menlo Park, CA in october.
So admittedly, Mac OS X is currently not the most friendly platform to work with Drizzle, mostly due to library issues.
OS X has several weird hacks in it due to licensing issues (libreadline comes into mind first). Sure, MacPorts, Darwin Ports and etc could get around this problem but should this be necessary? Personally I dislike resorting to these solutions. Fortunately I've been doing all my Drizzle work with Ubuntu on a dedicated server so I've yet to come across any build related issues. However, it kind of sucks not to be able to take my Mac out to a cafe in the weekend and work there without connectivity.
So to make my life happier, I installed Ubuntu on my MacBook Pro (alongside OS X of course).
I came across few problems like corrupted partition table in the process of getting Ubuntu working but the following Ubuntu threads helped greatly:
General Instructions
Boot related problems when using Hardy Heron (Ubuntu 8.04)
You know, getting Ubuntu running on my Mac was entertaining since I was talking to Monty Taylor about his thoughts on how using a Mac is selling out yesterday. Now what does this make me now?
Happy Hacking :)
Toru
So I've been fortunate enough to participate in developing Drizzle, which is a microkernel fork of MySQL that you can read more about on Brian Aker's blog post.
In brief we are getting rid of components that we find unnecessary in MySQL by default, and instead making them optional by refactoring the server to be modular, aka microkernel. Another words, we are trying to develop a lean, fast, simple and extensible RDBMS that would fit well in mid and large scale web applications.
How? well, take Query Cache for example. QC works well in a one-man database but it has very small (if not no) effect when we start thinking big, and especially in the web industry. So why bother keeping it? what would be better is if we could _optionally_ make Drizzle use a cluster of memcached for query caching, which would also allow many database instances to share a common cache. Same things can be said about many other components, such as ACL and Stored Procedures. This is exactly why we are moving to a microkernel architecture. If you want something special, you should be able to customize the server in a relatively easy fashion and satisfy your requirements, rather than having to refactor the server code yourself.
Indeed, not everyone needs a microkernel database, in fact I assume most people won't. However, there are enough web developers and companies in the small portion of the pie that would love a microkernel database to solve the problems that they are facing today. This is exactly why we don't consider Drizzle to be a MySQL replacement.
If you'd like more information, do check out our project page on Launchpad and browse through the mailing list archive. Drizzle development is done in a true open source fashion by using open resources and tools like Bazaar and Launchpad. This means that everyone is free to come up with improvement suggestions/patches and submit it to the drizzle community.
Drizzle has been very fun and I thank Brian for getting me involved in such a fun project :)
Btw, I wrote a blog post on Drizzle in Japanese on the Mixi engineering blog too.
こんばんは、久しぶりの日本語エントリーです。
私は普段はemobileから販売されているExpressCard/34 (D03HW) タイプの通信カードとMacBook Proで快適ワイヤレス生活を送っているのですが、最近になってUbuntu専用で使っているExpressCard/34に対応していないノートPCでもemobileを使いたくなったので、USBアダプターを購入して実験してみました。
結論からいうと、普通に動きます。
マシンはThinkPad X61で、OSはUbuntu 8.04 (Hardy Heron) です。購入したアダプターはサンワサプライ製のADR-EXUSBです。ヨドバシカメラで1700円ちょっとで売ってました。PPPの設定も簡単で、私は直接、/etc/ppp/peers/にemobileの設定ファイルを書き込みましたが、直書きが好きでない方は、ここのppppconfigを使った方法を参考にすると楽に接続できるはずです。
今回はウェブ調査をしてもアダプターを使った成功事例が見当たらず、ちょっとドキドキでしたので、動作してくれて何よりです。
So memcached is used to scale and improve the performance of almost every large websites that you love. This is not an exception for Japan either. For example, mixi, the largest social networking website in Japan rely greatly on memcached. I also know handful of other large websites here in Japan that utilizes memcached, though I will keep them unnamed for now.
Considering the amount of questions that flow through the memcached mailing list and IRC, I haven't seen many (if not any) questions being thrown by a Japanese developer. Why? IMHO there are three possible reasons behind this:
- Developers in Japan don't care about memcached.
- There is a user community in Japan where they exchange information.
- Developers aren't comfortable throwing a question in English.
The first possibility can't be true since Brian Aker and myself overloaded the registration for a memcached talk at MySQL in Tokyo within 24 hours of its announcement. The second reason can't be true since up until now there was no memcached community in Japan.
Therefore the third possibility seemed like the likely one to me. Though mind you, speaking from experience, developers in Japan can read English fine. People just don't feel comfortable when it comes to verbal communication.
Anyway, the important thing is that there was no reason to _not_ start a memcached community in Japan, since what makes an open source software great is the community. I asked what Brad thought of this thought and since he thought it was a good idea as well, I launched the group last friday :-)
The group is maintained using Google Groups:
I've also created an official summary for the group here:
Interested in how things are going? Well... we have now over 80 registered members within the first week of its launch! I really can't wait to throw an event full of web performance geeks in Japan :D
Last week I spoke at the MySQL seminar in Tokyo as a guest speaker with Brian Aker on "Memcached and MySQL". The seminar turned out to be very fun with just over one hundred attendees. You can checkout the photos from here.
At the end of the seminar, I showed a brief demo of the custom storage engine project that Trond Norbye and I have been working on for the last few weeks. If you're interested, you can read more about it on his blog, or come on over to the memcached channel on freenode :)
http://blogs.sun.com/trond/entry/memcached_and_customized_storage_engines
Even though I don't consider myself a Perl hacker, the two days at YAPC::Asia were surreal. It was great fun and it was nice to hear the current trend in Perl. Specifically, I found the "~~" operator in 5.10 to be quite nifty (must test to see if there is any overhead involved in using this VS doing your own check). Thumbs up to everyone that made this fantastic event come true :D
I was fortunate enough to meet Larry Wall at the dinner party too :D
Recently I transferred my domain (http://torum.net) to a Japanese domain management company since my old one had no attractive features to offer me. I also figured that it was a good occasion to start using the features that I haven't bothered using on Google Apps that heaps of people seem to rave about (I've only been using their mail feature).
Long story short, Google Apps is __NOT__ overrated at all. I've become used to their mail service so that was nothing new but their Page Creator feature really kicks ass. Thanks to the feature I prepared what you see on http://torum.net in like 10 minutes or so :D
A new symbolic tower is currently being built in Tokyo and is expected to be completed in 2011. The developers are now calling for votes to decide the name of the tower. This is a good thing but whats interesting are the candidates:
- Tokyo Edo Tower
- Tokyo Sky Tree
- Mirai Tower
- Yumemiyagura
- Rising East Tower
- Rising Tower
Interesting... wonder how these names survived the selection process.
Recent Comments
-
-
tmaesaka said:
Yes.I still need to find accommodation though. I would ask if I could crash at your place for few days... read more
on Perl, Binary and Memcached -
-
on memcached Users Group in Japan