4 posts tagged “memcached”

View other items tagged “memcached:” Photos, Audio, Videos, Books or Links. View “memcached” on Vox.

Micro Blogging Conference at Cybozu Labs

Microblogging is something that is becoming popular in Japan (though I wouldn't say "big"), and at the same time an interesting application that involves many technical challenges. A micro blogging infrastructure must be able to scale both write and read requests and like any problems, there are various ways of solving these problems, some better than others. 

Thanks to Kazuho Oku, who hosted a micro blogging conference at Cybozu Labs last night, web developers gained information on how real-world applications are solving these challenging problems. Speakers included folks from Wassr (one of the largest micro blogging service in Japan) and Mixi (the largest social networking service in Japan), who had recently began experimenting with micro blogging called "mixi Echo". If you're interested in the architecture of mixi Echo, you can take a look here (the actual number of servers are much more).

So what was I doing at the conference? I was there to speak about middleware development, namely the concept of modularizing the backend of memcached and how this feature could help reduce people from forking memcached. I also described how the plugin mechanism is achieved in the prototype developed by Trond Norbye from SUN and myself.

I would love to cover the details in this entry but unfortunately time is against me at the moment so I will save it until my next entry. For now I just wanted to get the word out that we're very serious about micro blogging infrastructure in Japan ;)
Post a comment Tags: conference, memcached

Perl, Binary and Memcached

The last few days I've been working on updating the binary protocol test in the latest memcached development branch to comply with the latest binary protocol specification. Prior to this update, the test client was sending an invalid request to the server, which as a consequence made the test hang and never finish.

In brief, this is the big difference:

Previously, CAS (compare and swap) value was treated as part of the extra header that is appended/serialized behind the request/response header. In the latest specification, CAS value is a required 8 byte field in the 24 byte request/response header (header size in the previous version was 16 bytes). Other than that, the rest were minor differences in the packet format of extra fields in certain commands. Easy work :)


As you can see, the test suite currently used by memcached is Perl based. This was fortunate for me since Perl is the second language to C that I like. I also made the code style to be more "perl-like" by fixing the indents. Although heh, I can see a Perl programmer arguing that the use of if/else blocks in the test is not best practice :P

You know, fixing the test was pretty meaningful to me since it had forced me to study the binary protocol specification, which I knew almost nothing about at the hackathon in Santa Clara, CA back in april. Hopefully I can make productive suggestions at the upcoming hackathon in Menlo Park, CA in october.
2 comments Tags: memcached, binary protocol

memcached Users Group in Japan

So memcached is used to scale and improve the performance of almost every large websites that you love. This is not an exception for Japan either. For example, mixi, the largest social networking website in Japan rely greatly on memcached. I also know handful of other large websites here in Japan that utilizes memcached, though I will keep them unnamed for now.

Considering the amount of questions that flow through the memcached mailing list and IRC, I haven't seen many (if not any) questions being thrown by a Japanese developer. Why? IMHO there are three possible reasons behind this:

  • Developers in Japan don't care about memcached.
  • There is a user community in Japan where they exchange information.
  • Developers aren't comfortable throwing a question in English.

The first possibility can't be true since Brian Aker and myself overloaded the registration for a memcached talk at MySQL in Tokyo within 24 hours of its announcement. The second reason can't be true since up until now there was no memcached community in Japan. 

Therefore the third possibility seemed like the likely one to me. Though mind you, speaking from experience, developers in Japan can read English fine. People just don't feel comfortable when it comes to verbal communication.

Anyway, the important thing is that there was no reason to _not_ start a memcached community in Japan, since what makes an open source software great is the community. I asked what Brad thought of this thought and since he thought it was a good idea as well, I launched the group last friday :-)

The group is maintained using Google Groups:
I've also created an official summary for the group here:

Interested in how things are going? Well... we have now over 80 registered members within the first week of its launch! I really can't wait to throw an event full of web performance geeks in Japan :D
1 comment Tags: memcached

Thoughts on making memcached modular

Although its been a week or two since my multi-storage modification was announced in the memcached mailing list (thanks Brian), figured I should write my thoughts and not just code on this matter.

The aim simply is to achieve a network platform that can be widely used in the web industry. By meaning "network platform", I am trying to emphasize that what can be plugged to memcached doesn't necessarily need to be a storage engine.

For example, you could plug a task-queue engine that will periodically perform the tasks in the queue. Representing a task with a <key, value> pair is fairly straight forward since we can use something along the line of <task_type, related_data>. Since we are not using a hash structure but a queue for this example, you don't have to worry about overwriting the previous task(s) that you registered with what used to be the "key" ("task_type" in this example). Hmm.. come to think about it, you can probably use append to do this too.

This is the beauty of making memcached modular. The module writer has control over what to do with data that memcached receives from the client.

What I've mentioned so far are just bonuses from achieving modularity. For my personal use/hobby, I want to link memcached to a fast hash storage engine (Tokyo Cabinet is my favorite at the moment) and use it for fast persistent storage.

So yeah, though the patch that is floating around atm is only my initial attempt, I've managed to get the idea in shape since discussing this idea with Brian Aker last year in Tokyo. l seem to have several people in the community interested and like Dormando has stated, hopefully we'll have something exciting for version 1.4 :-)

Post a comment Tags: oss, memcached
tmaesaka

About Me

tmaesaka
Japan
View my profile
Facebook:
218600382
LiveJournal:
tmaesaka
Pownce:
toru
Twitter:
tmaesaka

Neighborhood

Explore friends, family, friends & family, or entire neighborhood.

View my neighbors

Archives