My First Writable With Hadoop


Hadoop uses a simple and efficient serialization protocol to serialize data between the map and reduce steps.  There is a lot going on between these two steps, but this article is not about that.  Rather it focuses on what a developer needs to know in order to write a custom Writable class.  Just for the folks that are new to MapReduce in Hadoop, the OutputCollector in the Map and Reduce step accepts only the value as of type Writable.

Writable is an interface in Hadoop and it has two methods: void readFields(DataInput in) and void write(DataOutput out).  If you browse Hadoop Javadocs, ther are roughly about 43 classes in Hadoop that implements the Writable interface.

Depending on what your MapReduce application needs, one of the out-of-box Writables will do the job, but if there isn’t one, then it is fairly straight forward to write a custom Writable.  That’s what I had to do for my project.  What I discovered and there isn’t that much documentation on it is in addition to the two methods defined in the Writable interface, you also need to implement the toString() method if you want the data in your custom Writable to appear in the output file (this took me sometime to figure out).

One of the interesting Writables is the GenericWritable.  This comes in handy when the Map and Combiner output the same key type, but different value type.  The requirement in Hadoop is the values that are mapped out to reduce, only one value type is allowed.  The GenericWritable basically helps you wrap instances of value of different types.  See GenericWritable JavaDoc for more details.

Advertisements
This entry was posted in Lucene Hadoop, Uncategorized and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s