MapReduce in a Nutshell

Tue, 01 May 2012 00:44:22 +0000

For the last few weeks I've been meaning to start writing in my blog again. It's been more than a year since I've updated it regularly, an I wanted to start getting into the habit again. When my developer friend asked me about MapReduce it jumped out at it me as a good topic to ease back into tech blogging. Not to say there aren't many great sources of information on MapReduce already.

MapReduce is a simple parallel programming model for for processing large amounts of raw data. Created by Google, and inspired by the map and reduce combinator methods from functional languages like Lisp, it was developed as a mechanism to perform simple computations on massive datasets while hiding the details of parallelization, data distribution. In short, MapReduce is an abstraction that allows developers to implement scalable parallelism easily.

But don't let all that talk about distributed computing scare you. The beauty of MapReduce is that it is built to mitigate concurrency issues in our programs, and obfuscate the complexities of parallel programming from us.

Implementing a application that uses MapReduce is actually quite simple. I'll go over a simple Word Count example as I explain the basic concepts.

Phase I: Parallel Programming

In Map/Reduce algorithms, the original dataset, usually a large set of records or documents, are partitioned into many smaller workable subsets. These subsets are then passed along to a pool of workers operating concurrently (most likely on some queuing system like Resque or RabbitMQ) to process the data.

Phase II: Map

Once partitioned into manageable subsets, the data is given to worker processes which invoke map, passing the data as input, to be processed. Map will be invoked for each record/document in the original dataset, through multiple workers, working in parallel. The map function takes as input a function and a sequence of values. It then applies the function to each value in the sequence. The job of the function is usually to extract the data required from the record/document. Map uses this data to produce a set of intermediate key/value pairs. If you've ever used Ruby's map function on an enumerable, then you should have a pretty good idea of how this works. Map's primary logical operation is to group the data by keys. As it groups the data, the map function calls emit. Emit is the function used to feed the grouped data to the reduce function.

Word Count Example:

key = :page
val = "to be or not to be"
def map(key, val)
  val.split.map do |word|
    emit(word, 1.to_s)     
  end
end

Phase III: Reduce

Reduce will be called once for each unique key value emitted by the map stage, in parallel. The reduce function is used to take these groups (individually) and execute a function defined by the developer, with the grouped values as input. The function processes the values which were grouped together, to calculate a final result (Usually some sort of aggregation). Just as with map, as a ruby developer, you have probably used Ruby's reduce in some kind of accumulative operation (although it is more widely known by its synonym, inject). Common operations in reduce functions include: Average, Min/Max, and Sum.

Word Count Example:

# word - i.e. 'to' or 'be', etc.
# values - intermediate values  => ['1', '1', '1',...]
def reduce(word, values)
  result = values.inject(0) do |result, count|
    result += count.to_i
  end
  emit(word, result.to_s)
end

But since we're talking about reduce, we could also do it like this:

def reduce(word, values)
  result = values.reduce(:+) 
  emit(word, result.to_s)
end

Raising Gremlins

Mon, 01 Nov 2010 04:58:01 +0000

I was watching the old classic 'Gremlins' one lazy Saturday. I hadn't seen it since I was in college, and a lot has changed since then, one of the most important of which is the fact that I've started a family.

The older, slightly fatter version of me was struck with the similarities of raising gremlins (or Mogwai, their pre-pupal persona) and children. People without kids will probably gasp as they read this, shocked by my willingness to compare my own progeny to villainous little demons. But I take heart in the knowledge that my fellow parents will understand completely.

The first, and most obvious similarity is their mischievous nature. I can't speak for everyone, but my children, just like gremlins, are in everything! Touching, exploring, playing, breaking... I've went though 6 DVD players since having children. They climb on everything, and swing off of anything that will support their weight (even if it can't support it for very long, and usually doesn't). My two older sons are geniuses at taking things apart. Most of the time it's things I'd have preferred stay in one piece. They also have an uncanny knack for breaking their toys, and re-purposing the refuse into new gadgets the manufacturer never intended.

My kids curious (if sometimes destructive) nature, not withstanding, it was the parenting where I found the most striking resemblance. It was the rules! The rules that governed the Mogwai, the rules that their owner carelessly disregarded. It reminded of the discipline and vigilance that all parents must have, and the consequences of laziness. You have to become a parent fully appreciate the pressure of having someone's life in your hands.

We all want our kids to be cute little Mogwai. Some of us though, don't understand the huge part we play in that. It's our care and guidance that lead kids to become who they are. I guess my point is that if your kid is a gremlin, it's probably your fault...

Event Horizon

Wed, 16 Jun 2010 08:33:17 +0000

Having just recently moved my family from Chicago, the city where I spent most of my life, to the Bay Area, a place I barely knew before deciding to resettle here, I've been giving a lot thought to the forces that drive us to do what we do. You see, when I made up my mind to move, I didn't really know what I wanted, I just knew I had to find a better quality of life than the one I had. And once I decided to find that, there seemed to be something pulling my here to the Bay. Some unseen gravitational force seemed to guide my decisions and clear my path of any obstacles that kept me from here. And now, having settled in a town on the East Bay near Oakland, I feel it still. As if even though I have found a place to call home, my journey continues. That feeling has caused me to wonder lately, where is the destination? I recently read an article written by Michael Rubin, the CEO of the company I just started with. The article, Serendipity vs. the Act of Will, got me thinking a lot about the energies that move us through life, and how we relate to those energies, whether it be just allowing them to carry us from one event to the next, pushing against them, in a futile attempt to control our own fates, or trying to capitalize on them by reacting at the right time. How am I interacting with the force I've felt lately? Am I allowing it to carry me aimlessly, or am I harnessing it, like sailors harness the wind to take them where they want to go?

Answering that question has lead me to categorize this time in my life as an Event Horizon. In relativity, an event horizon is a boundary in space-time (most often an area surrounding a black hole) beyond which events cannot affect an outside observer. Light emitted from beyond the horizon can never reach the observer. I know, what? Hold on a second, I'm going somewhere with this. You see, I feel like this gravitational force has been pulling me to some point I need to reach in my life, and while I can see (or maybe feel) a basic direction, I can't quite see the destination I'm trying to reach. My "Event Horizon" is allowing me to feel the pull of something important, something that will profoundly affect me. At this stage in my life however, while the strength of it's gravity belies its significance, I as the outside observer, cannot fully comprehend it. In much the same way one might approach a blackhole. The event horizon prevents the outside observer from seeing beyond the blackhole, nevertheless it's gravitational pull is inescapable.

General relativity also explains that any object that approaches the horizon from the observer's side appears to slow down and never quite pass through the horizon, The traveling object, however, experiences no strange effects and does, in fact, pass through the horizon in a finite amount of proper time. In much the same way, wheresoever my journey is taking me, I can't know how close or how far I am from it, nor can I tell at what pace I approach it. I nevertheless, move closer to it each day, and time passes in spite of me. I have come to the conclusion that my new job is part of my journey. When I was approached about starting at PlaceBook, I had my reservations. But the same feeling that led me to California seemed to speak to me again. There was something about PlaceBook and the people involved that spoke to me. I don't know if my journey ends here, or is just beginning. What I do know, is that we're building something special.

Prince of Persia My A$$

Sat, 29 May 2010 09:37:28 +0000

I went to see Prince of Persia this evening. Not bad. A little escapist fantasy to take the edge off of a long, stressful week. But as I do at the end of all movies, I began to analyze. The gift (and curse) of an engineer's mind. If a movie is really good, I spend hours analyzing the characters, the plot, and the plot twists. But for a movie of this caliber, it's usually the choices made by the creators that occupies my mind. It was on the drive home that I asked myself the question: What is it about non-white leading characters that Hollywood power brokers find so grotesque? I can't figure it out. Jake Gyllenhaal is a good actor, but a white kid flipping through the slums of an ancient Persian capitol? Come on! Why is that not ridiculous to these Hollywood types. They can picture all the peasants and village people as dark-skinned, middle-eastern people, but royalty and heroes without blue eyes is unimaginable?

There's another summer popcorn flick coming out soon called The Last Air Bender. It's based on my son's favorite cartoon, so he's really looking forward to it. All the main characters in the cartoon version are of Asian decent, but in the movie version, they are all portrayed by white actors. It wouldn't bother me so much, except that this seems to be the rule, not the exception. Part of me says to just let it go, but the more I think about it, the more it vexes me. Not the act itself; it's the thinking behind it. Why? I just want to know why? Is it racism? Is it a money thing? Do they believe a movie won't make money because the main character isn't white?

Racism seems like the most obvious answer. But I can't believe every producer and director in the movie business is racist! So once you eliminate the impossible... Money? Could this all really be about money? Is it possible that the powers that be, in the movie industry subscribe to the insane notion that non-Caucasian main characters result in poor box office results? It couldn't be that! 20 years ago, they'd have an argument. But NOW?! In 2010?! This is the age of President Obama and Will f#(%ing Smith!!

I mean come on! Think about it! They can't possibly believe that the people who elected a black president, would have a problem with a olive-skinned Persian prince, rescuing a dark-skinned Arab princess. If minority leads result in poor box office showings, then how do they explain Will Smith, and Denzel Washington? Two box office titans, and the highest paid leading men in the business today. Even if you write off Mr. Smith and Mr. Washington as flukes, you'd still have Slum-Dog Millionaire and Tyler Perry's Why Did I Get Married?.

Still not convinced? Here's a fluke for you: Ninja Assassin. An American-made Ninja movie with an actual Japanese lead! Most U.S. Ninja movies feature a white guy in a Ninja costume, fighting a hundred Asian extras (beating them all, of course. Because we all know that it takes a white man to really master the Ancient Japanese art of Jujitsu). It goes without saying that most of those flicks sucked and didn't make a nickel. But here we are with a good Ninja flick, with a Japanese lead, and you know what? It did pretty well. There are those that would say that's progress. It is to them that I say this: Kiss my Ass!! Persians ain't White!!

My Thoughts On Interviewing

Mon, 24 May 2010 04:30:26 +0000

As an experienced software developer having served in junior, senior-level, and even lead roles, I have found myself on both sides of the interviewing table. I'd like to think that on my best days (and I've had some bad days, to be sure), I'm a fairly good interviewer. And since I've had my share of jobs, I assume that I must be an decent interviewee as well.

Over the years I've developed a few opinions on the particular interview strategies used to gauge whether someone will make a good addition to a software development team. Following are my feelings regarding those techniques.

1. Logic Puzzles

A lot of people I've worked with swear by these obscure gems to determine a candidates ability to solve intricate problems. I however don't think these puzzles are worth all the effort. For one, most of them have these incredibly over-thought "AHA!" solutions. Most of the time, I'm not interested in whether the candidate can come up with the solution to some odd conundrum. I am more interested in their grasp of and experience with well known, proven development methodologies, since those will be tools he's expected to use on a day-to-day basis. I can discern his reasoning abilities by watching him analyze actual code, rather than inferring his coding skills from his ability solve brain-teasers.

2. Text Book Talk

Judging candidates on their ability to recite obscure theory, rather than finding solutions to real-world problems. I've seen way too many interviewers put candidates (including myself) through these tedious exercises that ask O-notation for old sort algorithms. These are useful at companies like Google, where that knowledge may be pertinent. But not every company requires that kind of expertise. I've seen good candidates, with all the right skills and experience for the task at hand, get passed over because they didn't know some obscure fact that was irrelevant to doing the job to begin with.

3. Buzzword Bonanza

Lastly, I've watched interviewers, pass on seemingly knowledgeable candidates (usually in the initial phone screen) because they misspoke some new Web 2.0 term, or didn't know the latest tech acronym. If a candidate can show that they understand the concept you're asking about, then a misspeaking a term or buzzword should NOT be a deal breaker.

To Be or not to Be: Understanding Ruby Nil

Mon, 12 Apr 2010 06:01:03 +0000

As all Rubyists know, everything in Ruby is an object, including true, false, and the integer 1. But what about the absence of a value? Does 'nothing' constitute an object in Ruby? Well as a matter of fact, it does. Unlike languages like Java and C-sharp, where null is a keyword that represents a void reference (i.e. a pointer that points to nothing), the nil keyword in Ruby evaluates to a predefined global object. That object is the one and only instance of the NilClass.

Like all objects you can call methods on nil (assuming it responds to called method). Nil also has an object id. It's object id is always 4. Right behind true and false which have object id's of 2 and 0 consecutively. They have even object ids because odd numbers are used for integers.

  >> nil.class
  => NilClass
  >> nil.object_id
  => 4

As an object, ruby pre-defines nil with a few convenience methods to avoid non-critical situations that maybe complicated by an unexpected nil object. In Java, since null is NOT an object, any method call would result in a NullPointerException. In C-sharp, similarly, calling a method on a null reference would result in a thrown exception.

  >> nil.to_s
  => ""
  >> nil.to_i
  => 0

Another convenience method in ruby is "nil?". For all objects except for nil itself, this method returns false. As you've probably guessed, it returns true when called on nil.

  >> nil.nil?
  => true
  >> Object.new.nil?
  => false

Rails provides two other methods: "blank?" and "present?". In the case of "blank?", it returns true for nil or for empty strings and arrays. The method "present?", is just the opposite of "blank?".

  >> nil.blank?
  => true
  >> "".blank?
  => true
  >> [].blank?
  => true
  >> Object.new.blank?
  => false
  >> str = "str"
  >> str.blank?
  => false
  >> str.present?
  => true
  >> arr = [1]
  >> arr.blank?
  => false
  >> arr.present?
  => true

Another cool trick provided in Rails is the "try" method. This method is a shorthand for testing if an object is nil before calling a method on the object. So asuming you have a user object, and your're unsure if the object is nil or not. You may do something like this...

    name = user.name unless user.nil?

Well, you could use the "try" method to do the same thing. Like so...

    name = user.try(:name)

On most objects, the "try" method is just an alias for "send". But on the Nil class, it simply returns nil. Remember, however, that if the object is not nil, and the method does not exist, the MethodNotFound error will be raised as expected.

Understanding Ruby Procs and Lambdas

Thu, 11 Mar 2010 23:18:22 +0000

Earlier this year, I was searching for a job in the Bay Area. That search eventually lead me to my StrongMail, my current job where I hold the position of Principal Software Developer. But before finding this job, I was interviewing a lot. being the die-hard Rubyist that I am, I limited my search to companies with Rails shops. So, as you imagine, I got a lot of Ruby questions. After a few of interviews, I noticed a small selection of questions that seemed to be interviewer favorites. The 'Proc vs Lambda' question stuck out immediately because it was asked in every interview. Which brings me to the reason for this post. If you are considering applying for a job as a Ruby Developer, you're very likely to hear this question. If you're already a Ruby Developer, but you were never sure about differences between these two Ruby paradigms, this'll benefit you as well.

So on with it then! What is the difference between a Proc and a Lambda? Well, a Proc is an object that represents a reusable code block, while a lambda is essentially a method, albeit an anonymous one. If you understand that, then you're half-way there. But hold on, don't leave just yet, here's more... So what behavior does the aforementioned distinction suggest? To answer that, you have to think about how Ruby handles procedures and methods.

The first notable difference in behavior is that lambdas, like all ruby methods check the number of arguments it is passed, but Proc implicitly sets missing args to nil.

  >> block = lambda {|a, b, c| "arg1: #{a.inspect}, arg2: #{b.inspect}, arg3: #{c.inspect}"}
  >> block.call(:x, :y)
  => ArgumentError: wrong number of arguments (2 for 3)

  >> block = Proc.new {|a, b, c| "arg1: #{a.inspect}, arg2: #{b.inspect}, arg3: #{c.inspect}"}
  >> block.call(:x, :y)
  => "arg1: x, arg2: y, arg3: nil"

Another difference in behavior is in returns. A Proc return stops the method and return the generated value, lambdas return their value to the method and lets the method continue on. Why? Because a Proc is an object that represents a snippet of code, with no scope or context of its own.

So this...

def foo
  x = 'foo'
  Proc.new { 
    y = 'bar'
    return y
  }.call
  return x
end

>> foo
=> 'bar'

Is equivalent to this...

def foo
  x = 'foo'
  y = 'bar'
  return y
  return x
end

>> foo
=> 'bar'

In this example foo will always return 'bar', return x is never reached, because proc is simply a block of code within foo, so the proc's return is the method's return. Lambda's on the other hand, are methods with their own return, so just like any other method within the class or module, it returns the value to the caller.

So this...

def foo
     x = 'foo'
     lambda { 
       y = 'bar'
       return y
     }.call
     return x
   end

>> foo
=> 'foo'

Is equivalent to this...

def foo
  x = 'foo'
  bar
  return x
end

def bar
  y = 'bar'
  return y          
end

>> foo
=> 'foo'

In the above example a lambda behaves as any other method, so the method always returns 'foo', since the return value of 'bar', is never assigned, or referenced outside of the original method call.

Life After Death

Wed, 17 Feb 2010 23:49:00 +0000

For all you Rails heads waiting for the development posts, just hold your horses, they're comin'. I just got a lot of stuff I want to say first... Anyway, It was just over a year ago, that I almost died. Yep, that's right. Died. I laid in a coma for 9 days with a pneumonia the docs didn't think I could beat. They were preparing my wife for the worst. But since you're reading this, you already know that I don't, so I'll skip all the melodrama. The point is, you're not promised anything. You have live while you got the chance. And speaking of living, for a while, after the illness, I'd given up. I lived only to watch my best friend die... but that's another post. But for that and other reasons, I wasn't thankful for the new lease on life that I'd been given. It took time, but I came out of it. And I'm back! I'm celebrating everyday now! Cuz I won! That's right, I'm still here, and I'm still standing!

Blog Init

Tue, 16 Feb 2010 14:26:00 +0000

Wow, after spending the last few nights coding, my personal blog is finally ready for it's first post. If you're reading ths, you most likely already know that I'm Rahmal Conda, Software Developer, Ladies Man... and if you're wondering if you should continue reading, or of you should comeback for the next post, then by all means continue if you're interested in Ruby, Ruby on Rails, or Software Development in general, or just find my personal musings, observations and rants amusing, then by all means, stop by anytime.