One of the questions I’m often asked – especially from a BI perspective – is how a BI person should look at HBase.  After all, HBase is often described quickly as an in-memory column store database – isn’t that what SSAS Tabular is?   Yet calling HBase an in-memory column store database isn’t quite right because in this case, the terms column, database, tables, and rows do not quite mean the same thing as one would think from a relational database aspect of things.

Setting the Context

How I usually start off is by providing a completely different context before I go back to BI.  The best way to kick this off is to know that HBase is an integral part of Facebook’s messaging system.  Facebook’s New Real-time Messaging System: HBase to Store 135+ Billion Messages a Month is a great blog post providing you the architecture details on how HBase allows Facebook to deal with excessively large volumes of volatile messages.  This isn’t something you would typically would see in the BI world, eh?!

Understanding HBase and BigTable

With the context set, let’s go back to understanding more about HBase by reviewing Google’s BigTable.  Understanding HBase and BigTable is a must read, the blog’s key concepts are noted below but attribution goes to Jim R. Wilson.

Concepts refresher for the definition of BigTable is:

A BigTable is a sparse, distributed, persistent multidimensional sorted map. The map is indexed by a row key, column key, and a timestamp; each value in the map is an uninterpreted array of bytes.

With the keywords here being:
map | persistent | distributed | sorted | multidimensional

Following JSON notation, a map can be seen as:

{
  "zzzzz" : "woot",
  "xyz" : "hello",
  "aaaab" : "world",
  "1" : "x",
  "aaaaa" : "y"
}

while a sorted map is

{
  "1" : "x",
  "aaaaa" : "y",
  "aaaab" : "world",
  "xyz" : "hello",
  "zzzzz" : "woot"
}

{
  "1" : {
    "A" : "x",
    "B" : "z"
  },
  "aaaaa" : {
    "A" : "y",
    "B" : "w"
  },
  "aaaab" : {
    "A" : "world",
    "B" : "ocean"
  },
  "xyz" : {
    "A" : "hello",
    "B" : "there"
  },
  …
}

As for the description of persist and distributed, the best way to see this is through a picture from Lars George’s great post: HBase Architecture 101 – Storage

Some of the key concepts here are:

• HBase is extremely efficient at Random Reads/Writes
• Distributed, large scale data store
• Availability and Distribution is defined by Regions (more info at: http://hbase.apache.org/book/regions.arch.html)
• Utilizes Hadoop for persistence
• Both HBase and Hadoop are distributed

If the concepts are still a little vague, please do read Jim R. Wilson’s post   Understanding HBase and BigTable and then read it again! 

So what about Analytics?

Through all this terminology and architecture design, can HBase do analytics?  And the answer is a yes, but its design is not about creating star schemas in the relational sense but creating column families which fits nicely into real time analytics.  A great way to dive into this is to check out Dani Abel Rayan’s Making Sense of Streaming Big Data Flume- HBase.

There are many other excellent references here and this is hardly an exhaustive post. But the key thing here is that while there are many similarities, HBase analytics and SQL Business Intelligence have different contexts.  Because of its in-memory column families, its easy to adapt HBase for real time analytics (in addition to its ability to handle volatile messages).  On the other hand, BI is about the ability to slide and dice using familiar BI tools against immense amounts of historical data.  Over time we may see the concepts merge – Google BigQuery comes immediately to mind – but it’ll be awhile.

Meanwhile, I encourage folks that are comfortable with SQLBI to stretch the envelope a bit into the world of NoSQL like HBase.  It’s a paradigm shift in some ways,… yet it also isn’t!

References

• Apache HBase Reference http://hbase.apache.org/book.html
• HBase: The Definitive Guide by Lars George http://www.amazon.com/HBase-Definitive-Guide-Lars-George/dp/1449396100
• HBase @ Facebook – The technology behind messages by Kannan Muthukkaruppan.