Braised Lamb Shank in Cumin and Star Anise

Photo credit goes to Steph L. on Yelp

Vancouver is regularly ranked in the top liveable city list (for awhile, they were #1) – the temperate environment, beautiful scenery, and general aura of the city is just amazing.  Just as awesome is the city’s quality and diversity of great food.

Amazing Malaysian Cuisine

One of those places is most certainly Banana Leaf Malaysian Cuisine.  As of this post, they have four restaurants in the Vancouver area – I personally have went to both the one on Denman and the one in Kits – both are amazingly good.

Normally, I’m one to try all sort of food – I rather pride myself on the more than willingness to eat all sorts of things that people would find … weird.  In this, I am definitely Chinese.  

As noted in the movie “The Corruptor” (not the best movie even with the great actors Chow Yun-Fat and Mark Wahlberg)

You wanna be Chinese, you gotta eat the nasty stuff”

Fortunately, this is certainly NOT the case for Banana Leaf – aromatic, bold, delictable, and flavourful (not salty – and yes, I did spell flavour the Canadian way) are the words that come to mind.

While there are many great dishes – my personal call outs are:

Braised Lamb Shank in Cumin & Star Anise: Just as the menu describes it, the lamb shank is so tender it literally falls off the bone.  Amazingly good with just the right amount of spices to bring out the flavour of the lamb instead of over powering it.  The lamb shank is cooked just right to ensure that the meat is amazingly tender (over cook lamb, and you’ve got yourself one awesome piece of rubber).  If you order nothing else, this IS the dish you order. Period.

Pineapple Fried Rice with Seafood & Chicken: I almost never, ever, ever, ever, ever, …, ever order fried rice from an Asian restaurant…ever.  Often its too much soy sauce, too much MSG, adding ketchup (seriously WTF, fried rice made with ketchup!!!), …. etc.  This is seriously not the case.  Served in a scooped out pineapple – the fried rice is lightly sweet with solid heapings of seafood and chicken.  Quite filling – in a good way!

And the final call out of course goes to dessert.  I certainly have what one would call a “sweet tooth”.  And the Pisang Goreng is my current dessert choice du jour.  Crispy fried banana, vanilla ice cream, crushed peanuts, and gula melaka (which I found out from Wikipedia is palm sugar) – what’s not to love!

So the next time you’re in Vancouver and you’re up for some good Malaysian cuisine – or just good food in general – check it out.  I leave you with the wisdom of George Bernard Shaw:

There is no love sincerer than the love of food.

Apologies for the nerdy chemistry references in advance – which starts with this picture of Dr. Arthur Grosser (more later)

As noted in the previous post Scale Up or Scale Out your Data Problems? A Space Analogy, the decision to scaling up or scaling out your data problem is a key facet in your Big Data problem.  But just as important as the ability to distribute the data across commoditized hardware, another key facet is the movement of data.

Latencies (i.e. slower performance) are introduced when you need to move data from one location to another.  To solve this problem within the data world, you can solve this by making it easier to move the data faster (e.g. compression, delta transfer, faster connectivity, etc.) or you design a system that reduces the need to move the data in the first place (i.e. moving data to compute or compute to data).

Scaling Up the Problem / Moving Data to Compute

To help describe the problem, the diagram below is a representation of a scale up traditional RDBMS.  The silver database boxes on the left represent the database servers (each with blue platters representing local disks), the box with 9 blue platters represents a disk array (e.g. SAN, DAS, etc.), the blue arrows represent fiber channel connections (between the server and disk array), and the green arrows represent the network connectivity.

In an optimized scale up RDBMS, we often will setup DAS or SANs to quickly transfer data from the disk array to the RDBMS server or compute node (often allocating the local disk for the compute node to hold temp/backup/cache files).  This scenario works great under the specific scenario that you can ensure low latencies.

And this is where things can get complicated, because if you were to lose disks on the array and/or fiber channel connectivity to the disk array – the RDBMS would go offline.    But as described in the above diagram, perhaps you setup active clustering so the secondary RDBMS can take over.

Yet, if you were to lose network connectivity (e.g. the secondary RDBMS is not aware the primary is offline) or lose fiber channel connectivity, you would also lose the secondary.

The Importance of ACID

It is important to note that many RDBMS systems have features or designs that work around these problems.  But to ensure availability and redundancy, if often requires more expensive hardware to work around the problematic network and disk failure points.

As well, this is not to say that RDBMS are based design – they are designed with ACID in mind – atomicity, consistency, isolation, and durability – to guarantee the reliability and robustness of database transactions (for more info, check out the Wikipedia entry: ACID).

Scaling Out the Problem / Moving Compute to Data

In a scale out or distributed solution, the idea is to have many commodity servers; they are many points of failure but there are also many paths for success.

Key to a distributed system is that as data comes in (the blue file icon on the right represent data such as web logs), the data is distributed and replicated in chunks to many nodes within the cluster.  In the case of Hadoop, files are broken into 64MB / 128MB chunks and each of these chunks are placed into three different locations (if you set the replication factor to 3).

While you are using more disk space to replicate the data, now that you have placed the data into the system, you have ensured redundancy by replicating the data within it.

What is great about these types of distributed systems, they are designed right from the beginning to handle latency issues whether they be disk or network connectivity problems to out right losing a node.  In the above diagram, a user is requesting data, but there is a loss to some disks and some network connections.

Nevertheless, there are other nodes that do have network connectivity and the data has been replicated so it is available.    Systems that are designed to scale out and distribute like Hadoop can ensure availability of the data and will complete the query just as long as the data exists (it may take longer if nodes are lost, but the query will be completed).

The importance of BASE

By using many commodity boxes, you distribute and replicate your data to multiple systems.  But as there are many moving parts, distributed systems like these cannot ensure the reliability and robustness of database transactions.  Instead, they fall under the domain of eventual consistency where over a period of time (i.e. eventually) the data within the entire system will be consistent (e.g. all data modifications will be replicated throughout the cluster).  This concept is also known as BASE (as opposed to ACID) – Basically Available, Soft State, Eventually Consistent.  For more information, check out the Wikipedia reference: Eventual Consistency.

Discussion

Similar to the post Scale Up or Scale Out your Data Problems? A Space Analogy, choosing whether ACID or BASE works for you is not a matter of which one to use – but which one to use when.  For example, as noted in the post What’s so BIG about “Big Data”?, the Yahoo! Analysis Services cube is 24 TB (certainly a case of moving data to compute with my obsession on random IO with SSAS) and the source of this cube is a 2PB of data from a huge Hadoop cluster (moving compute to data).

Each one has its own set of issues – scaling out increases the complexity of maintaining so many nodes, scaling up becomes more expensive to ensure availability and reliability, etc.   It will be important to understand the pros/cons of each type – often it will be a combination of these two.   Another great example can be seen in Dave Mariani (@mariani)’s post: Big Data, Bigger Brains at Klout’s blog.

ACID and BASE each have their own set of problems, the good news is that mixing them together often neutralizes the problems.

—

Okay, what’s with the picture of Dr. Arthur Grosser?

Oh, Dr. Arthur Grosser is an actor whose filmography includes Assassin’s Creed II, Splinter Cell, and the 90s TV show Urban Angel. But more importantly – to me anyways – is that he was my chemistry professor at McGill University. He was a great professor able to balance deep academic research and learning with making chemistry fun and entertaining. He also showed to me (and I think many other students) that dorky and nerdy could still be cool.

How To: Upload Data and Use the WordCount Sample with Hadoop Services for Windows Azure (video)

Run the Pi Estimator Sample on Hadoop on Windows Azure (video)

But before we dive into the specifics surrounding relational databases compared to Hadoop / Big Data, we need to first talk about the differences between solving the a data problem by scaling up the problem or scaling it out.

One way to understand the difference is to use a space analogy.  (If you’re part of the SQL Twitter community, you’ll notice a prevalence of NASA and Space tweets)

Scale Up Space Analogy

The amazing image above is The Antennae Galaxies / NGC 4038-4039 from the Hubble Telescope – click on the image to see the full extra-large image, it is magnificent.

In terms of space analogies, the Hubble Telescope is analogous of a scale up technology such as your traditional relational database.

• It is often utilizes non-commodity hardware (or if it is commodity, it’s enterprise commodity hardware).
• Specialized equipment was designed and built for the Hubble telescope.  While not as astronomic (!) in price, enterprise database performance requires more specialized (and expensive) hardware.
• The Hubble telescope in itself is a single point of failure in that if we were to lose the telescope (or a lens), we would lose the ability to get all of these amazing images.

This is NOT to say scale up is a bad thing, after all we get these amazing images from the Hubble telescope precisely because we (well, NASA scientists) have focused on creating Hubble with non-commodity specialized hardware.  Following the same analogy, relational database systems (or cube systems) also benefit from this scale up approach because it becomes possible to provide query results quickly.

Scale Out Space Analogy

The flow of the above image is my representation of the Search for Extra Terrestrial Intelligence (SETI) project.  SETI itself has the ultimate example of scale out distribution with their SETI@Home project.  The search for extra terrestrial intelligence:

• Begins with the search of radio waves through out the galaxies.  The image above is that of the Giant Galactic Nebula NGC 3603 (also from the Hubble telescope)
• Those radio waves are detected by radio observatories located in various locations on the planet.
• All of this data is stored and initially crunched by super computers like the Barcelona Supercomputing Center.
• Yet how a lot of this crunching is done by BOINC-based projects like SETI@Home – i.e. making use of screensavers on one’s home machine.

That is, a sizeable chunk in the search for radio waves in the astrological heavens is being done by home computer screensavers – around 5.2 Million participants processing 769 teraFLOPS (11/14/2009) of data!

The key facets of commoditized distributed computing are then:

• The problems can be broken down into small enough chunks so they can be distributed and calculated locally.
• The system is designed – right from the beginning – to engage with hundreds or thousands of machines.
• The system can easily handle many points of failure transparently.  Auto-replication is one of the ways to prevent this from being a problem.

BOINC projects like SETI@Home are designed to handle the problems associated with distributed computing – network latency, loss of connectivity, tracking tasks, tracking jobs, etc. The fundamental being the ability to break down the problem into a small enough chunks so data can be easily transferred, processed, and transferred back – while keeping track of all of those chunks to ensure that data processing has been completed.

Bring this back to data systems, Hadoop and distributed data systems are able to take the problem and distribute this across tens / hundreds / thousands of machines with ease.  This is because they were designed with the idea of distributed processing in the first place (e.g. replication, fault tolerance, task restart-ability, etc.).

Discussion

There are many more concepts that need to be covered when we really dive into relational databases compared to Hadoop / Big Data systems.  But the fundamental to start with is that of “scale up” and “scale out”.

As you can see with the Hubble Telescope / SETI projects analogy – both are important and both solve their respective problems in different ways.  This doesn’t mean one is right and one is wrong – this is really more about the adage of “Use the right tool for the right problem”.   After all, it would be really hard to get the amazing images of Hubble telescope by using hundreds or thousands of smaller commodity telescopes from Costco.   Nor would it be possible for a single powerful telescope to examine all of the radio waves from the observatories on Earth.

So when it comes to scaling up or scale out for data problems – it’s not about which one to use, it’s about which one to use when.

This is a picture of an isolated beach in the wonderful town of Tofino (yes, I actually took it!).  Located on the west coast of Vancouver Island – if you are surfer, camper, hiker, or just plain old nature lover – this is a beautiful place to hang out.  Drive up from Victoria to Nanaimo (yes, of the famed Nanimo bars) and then cut through Vancouver Island to its west coast – it is a wonderfully scenic drive (this from a person that doesn’t like driving). 

For more information on Tofino, check out Tofino’s Wikipedia page – and check out Bing’s images of Tofino.  Oh, and if you go there – I highly suggest the local seafood joint The Schooner Restaurant. 

—

About “Sunny Sunday”: The Sunny Sunday blog posts are photos from various travel and/or outdoor (hiking) trips.

“I caught a fish thiiiiis biiig”

– On stage with Ted Kummert during the PASS 2011 Keynote on Big Data (thanks to Karen Lopez @datachick for the pic)

.
.
During the PASS 2011 Keynote (back in October 2011), I had the honor to demo Hadoop on Windows / Azure.   One of the key showcases during that presentation was to show how to connect PowerPivot to Hadoop on Windows.  In this post, I show the steps on how to connect PowerPivot to Hadoop on Azure.

Pre-requisites

• PowerPivot for Excel (as of this post, using SQL Server 2012 RC1 version)
• Access to Hadoop on Azure CTP

.

Configuration Steps

1) Reference the following steps from How To Connect Excel to Hadoop on Azure via HiveODBC

The steps to follow are the:

• Install the HiveODBC Driver (we will configure the DSN later)
• Steps 1 – 3 from Using the Excel Hive Add-In to open the ports in Hadoop on Azure

.
.
2) Create a Hive ODBC Data Source > File DSN

Here, we will go about creating a File DSN Hive ODBC Data Source.

Thanks to Andrew Brust (@andrewbrust), the better way to make a connection from PowerPivot to Hadoop on Azure is to create a File DSN.  This allows the full connection string to be stored directly within the PowerPivot workbook instead of relying on an existing DSN.

To do this:

• Go to the ODBC Data Sources Administrator and click on the File DSN tab.

• Click on Add, Choose HIVE, Click Next, Click Browse to choose a location of the file; click Finish.

• Open the File DSN you just created and click Configure.  The ODBC Hive Setup and configure the host (e.g. [clustername].cloudapp.net) and authentication information (the username is what you had specified when you had created the cluster)

.
.
3) Connect PowerPivot to Hadoop on Azure via the HiveODBC File DSN

• Open up the PowerPivot ribbon and click on the Get External Data from Other Sources.

• From the Table Import Wizard, click on the Others (OLEDB/ODBC) and click Next.

• From here, click Build and the Data Link Properties, click on Provider, and ensure the Microsoft OLEDB Provider for ODBC Drivers is selected. Click Next.

• In the Data Link Properties dialog, choose “Use connection string”, and click Build and choose the File DSN you had created from Step #2.  Enter in the password to your Hadoop on Azure cluster.  Click OK.

• The Data Link Properties now contains a connection string do the Hadoop on Azure cluster.

Note, after this dialog, verify that the password has been entered into the connection string that that has been built into the Table Import Wizard.  Note, the blue arrow points to a lack of a PWD=<password> clause.  If the password isn’t specified, make sure to add it back in.

• Click OK, click Next.  From here you will get the Table Import Wizard and we are back to the usual PowerPivot steps.

• Click on “Select from a list of tables and views to choose the data to import”

• Choose your table (e.g. hivesampletable) and import the data in.

It looks like a lot of steps but once you get into the flow of things, it’s actually a relatively easy flow.

Enjoy!

.

.

.

I love Pittsburgh, they put fries on nachos here.

– Pete Lattimer, Warehouse 13

.

.

.
For those un-familar with the reference, Warehouse 13 is an awesome Syfy show…and Costco is a warehouse store – yeah, weak connection here.

Yet another themed blog series

Starting with the recent Foodie Friday blog post (Foodie Friday: Taiwanese dessert 芋圓), figured I should add another non-geek themed blog post series – Travel Tuesday – probably every week or two, eh?!

Okay…so what’s this about Costco?

yeah right – so back to the title of this post – what are the top 3 reasons to go to Costco in Taiwan (from a US ex-pat).  The background is that my family and I are hanging out in Taiwan for the next few month – so lots of weird tips and foodie tips from Taiwan over the next few months, eh?!

5) Well, it’s Costco after all!

Where else can I buy enough toilet paper to survive the next ice age?  Or have such easy returns?  Or have actually good customer service?  But what’s great is that this is the same here in Taiwan too!

It’s still a huge warehouse with tonnes of the stuff that us ex-pats recognize but also plenty of stuff that’s made for the local market such as great Korean pears, Japanese quality fruit, etc.

4) OMG – Spacious Parking!!

If you drive around in Taiwan – almost anywhere in Taiwan that isn’t a highway – you are absolutely surrounded by a million scooters.  And if you don’t get a migraine from avoiding running over any of the scooters, parking in Taiwan…well, parking just sucks.  Small parking spaces, vehicles parking in places that just …aren’t parking spaces (e.g. in the middle of the road), …, ugh!

And at Costco – that’s just nice.  Wide lanes so two cars can actually comfortably fit, plenty of spots, and most importantly – spacious parking spots so I can actually park, get out of the car, and not worry that another car will trap me from getting into the car.

3) “Reminds me” of Seattle

And as many of you know, I’m based out of Seattle… and so is Costco! Costco’s home office is in Issaquah (suburb of Seattle). Costco’s Kirkland brand name is in homage to their original headquarters which was in Kirkland, WA (another suburb of Seattle).

So whenever we’re missing home – we just head off to Costco and we’re good to go!  Sort of reminiscent Garbage’s Only Happy When it Rains!

2) The ability bulk order scotch

Yeah, that’s just cool!  ‘nuff said!

…

But the primary reason you want to go to Costco when in Taiwan (from a US-expat)

1) Toilet Seat Covers!

I don’t think I need (nor do you want me) to explain this one!

Confucius says: Man who stand on toilet is high on pot!

When you hike up to Rattlesnake Ridge, you get a nice 270’ view of Rattlesnake Ridge area.  For more info, check out: ttp://www.wta.org/go-hiking/hikes/rattle-snake-ledge

—

About “Sunny Sunday”: The Sunny Sunday blog posts are photos from various travel and/or outdoor (hiking) trips.

To do this, let’s create a Hive table within Hadoop on Azure CTP that is connected to the files you uploaded to your Azure Blob storage account and query it.  We will be referencing the scenario noted at: Hadoop on Azure Scenario: Query a web log via HiveQL

The tasks we will be performing are:

1. Setup Azure Blob Store for Hadoop on Azure CTP
2. Create a Hive table referencing the files in the Azure Blob Storage account
3. Execute a simple query

1) Setup Azure Blob Store for Hadoop on Azure CTP

To do this, please refer to Setup Azure Blob Store for Hadoop on Azure CTP

.

2) Create a Hive table referencing the files in the Azure Blob Storage account

Following the Hadoop on Azure Scenario: Query a web log via HiveQL scenario

• Go to the Hadoop on Azure Interactive Hive Console
• Create a Hive table using the statement below

CREATE EXTERNAL TABLE weblog_sample_asv (
evtdate STRING,
evttime STRING,
svrsitename STRING,
svrip STRING,
csmethod STRING,
csuristem STRING,
csuriquery STRING,
svrport INT,
csusername STRING,
cip STRING,
UserAgent STRING,
Referer STRING,
scstatus STRING,
scsubstatus STRING,
scwin32status STRING,
scbytes STRING,
csbytes STRING,
timetaken STRING
)
COMMENT ‘This is a web log sample ASV’
ROW FORMAT DELIMITED FIELDS TERMINATED by ’32′
STORED AS TEXTFILE
LOCATION ‘asv://weblog/sample’;

Note that the only difference between the original HiveQL script (which goes to HDFS) and the one that goes to the Azure Blob storage is the highlighted LOCATION statement using the asv protocol.

NOTE: As noted in Setup Azure Blob Store for Hadoop on Azure CTP, we are using the protocol of asv://<container>/<folder> so that way its possible for Hadoop to view any and all files uploaded to the sample folder.

3. Execute a simple query

Now that you have created a Hive EXTERNAL table that points to the files located in the weblog/sample folder of your Azure Blob storage account, you can now query it.

The query below is the result from:

select * from weblog_sample_asv limit 10;

hadoop fs –ls /

In the case of accessing files within Azure Blob storage, you can run the command:

hadoop fs –ls asv://<container>/<folder>

The basic steps are:

1. Obtain the Azure Blobstore Storage Account Name and Access Key.
2. Set up ASV connection between Hadoop on Azure CTP and your Windows Azure Blob Storage account.
3. Upload files to your Azure Blob Storage account

1) Obtain the Azure Blobstore Storage Account Name and Access Key

Access your Azure Blobstore Storage account through the Windows Azure Platform dashboard via http://windows.azure.com/.  From here, the navigation path is [Hosted Services, Storage Accounts & CDN] (bottom left) –> [Storage Accounts] (mid-top left).

• The name blobstore account name is the Storage Account under the subscription as noted within the middle pane.  In this case, I have a storage account called isocatstore.
• To get the access key, click on the [View] button on the properties right pane after clicking on the storage account in question.

2) Set up ASV connection between Hadoop on Azure CTP and your Windows Azure Blob Storage account.

From the Hadoop on Azure CTP portal page, click on the [Manage Data] tile.  From here, click on the [Set up ASV] button on the right.

From here, you can supply the credentials of your Azure Blob Storage account that you had obtained in Step 1.

Click on [Save Settings] and you are good to go.

3) Upload files to your Azure Blob Storage account

A great way to upload files to your Azure Blob Storage account is to use CloudXplorer – you can download it from here: http://clumsyleaf.com/products/cloudxplorer

NOTE: When you upload the files, please ensure to place the files within a folder within a container of your blobstore account.  It is important to do this so that way Hadoop will be able to list all of the files within the folder instead of you needing to access each file individually (which is what would happen if you placed the files directly within the container).

From CloudXplorer, you can quickly create a container and a folder; in this case, I had created the weblog container and the sample folder.

Using the intuitive UI, copy your files from your local box to the Azure Blob Storage account.

By doing it in this fashion, you will be able to get a listing of your files from the Hadoop command line interface using the command:

hadoop fs –ls asv://weblog/sample

As well, from the Hadoop on Azure JavaScript Interface, you can view a listing of files using the command

#ls asv://weblog/sample