For the 2012 Hadoop Summit, I will have the honor to co-present with Dave Mariani (@dmariani) from Klout in our session How Klout is changing the landscape of social media with Hadoop and BI.

Our session is currently scheduled for June 13th at 3:35pm but it is subject to change.  Check out our session and some pretty amazing others on the Hadoop Summit Schedule at: http://hadoopsummit.org/schedule/.  Our session info is:

—

How Klout is changing the landscape of social media with Hadoop and BI.

In this age of Big Data, data volumes grow exceedingly larger while the technical problems and business scenarios become more complex. Compounding these complexities, data consumers are demanding faster analysis to common business questions asked of their Big Data. This session provides concrete examples of how to address this challenge. We will highlight the use of Big Data technologies—including Hadoop and Hive —with classic BI systems such as SQL Server Analysis Services.

Session takeaways:

• Understand the architectural components surrounding Hadoop, Hive, Classic BI, and the Tier-1 BI ecosystem
• Get strategies for addressing the technical issues when working with extremely large cubes
• See how to address the technical issues when working with Big Data systems from the DBA perspective

–

And for fun, check out my and Dave’s interview at: Meet the Presenters: Dave Mariani from Klout and Denny Lee from Microsoft

Personally, I have to say its been pretty awesome ride this last year and a half – and its only going to get more exciting!

Woot!

References:

• Installing Hadoop on Mac OSX Lion by Ritesh Agrawal
• Flume and Hadoop on OSX by Arbo v. Monkiewitsch (@arbovm)

In fact, originally I was just going to provide links to their blog posts but ran into some hiccups along the way.  To make it easier to read, I’m going to call out all the steps (hopefully!)

Install the OSX Lion Prerequisites

Ensure that you have installed – as of this post -

• XCode 4.3.2
• Java Developer for 10.7

You can find both updates at https://developer.apple.com/downloads/index.action

Make sure to update XCode (Preferences > Downloads) so that the Command Line Tools are also installed (requirement for HomeBrew)

Install Homebrew

If you are not familiar with Homebrew, you are definitely missing the package manager for OSX (it’s their tag line, but it’s also very true).  Homebrew will install packages into their own folders and use symlinks back to the /usr/local folder.  This will allow for much easier removal and isolation of UNIX packages – such as Hadoop.  As well, Homebrew will not require you to use sudo to perform a brew install.

Install Homebrew by going to the Homebrew installation link and running the provided ruby script.  In this case, please go to the link so that way if they change it, these instructions are more or less still up to date.

After installing Homebrew, from the Terminal.app, ensure you run the following:

• brew doctor – This validates that the installation is okay.  Sometimes it will catch errors (e.g. XCode command line paths are not as expected)
• brew update – To ensure that the latest Formulas are available and installed in the /usr/local/Library/Formula folder.  Formulas are Ruby scripts that define the installation of a package, you can find the list in the folder or online at Homebrew Library Formula; to make your own, check out the Formula Cookbook.

Installing Hadoop 1.0.1

If all is well, the only thing you have to type into the Terminal.app to install Hadoop is to run:

brew install hadoop

Error: Cannot allocate memory – connect(2)

Yet, one of the reasons I’m writing this post is because I ran into the following error (hopefully you don’t):

Error: Cannot allocate memory – connect (2)

Looking at the above URL, the http://www.apache.org/dyn/closer.cgi?path=hadoop/core/hadoop-1.0.1/hadoop-1.0.1.tar.gz, it points to the page:

i.e. its in fact a redirection error not a memory allocation error.

To fix it immediately, you will need to update the Hadoop Formula locally which can be found in the /usr/local/Library/Formulas/hadoop.rb.  Update the link so it points to the mirror site instead of this path.

Once you update the hadoop.rb file, you can then go back and re-run

brew install hadoop

and the installation will proceed.

Configure Hadoop

Now you will need to update the Hadoop configuration files to get everything up and running.  The Hadoop configuration files can all be found in the /usr/local/Cellar/hadoop/1.0.1/libexec/conf folder.  The files to be updated are:

• hadoop-env.sh: These are the Hadoop environment variables, a quick change is suggested to suppress an error message.
• core-site.xml
• hdfs-site.xml
• mapred-site.xml

And thanks to @arbovm for his details at: Flume and Hadoop on OS X.

hadoop-env.sh

Suppress the “Unable to load realm info from SCDynamicStore” error.  As noted in https://issues.apache.org/jira/browse/HADOOP-7489, for a single box deployment, update the text to include the line below.

export HADOOP_OPTS=”-Djava.security.krb5.realm= -Djava.security.krb5.kdc=”

It should look something like this

core-site.xml

The core-site.xml file should look this

In this particular case, I have created the Hadoop temporary folder in the /usr/local/ folders as well (this way you do not need sudo to create the HDFS folders).  To match the above configuration, please execute the following commands

hdfs-site.xml

Below is a quick suggested configuration for the hdfs-site.xml file.  In this case, the configuration is to set the Hadoop Distributed File System (HDFS) replication to 1 as this is a single box for demos and functional test.

mapred-site.xml

The below configuration is to set the Job tracker connection port.

Enable SSH to localhost

Please note that Hadoop will connect to localhost using ssh; to configure it so that way you can connect from-and-to localhost without needing a password, you will need public keys to your authorized keys.  To do this ensure that “Remote Login” is enabled (System Preferences > Sharing; ensure “Remote Login” is checked).

If you haven’t already done so, create your own public keys following the instructions at:
http://stackoverflow.com/questions/7134535/setup-passphraseless-ssh-to-localhost-on-os-x

ssh-keygen -t dsa -P ” -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Test it by running the command

ssh localhost

Running Hadoop

To get yourself off the ground, go ahead and run the following commands:

Format the Hadoop Namenode using:
hadoop namenode -format

Start Hadoop by running the script:
/usr/local/Cellar/hadoop/1.0.1/libexec/bin/start-all.sh

Run some quick tests
hadoop dfs -ls /

cd /usr/local/Cellar/hadoop/1.0.1/libexec
hadoop jar hadoop-examples-1.0.1.jar pi 10 100

Don’t forget you can access the Map Reduce Admin and HDFS Admin through the web browser

• http://locahost:50030 – Map Reduce Administrator
• http://localhost:50070 – HDFS Administrator

To Stop Hadoop
/usr/local/Cellar/hadoop/1.0.1/libexec/bin/stop-all.sh

Enjoy!

Enjoy!

Note, as well, there is a great end-to-end Microsoft Hadoop on Azure and Windows presentation available at:

.

Why Report Builder 3.0 when there is Power View?

Nevertheless, there are some situations where SQL Server 2008 R2 tool – Report Builder 3.0 – can still be quite helpful. For example, if you need to export out the values from your table or overlay the report on top of a map, you can do this in Report Builder 3.0 (but not in Power View – at least not yet).

.

Changes for SQL Server 2012

For SQL Server 2008 R2, to install Report Builder 3.0 with SharePoint 2010, there were a number of manual steps on how to do this.  For more information, check out: PowerPivot for SharePoint: Single Server Installation.  The good news is that the installation is easier for SQL Server 2012 because Report Builder is a stand-alone install package that does not require server.

But, it is a little more complicated to make a connection to a PowerPivot workbook from Report Builder 3.0 as the “create the report” option is for Power View (the middle icon, left is the Excel report, right is the Data Refresh).

.

So how do I connect Report Builder 3.0 to a SQL Server 2012 PowerPivot workbook?

Software Involved:

• SQL Server 2012
• SharePoint 2010 Enterprise SP1
• Report Builder 3.0 (SQL Server 2012 Reporting Services version)
• PowerPivot V2 (SQL Server 2012 Analysis Services version)

.
1) For starters, download and install the SQL Server 2012 Reporting Services version of Report Builder 3.0. http://www.microsoft.com/download/en/details.aspx?id=29072

.
2) Configure SharePoint 2010 to allow the management of content types (in some cases, this is already configured).

• Using your browser, go to the SharePoint library that you want to configure (e.g. PowerPivot Gallery)
• Go to the SharePoint 2010 Library ribbon and click on Library Settings

• Click on Advanced Settings which is under the General Settings frame
• Under Content Types, ensure that the Allow management of content types is set to Yes.

.
3) Include Report Data Source is included in the available Content Types.

• From the SharePoint 2010 Library ribbon, click on Library Settings in the same SharePoint library (e.g. PowerPivot Gallery)
• Under the Content Types frame, click on Add from existing site content types. The Select Content Types dialog appears, add the Report Data Source (scroll down to it in the “Available Site Content Types”) and click Add.

• When you are done, under the Content Types frame, you should see the Report Data Source now available.

.
4) Now that you have configured the availability of the Report Data Source content type, go back to the SharePoint 2010 Library ribbon, click on Documents, and then click on New Document.  Now you can click on Report Data Source.

.
5) The new document page (for a data source) will appear as noted below.  They data source parameters to configure are:

Data Source Type: Since you are connecting to a SQL Server 2012 PowerPivot workbook (that you have already uploaded to SharePoint library), the data source type is Microsoft SQL Server Analysis Services.

The Connection String is in the format of:

Data Source=http://server/PowerPivot%20Gallery/PowerPivot%20Workbook.xlsx.

Note the use of %20 to replace any spaces.  In the example below, the connection string is:

Data Source=http://intranet.contoso.com/SelfService%20BI/PowerPivot%20Mobile%20Hive%20Sample.xlsx.

.
6) Now that you have created a Report Data Source, you can now connect Report Builder 3.0 to it.

• Open up Report Builder 3.0 (from the Start menu) and it will give you the Getting Started dialog

• Click on you’re the Wizard of your choice to start creating your report.  With a new table or matrix, Report Builder will first ask for you to create a dataset or choose an existing dataset.  Since it’s your first time, choose the default of create a dataset and click Next.
• Because you had already created a Report Data Source in Step 5, click on Browser, and go to the location of your report data source (e.g. http://server/PowerPivot%20Gallery).  Click to open it.

• Now that you’ve chosen your data source

• Click Next and you’re back to familiar territory of Designing your Query in Report Builder 3.0.  To know more about how to work with Report Builder, check out the primer Getting Started with Report Builder 3.0.

.

Discussion

From the Report Builder application, you can create maps, charts, and/or tables which can also be exported as noted in the screenshot below.

Now that I have built my Report Builder 3.0 report, I can upload it to SharePoint (like any other document).   When you click to Save the report, by default it will connect to the SharePoint farm you were originally connected to.

In this case, I have saved the report in my PowerPivot Gallery; when I go back to it, I can access the Report Builder report I had just created.  And just like in the stand-alone application, from SharePoint I can export the table values of my Report Builder report.

.

Final

I hope this How to guide has been helpful on how to connect Report Builder 3.0 to a SQL Server 2012 PowerPivot workbook.  And thanks to Robert Bruckner for setting me straight on some of this stuff, eh?!

Enjoy!
.

Reference documents

• Add and Verify a Data Connection or Data Source (Report Builder and SSRS)
• Create and Manage Shared Data Sources (Reporting Services in SharePoint Integrated Mode)
• Add Report Server Content Types to a Library (Reporting Services in SharePoint Integrated Mode)
• Getting Started with Report Builder 3.0

1) Log into your Amazon AWS Account  and click onto Security Credentials

2) Obtain your access credentials – you’ll need both your Access Key ID and Secret Access Key.

3) From here, log into your Hadoop on Azure account, click the Manage Cluster live tile, and click on Set up S3.  From here, enter your Access Key and Secret Key and click Save Settings.

4) Once you have successfully saved your Amazon S3 settings, you can access your Amazon S3 files from Hadoop on Azure.  For example, I have a bucket called tardis6 with folder weblog with a sample weblog file.

To access this file, you can use the s3n protocol from Hadoop on Azure.  For example, click on the Interactive JavaScript console and type in the command:

#ls s3n://tardis6/weblog

Enjoy!!

These great pictures were taken by Kelly Hedlund; she, Brian Hedlund, and I went up to Mount Rainier in July 2000.

—

About “Sunny Sunday”: The Sunny Sunday blog posts are photos from various travel and/or outdoor (hiking) trips.

As noticed by GigaOM’s article Microsoft’s Hadoop play is shaping up, and it includes Excel; the great call out is:

to make Hadoop data analyzable via both a JavaScript framework and Microsoft Excel, meaning many millions of developers and business users will be able to work with Hadoop data using their favorite tools.

 .

Big Data for Everyone!

The title of the Microsoft BI blog post says it the best: Big Data for Everyone: Using Microsoft’s Familiar BI Tools with Hadoop – it’s about helping make Big Data accessible to everyone by use of one of the most popular and powerful BI tools – Excel.

So what does accessible to everyone mean – in the BI sense?  It’s about being to go from this (which is a pretty nice view of Hive query against Hadoop on Azure Hive Console)

and getting it Excel or PowerPivot.

The most important call out here is that you can use PowerPivot and Excel to merge data sets not just from Hadoop, but also bring in data sets from SQL Server, SQL Azure, PDW Oracle, Teradata, Reports, Atom feeds, Text files, other Excel files, and via ODBC – all within Excel! (thanks @sqlgal for that reminder!)

From here users can manipulate the data using Excel macros and PowerPivot DAX language respectively.  Below is a screenshot of data extracted from Hive and placed into PowerPivot for Excel.

But even more cooler – data visualization wise – your PowerPivot for Excel workbook (once uploaded to SharePoint 2010 with SQL Server 2012) and you can create an interactive Power View report.

For more information on how to get PowerPivot and Power View to connect to Hadoop (in this case, its Hadoop on Azure but conceptually they are the same), please reference the links below:

• How To Connect Excel to Hadoop on Azure via HiveODBC
• Connecting PowerPivot to Hadoop on Azure – Self Service BI to Big Data in the Cloud
• Connecting Power View to Hadoop on Azure
• Connecting Power View to Hadoop on Azure [Video]

 .

So what’s so Big about Big Data?

As noted by in the post What’s so Big about Big Data?, we call out that Big Data is important because of the sheer amount of machine generated data that needs to be made sense of.

As noted by Alexander Stojanovic (@stojanovic), the Founder and General Manager of Hadoop on Windows and Azure:

It’s not just your “Big Data” problems, it’s about your BIG “Data Problems”

 .

To learn more, check out the my 24HOP (24 Hours of PASS) session:

Tier-1 BI in the Age of Bees and Elephants

In this age of Big Data, data volumes become exceedingly larger while the technical problems and business scenarios become more complex.  This session dives provides concrete examples of how these can be solved. Highlighted will be the use of Big Data technologies including Hadoop (elephants) and Hive (bees) with Analysis Services.  Customer examples including Klout and Yahoo! (with their 24TB cube) will highlight both the complexities and solutions to these problems.
 .

Making this real, a great case study showcasing this includes the one at Klout, which includes a great blog post: Big Data, Bigger Brains.   And below is a link to Bruno Aziza (@brunoaziza) and Dave Mariani’s (@dmariani) YouTube video on how Klout Leverages Hadoop and Microsoft BI Technologies To Manage Big Data.

Enjoy!

–

Disclaimer: This blog post (like other blog posts on dennyglee.com) are written by the author Denny Lee. I am a Microsoft employee but the opinions below are my own. I have been working with the Isotope team (code name for Hadoop on Windows and Hadoop on Azure) since its inception while part of the SQL Customer Advisory Team.

Why is this important and cool at the same time?  As one can note with amazing projects like node.js, JavaScript is being seen by many as its own first class programming / application language: The rise of Node.js: JavaScript graduates to the server.

In the realm of Big Data, Hadoop on Azure is showcasing the ability to use JavaScript to create MapReduce jobs as well as interact with Pig and Hive from a browser.  This opens up the possibility of a new path for the many JavaScript developers to jump onboard into the world of Big Data.  Hence my opinion:

Hadoop JavaScript – Microsoft’s VB shift for Big Data

Just like Microsoft brought VB developers into the Enterprise by COM and later .NET (so VB forms creators could become Enterprise application developers back in the 90s), Hadoop JavaScript is a way to help bring JavaScript developers into the world of Hadoop using their own already powerful skillset.

And yes, the JavaScript layer is something Microsoft intends to give back to the Apache community.  Check out the jira HADOOP-8079: Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments.

To know more about the Hadoop + JavaScript, check out the links below as well as the Introduction to the Hadoop on Azure Interactive JavaScript Console video.

• Fluent Queries on the Interactive JavaScript Console
• Interactive JavaScript console on MDH
• Data Visualization on the Interactive JavaScript Console
• HDFS Operations on the Interactive JavaScript Console
• Interactive JavaScript Console Session Management

As well, if you’re going to 2012 Strata Conference in Santa Clara, check out Asad Khan’s session: Hadoop + JavaScript: what we learned.

Enjoy!

—-

Disclaimer: This blog post (like other blog posts on dennyglee.com) are written by the author Denny Lee. I am a Microsoft employee but the opinions below are my own. I have been working with the Isotope team (code name for Hadoop on Windows and Hadoop on Azure) since its inception while part of the SQL Customer Advisory Team.

Above is a screenshot of the Power View Mobile Hive Sample that is built on top of the PowerPivot workbook created in the Connecting PowerPivot to Hadoop on Azure blog post.  But taking a different medium, the steps to create a Power View report with Hadoop on Azure source can be seen in the YouTube video below.

Power View Report to Hadoop on Azure

Enjoy!