Introduction to Apache Spark 2.0

A Primer on Spark 2.0 Fundamentals and Architecture I’m proud to post that my O’Reilly video series Introduction to Apache Spark 2.0: A Primer on Spark 2.0 Fundamentals and Architecture is now available on O’Reilly Safari (Start your ten-day free trial) or you can purchase the video series directly. This video series highlights what’s new in Apache 2.0 and reviews its core concepts. The course starts with a high-level overview of Spark’s components and then dives into Spark 2.0’s three main themes: simplicity, speed, and intelligence. The simplicity section describes how Spark 2.0 unifies the Spark APIs and Spark session,…

Rate this:

Jump Start into Python and Apache Spark with Learning PySpark

For the last few years, I have had the opportunity to work on some of the coolest Apache Spark committers, contributors, and projects.  As luck would have it, I got the opportunity to meet my co-author Tomasz Drabas (author of the awesome Practical Data Analysis Cookbook) while we were solving some other cool Apache Spark projects.  In the process, we joined forces to share our lessons learned that will hopefully help you jump start your Python and Apache Spark projects with our book: Learning PySpark. And just to make sure, this book was reviewed by the incomparable Holden Karau, author of the…

Rate this:

On-Time Flight Performance with GraphFrames for Apache Spark

Feature Image: NASA Goddard Space Flight Center: City Lights of the United States 2012 This is an abridged version of the full blog post On-Time Flight Performance with GraphFrames. You can also reference the webinar GraphFrames: DataFrame-based graphs for Apache Spark and the On-Time Flight Performance with GraphFrames for Apache Spark notebook. An intuitive approach to understanding flight departure delays is to use graph structures. Why Graph? The reason for using graph structures is because it is a more intuitive approach to many classes of data problems: social networks, restaurant recommendations, or flight paths.  It is easier to understand these data problems…

Rate this:

Presentation: Jump Start into Apache® Spark™ and Databricks

These are the slides from the Jump Start into Apache Spark and Databricks webinar on February 10th, 2016. — Apache Spark is a fast, easy to use, and unified engine that allows you to solve many Data Sciences and Big Data (and many not-so-Big Data) scenarios easily. Spark comes packaged with higher-level libraries, including support for SQL queries, streaming data, machine learning, and graph processing. We will leverage Databricks to quickly and easily demonstrate, visualize, and debug our code samples; the notebooks will be available for you to download. You can view the on-demand webinar Jump Start into Apache® Spark™…

Rate this:

Data Exploration with Databricks

Today, it was also featured on InsideBigData: Data Exploration with Databricks.  Awesome!   This Data Exploration on Databricks jump start video will show you how go from data source to visualization in a few easy steps. Specifically, we will take semi-structured logs, easily extract and transform them, analyze and visualize the data using Spark SQL, so we can quickly understand our data. For more information and to check out other Spark notebooks, check out Selected Notebooks > Databricks Jump Start.  

Rate this:

Apache Spark is the Smartphone of Big Data

Similar to the way the smartphone changed the way we communicate – far beyond its original goal of mobile voice telephony – Apache Spark is revolutionizing Big Data. While portability may have been the catalyst of the mobile revolution, it was the ability to have one device perform multiple tasks very well with the ability to easily build and use a diverse range of applications that are the keys to its ubiquity. Ultimately, with the smartphone we have a general platform that has changed the way we communicate, socialize, work, and play. The smartphone has not only replaced older technologies…

Rate this:

Interested in career in Data Sciences? Read Freakonomics first!

Over the last few years, a common question I’ve been asked is what does it take to become a data scientist?  Often my answers surrounded the technology – i.e. learn Spark, Python, and/or R; take courses in Data Sciences; play with data sets; etc.   Yet, I was never fully satisfied with that answer because I had always felt that the heart of Data Sciences (and Big Data in more generic terms) is the data – or more specifically, the ability to understand the data. Recently, I re-read “Freakonomics: A Rogue Economist Explores the Hidden Side of Everything” and it dawned…

Rate this: