Data Engineering Reading Materials: Spark, Machine Learning, and Distributed Systems Resources

Over the last few weeks, a regular question that I’ve been asked are where I can find resources about Spark, Machine Learning, and Distributed Systems.  While they seem to be disparate problems, the fact is that as a Data Engineer (or someone in Data Sciences Engineering or a Data Scientist that loves scalability and performance) you need to have your feet wet in all three disciplines to truly excel. Apache Spark Let’s start with Apache Spark (disclosure, I am with Databricks – the company was founded by the creators of Apache Spark).   I am a big fan of Apache Spark because of its…

Rate this:

Simplify Machine Learning on Spark with Databricks

As many data scientists and engineers can attest, the majority of the time is spent not on the models themselves but on the supporting infrastructure.  Key issues include on the ability to easily visualize, share, deploy, and schedule jobs.  More disconcerting is the need for data engineers to re-implement the models developed by data scientists for production.  With Databricks, data scientists and engineers can simplify these logistical issues and spend more of their time focusing on their data problems. Simplify Visualization An important perspective for data scientists and engineers is the ability to quickly visualize the data and the model…

Rate this:

Notebook Gallery

Here are some of the notebooks created to showcase various Apache Spark use cases. These are all using Databricks Community Edition which you can get at Try Databricks. You can also access the source from : https://github.com/dennyglee/databricks. JSON Support GLM in SparkR Window Functions  Random Forests DataFrame API ML Operations   Decision Trees Statistical Functions  Data Import  Data Exploration Quick Start Python Quick Start Scala  Ad-Tech Example Flight Delays  Genomics Mobile Sample   Pop vs. Price LR  Pop vs. Price DF  Salesforce Leads Spark 1.6 (Multiple)   Spark 1.6  

Rate this:

Join the WearHacks experience!

If you are interested in learning more about wearables, check out WearHacks.com – one of the fastest growing communities empowering hackers on wearable technology.   Last year, WearHacks Montréal was the biggest wearable hackathon in North America.  In 2015, there are already 13 cities hosting hackathons with mentors, industry experts, and access to technology so you can build new wearable and connected technologies. Below is an impressive infographic on how much WearHacks has grown in the last year. Check out the full infographic at: https://magic.piktochart.com/output/5369103-wearhacks_-2014-vs-2015#

Rate this:

Spark Ecosystem & Spark Streaming Fundamentals

This is a re-post from Spark Ecosystem & Spark Streaming Fundamentals post on the Concur blog. For the March 2015 Seattle Spark Meetup group convened for our monthly meeting. It was a special meeting since it was actually a joint Seattle Spark Meetup and Pacific Northwest Cloudera User Group session held at the Concur Technologies headquarters in Bellevue. We had special guest speakers, Mike Olson (@mikeolson) and Hari Shreedharan (@harisr1234), whose presentations discussed the basics of Spark and Spark Streaming as well as the Spark ecosystem, specifically, Cloudera’s investments into it. For those who missed it, we’re sharing Mike and…

Rate this:

Presentation: Concur Discovers the True Value of Data

Concur, the leading provider of spend management solutions and services, will be joining us to discuss how they implemented Cloudera for data discovery and analytics. Using an enterprise data hub, Concur was able to provide their data scientists a centralized environment that allowed for faster and smarter analytic development.  

Rate this:

Feb Spark Events: Data Discovery, Dato & Spark, and Spark Camp

An awesomely busy February coming up for those whom are interested in all things Apache Spark! Concur Discovers the True Value of Data A joint Cloudera and Concur webinar on February 2nd, 2015 where we will discuss the benefits of utilizing CDH5 within Concur’s modern Big Data architecture (including Spark of course!) Better Together: Dato + GraphLab We’ve got a great Seattle Spark Meetup event feature three speakers highlighting the integration between Dato’s GraphLab Create and Apache Spark.   Come join us at Concur’s new Bellevue Training Center to learn more about GraphLab Create, Apache Spark, and network. Strata +…

Rate this: