learning-pyspark-banner-final

Jump Start into Python and Apache Spark with Learning PySpark

For the last few years, I have had the opportunity to work on some of the coolest Apache Spark committers, contributors, and projects.  As luck would have it, I got the opportunity to meet my co-author Tomasz Drabas (author of the awesome Practical Data Analysis Cookbook) while we were solving some other cool Apache Spark projects.  In the process, we joined forces to share our lessons learned that will hopefully help you jump start your Python and Apache Spark projects with our book: Learning PySpark. And just to make sure, this book was reviewed by the incomparable Holden Karau, author of the…

Rate this:

NASA image acquired April 18 - October 23, 2012

This image of the United States of America at night is a composite assembled from data acquired by the Suomi NPP satellite in April and October 2012. The image was made possible by the new satellite’s “day-night band” of the Visible Infrared Imaging Radiometer Suite (VIIRS), which detects light in a range of wavelengths from green to near-infrared and uses filtering techniques to observe dim signals such as city lights, gas flares, auroras, wildfires, and reflected moonlight.

“Nighttime light is the most interesting data that I’ve had a chance to work with,” says Chris Elvidge, who leads the Earth Observation Group at NOAA’s National Geophysical Data Center. “I’m always amazed at what city light images show us about human activity.” His research group has been approached by scientists seeking to model the distribution of carbon dioxide emissions from fossil fuels and to monitor the activity of commercial fishing fleets. Biologists have examined how urban growth has fragmented animal habitat. Elvidge even learned once of a study of dictatorships in various parts of the world and how nighttime lights had a tendency to expand in the dictator’s hometown or province.

Named for satellite meteorology pioneer Verner Suomi, NPP flies over any given point on Earth's surface twice each day at roughly 1:30 a.m. and p.m. The polar-orbiting satellite flies 824 kilometers (512 miles) above the surface, sending its data once per orbit to a ground station in Svalbard, Norway, and continuously to local direct broadcast users distributed around the world. Suomi NPP is managed by NASA with operational support from NOAA and its Joint Polar Satellite System, which manages the satellite's ground system.

NASA Earth Observatory image by Robert Simmon, using Suomi NPP VIIRS data provided courtesy of Chris Elvidge (NOAA National Geophysical Data Center). Suomi NPP is the result of a partnership between NASA, NOAA, and t

On-Time Flight Performance with GraphFrames for Apache Spark

Feature Image: NASA Goddard Space Flight Center: City Lights of the United States 2012 This is an abridged version of the full blog post On-Time Flight Performance with GraphFrames. You can also reference the webinar GraphFrames: DataFrame-based graphs for Apache Spark and the On-Time Flight Performance with GraphFrames for Apache Spark notebook. An intuitive approach to understanding flight departure delays is to use graph structures. Why Graph? The reason for using graph structures is because it is a more intuitive approach to many classes of data problems: social networks, restaurant recommendations, or flight paths.  It is easier to understand these data problems…

Rate this: