Spark SQL is a Spark extension for structured data processing. Unlike the basic Spark RDD API, the API interfaces provided by Spark SQL provide the engine Spark with more insight into the internal structure of both the computation and data being processed. Spark SQL uses this insight to perform extra optimizations. There are several ways to interact with Spark SQL including SQL, Dataset, and even interoperability with RDD API.
Spark SQL may be used to execute SQL queries. This opens the door for those who already know SQL to apply it to Spark. When running SQL from within another programming language the results will be returned as a Dataset/DataFrame.
A Dataset is a distributed collection of data. A DataFrame is a Dataset organized into named columns. The Dataset interface was added in Spark 1.6. Dataset provides the benefits of RDDs such as strong typing and lambda functions with the benefits of an optimized execution engine. A DataFrame is a Dataset organized into named columns. Conceptually, a DataFrame is similar to a table in a relational database or a data frame in R/Python.
Spark SQL provides two different mechanisms for converting existing RDDs into Datasets. The first method uses reflection to infer the schema. The reflection approach can work well if the data source schema is known.
The second method for creating Datasets is through a programmatic interface that allows you to construct a schema and then apply it to an existing Spark RDD. While this method is more verbose, it allows you to construct Datasets when the columns and their types are not known until runtime.
In this tutorial, we will cover using Spark SQL with a mySQL database. Overview Let’s show examples of using Spark SQL mySQL. We’re going to use mySQL with Spark in this tutorial, but you can apply the concepts presented here to any relational database which has a JDBC driver. By the way, If you are […]
This tutorial covers using Spark SQL with a JSON file input data source in Scala. If you are interested in using Python instead, check out Spark SQL JSON in Python tutorial page. Spark SQL JSON Overview We will show examples of JSON as input source to Spark SQL’s SQLContext. This Spark SQL tutorial with JSON […]
In this Spark SQL tutorial, we will use Spark SQL with a CSV input data source. We will continue to use the baby names CSV source file as used in the previous What is Spark tutorial. This tutorial presumes the reader is familiar with using SQL with relational databases and would like to know how […]
