PySpark SQL is a subset of PySpark which provides an interface for Apache Spark in Python. PySpark supports most of Spark’s core features and extensions such as Spark SQL, DataFrame, Streaming, MLlib (Machine Learning).
Unlike the basic Spark RDD API, the interfaces provided by PySpark SQL provide more information both about the structure of the data and the compute type being performed. There are different ways to interact with Spark SQL including SQL and the Dataset API.
Python does not have the support for the Dataset API, but many of the benefits found within the Spark Dataset API are already available from Python. For example, Python developers can already access a field in a row by name; ie. row.columnName.
Because of the PySpark SQL aforementioned interfaces providing more insight into data structure and compute type, certain performance optimizations can be realized.
pyspark.sql.functions is a module in PySpark providing a collection of built-in functions for working with structured data in Spark. These functions can be used for data manipulation, aggregation, filtering, and other operations on Spark DataFrames and Datasets. Some examples of commonly used functions in pyspark.sql.functions include col(), lit(), concat(), substring(), when(), sum(), avg(), max(), min(), count(), and many more. These functions are designed to be highly optimized for distributed computing, making them efficient for working with large-scale datasets in Spark.
PySpark SQL tutorials are available below, but if you are a Python programmer coming to PySpark SQL from Pandas or NumPy, then you should familiarize yourself with Apache Arrow for performance reasons when converting a Spark DataFrame to a Pandas DataFrame and vice versa.
PySpark SQL requires transfer of data between JVM and Python processes. In recent versions of Apache Spark, this transfer implementation is through Apache Arrow.
Currently, this is most beneficial to Python users working with Pandas/NumPy data. Apache Arrow usage is not automatic and disabled by default.
To use Apache Arrow in PySpark, PyArrow should be installed. If you install PySpark using pip, then PyArrow can be brought in as an extra dependency of the SQL module with the command pip install pyspark[sql]
To use Arrow when executing conversions between Spark DataFrame and Pandas DataFrame and vice versa, set the Spark configuration `spark.sql.execution.arrow.pyspark.enabled` to `true`. This is disabled by default.
And now on to the PySpark SQL tutorials…
One of PySpark’s many strengths is its ability to handle JSON data. JSON, or JavaScript Object Notation, is a popular data format used for web applications and APIs. With PySpark, users can easily load, manipulate, and analyze JSON data in a distributed computing environment. This PySpark JSON tutorial will show numerous code examples of how … Read more
In order to use PySpark with MySQL, we must first establish a connection between the two systems. This can be done using a JDBC (Java Database Connectivity) driver, which allows PySpark to interact with MySQL and transfer data between the two systems. Once this connection is established, PySpark can extract data from MySQL, perform transformations … Read more
In this pyspark read csv tutorial, we will use Spark SQL with a CSV input data source using the Python API. We will continue to use the Uber CSV source file as used in the Getting Started with Spark and Python tutorial presented earlier. Also, this Spark SQL CSV tutorial assumes you are familiar with using … Read more
PySpark SQL functions are available for use in the SQL context of a PySpark application. These functions allow us to perform various data manipulation and analysis tasks such as filtering and aggregating data, performing inner and outer joins, and conducting basic data transformations in PySpark. PySpark functions and PySpark SQL functions are not the same … Read more
The PySpark withColumn function is used to add a new column to a PySpark DataFrame or to replace the values in an existing column. To execute the PySpark withColumn function you must supply two arguments. The first argument is the name of the new or existing column. The second argument is the desired value to … Read more
A PySpark UDF, or PySpark User Defined Function, is a powerful and flexible tool in PySpark. They allow users to define their own custom functions and then use them in PySpark operations. PySpark UDFs can provide a level of flexibility, customization, and control not possible with built-in PySpark SQL API functions. It can allow developers … Read more
In PySpark, the DataFrame filter function, filters data together based on specified columns. For example, with a DataFrame containing website click data, we may wish to group together all the platform values contained a certain column. This would allow us to determine the most popular browser type used in website requests. Solutions like this may … Read more
In PySpark, the DataFrame groupBy function, groups data together based on specified columns, so aggregations can be run on the collected groups. For example, with a DataFrame containing website click data, we may wish to group together all the browser type values contained a certain column, and then determine an overall count by each browser … Read more
Use PySpark joins with SQL to compare, and possibly combine, data from two or more datasources based on matching field values. This is simply called “joins” in many cases and usually the datasources are tables from a database or flat file sources, but more often than not, the data sources are becoming Kafka topics. Regardless … Read more
PySpark joins are used to combine data from two or more DataFrames based on a common field between them. There are many different types of joins. The specific join type used is usually based on the business use case as well as most optimal for performance. Joins can be an expensive operation in distributed systems … Read more
