---
id: pyspark_utilities
title: PySpark utilities
linkTitle: PySpark utilities
weight: 6
keywords:
  - pyspark
  - spark
  - pyspark
  - python
  - etl
---


The link:datastacks.adoc[**Datastacks**] Python library contains utilities built to streamline data processing workflows with Apache Spark and is an integral part of data workloads in Ensono Stacks.

Datastacks also provides the foundation for the link:./data_quality_azure.adoc[**Data Quality framework**] in Stacks.

== Prerequisites

To develop and test using PySpark with Datastacks, the following setup is required:

* Install the link:../getting_started/requirements_data_azure.adoc#local-development[**local development requirements**].
* link:../getting_started/dev_quickstart_data_azure.adoc[**Setup your local development environment**].

To interact with Azure resources, the following additional environment variables are currently required:

* `AZURE_TENANT_ID`: Directory ID for Azure Active Directory application.
* `AZURE_CLIENT_ID`: Application ID for Azure Active Directory application.
* `AZURE_CLIENT_SECRET`: Service Principal Secret.
* `ADLS_ACCOUNT`: ADLS account name.
* `BLOB_ACCOUNT`: Blob Storage account name.

These are defined in Databricks clusters - see link:../getting_started/dev_quickstart_data_azure.adoc#optional-pyspark-development-in-databricks[**development in Databricks**] for further details.
