---
id: data_processing
title: Data Processing Workloads
linkTitle: Data Processing Workloads
weight: 4
keywords:
  - silver
  - adf
  - etl
  - databricks
---

Data processing workloads in Ensono Stacks are jobs which:

1. Take data in the data lake as input (this can be various formats e.g. CSV, Parquet, JSON, Delta).
2. Perform some form of data transformation / cleansing / modelling / aggregation over the data.
3. Output the into the data lake into a structured link:https://delta.io/[Delta Lake] format.

Data processing workloads are based upon running Apache Spark / Python jobs on Databricks. These workloads may be used for various levels of data transformation and preparation within the data lake. Within the link:./data_engineering_intro_azure.adoc#medallion-architecture[medallion architecture] these will include:

- Bronze to Silver
- Silver to Gold

Processing workloads will utilize link:./pyspark_utilities.adoc[Datastacks' PySpark utilities], to support and standardize common tasks. Similar to ingest workloads, data processing workloads can also optionally include a link:./data_quality_azure.adoc[Data Quality validations] step.

The following example data processing workloads are included for reference:

- link:https://github.com/Ensono/stacks-azure-data/tree/main/de_workloads/processing/silver_movies_example[silver_movies_example]: Performs Bronze to Silver layer data processing over the link:../getting_started/example_data_source.adoc[example dataset].
- link:https://github.com/Ensono/stacks-azure-data/tree/main/de_workloads/processing/gold_movies_example[gold_movies_example]: Aggregates and joins data from the Silver layer example above to create a Gold layer dataset.

== Data processing pipeline overview

Within Stacks, processing activities are performed using Python PySpark jobs. These jobs are orchestrated by pipelines in Data Factory, and executed in Databricks. Using PySpark jobs - as opposed to notebooks - gives full control over the processing activities (for example ensuring thorough link:./testing_data_azure.adoc[test coverage] and quality control).

image::../../../../../images/ADF_SilverGoldPipelineDesign.png[ADF_SilverGoldPipelineDesign.png]

The Python PySpark script executed as part of a data processing data workload is kept inside the `spark_jobs` directory for the workload. This job will utilize the link:./datastacks.adoc[Datastacks library], which provides a wealth of reusable utilities to assist with data transformations and loading data from/into to the data lake.

=== Data Factory pipeline design

Within Data Factory, the processing pipelines are kept within the `Process` folder:

image::../../../../../images/ADF_SilverPipelinesList.png[ADF_SilverPipelinesList.png]

In Data Factory a data processing pipeline can be as simple as this example:

image::../../../../../images/ADF_silver.png[ADF_Silver.png]

It contains just one step - Python Databricks, configured to run a `silver.py` script, which gets deployed to DBFS (`dbfs:/FileStore/scripts/silver/silver.py`). The Datastacks package and library is deployed to DBFS, and made available to the cluster.

=== Passing parameters from Data Factory

You may pass parameters from Data Factory to the Python job executed in Databricks. These are defined in the Settings section of the Databricks Python activity:

image:../../../../../images/adf-databricks-parameters.png[adf-databricks-parameters.png]

These parameters can then be accessed from the Python file, by using the `get_data_factory_param` Datastacks function - for example the below could be used to access the first parameter shown above:

[source,python]
----
from datastacks.pyspark.etl import get_data_factory_param

run_id = get_data_factory_param(1, "default_run_id")
----