---
id: datastacks
title: Datastacks overview
linkTitle: Datastacks
weight: 2
keywords:
  - data
  - python
  - etl
  - cli
  - azure
  - template
---


Datastacks provides a suite of utilities built to accelerate development within an Ensono Stacks Data Platform, supporting common tasks such as generating new data engineering workloads and running Spark jobs. Datastacks consists of:

- link:./datastacks.adoc[Datastacks CLI] - A command-line interface for data engineers, enabling interaction with Datastacks' various functions.
- link:./generating-data-workloads.adoc[Data workload generation] - Generate new data workloads based upon common templates.
- link:./pyspark_utilities.adoc[PySpark utilities] - A suite of reusable utilities to simplify development of data pipelines using Apache Spark and Python.
- link:./data_quality_azure.adoc[Data quality utilities] - Utilities to support the data quality framework implemented in Stacks.
- Azure utilities - Utilities to support common interactions with Azure resources from data workloads.
- Behave utilities - Common scenarios and setup used by link:./testing_data_azure.adoc#end-to-end-tests[Behave end-to-end tests].

== Using the Datastacks CLI

- link:../getting_started/dev_quickstart_data_azure.adoc[Setup project environment]

[source,bash]
----
# Option 1: Run Datastacks CLI using Poetry's interactive shell (recommended for local development)
poetry shell
datastacks --help

# Option 2: Run Datastacks CLI using poetry run (recommended where Poetry shell cannot be used, e.g. CI/CD pipelines)
poetry run datastacks --help
----

== Generating data workloads

Datastacks can be used to generate all the resources required for a new data engineering workload - for example a link:./ingest_data_azure.adoc[data ingest] or link:./data_processing.adoc[data processing] pipeline. This will create all resources required for the workload, based upon templates (held within link:https://github.com/ensono/stacks-azure-data/tree/main/datastacks/datastacks/generate/templates[datastacks.generate.templates]).

The link:../architecture/architecture_data_azure.adoc#data-engineering-workloads[deployment architecture] section shows the workflow for using Datastacks to generate a new workload.
The link:../getting_started/getting_started.adoc[getting started] section includes step-by-step instructions on deploying a new ingest or processing workload using Datastacks.

=== Commands

* link:./datastacks.adoc#generate[generate]: Top-level command for generating resources for a new data workload.
    ** `ingest`: Subcommand to generate a new link:./ingest_data_azure.adoc[data ingest workload], using the provided configuration file. An optional flag (`--data-quality` or `-dq`) can be included to specify whether to include data quality components in the workload.
    ** `processing`: Subcommand to generate a new link:./data_processing.adoc[data processing workload], using the provided configuration file. An optional flag (`--data-quality` or `-dq`) can be included to specify whether to include data quality components in the workload.

=== Examples

[source,bash]
----
# Activate virtual environment
poetry shell

# Generate resources for an ingest workload
datastacks generate ingest --config="de_workloads/generate_examples/test_config_ingest.yaml"

# Generate resources for an ingest workload, with added data quality steps
datastacks generate ingest --config="de_workloads/generate_examples/test_config_ingest.yaml" --data-quality

# Generate resources for a processing workload
datastacks generate processing --config="de_workloads/generate_examples/test_config_processing.yaml"

# Generate resources for a processing workload, with added data quality steps
datastacks generate processing --config="de_workloads/generate_examples/test_config_processing.yaml" --data-quality
----

=== Configuration

In order to generate a new data engineering workload, the Datastacks CLI takes a path to a config file. This config file should be YAML format, and contain configuration values as specified in the table below. Sample config files are included in the link:https://github.com/ensono/stacks-azure-data/tree/main/de_workloads/generate_examples[de_workloads/generate_examples] folder. The structure of config is link:https://github.com/Ensono/stacks-azure-data/blob/main/datastacks/datastacks/pyspark/data_quality/config.py[validated using Pydantic].

==== All workloads

|===
| Config field | Description | Required? | Format | Default value | Example value

| pipeline_description | Description of the pipeline to be created. Will be used for the Data Factory pipeline description. | Yes | String | _n/a_ | "Ingest from demo Azure SQL database using ingest config file."
| ado_variable_groups_nonprod | List of required variable groups in non-production environment. | Yes | List[String] | _n/a_ | - amido-stacks-de-pipeline-nonprod<br />- stacks-credentials-nonprod-kv
| ado_variable_groups_prod | List of required variable groups in production environment. | Yes | List[String] | _n/a_ | - amido-stacks-de-pipeline-prod<br />- stacks-credentials-prod-kv
| default_arm_deployment_mode | Deployment mode for terraform. | No | String | "Incremental" | Incremental
|===

==== Ingest workloads

|===
| Config field | Description | Required? | Format | Default value | Example value

| dataset_name | Dataset name, used to derive pipeline and linked service names, e.g. AzureSql_Example. | Yes | String | _n/a_ | azure_sql_demo
| data_source_password_key_vault_secret_name | Secret name of the data source password in Key Vault. | Yes | String | _n/a_ | sql-password
| data_source_connection_string_variable_name | Variable name for the connection string. | Yes | String | _n/a_ | sql_connection
| data_source_type | Data source type. | Yes | String | _n/a_ | azure_sql
| bronze_container | Name of container for landing ingested data. | No | String | raw | raw
| key_vault_linked_service_name | Name of the Key Vault linked service in Data Factory. | No | String | ls_KeyVault | ls_KeyVault
| trigger_start | Start datetime for Data Factory pipeline trigger. | No | Datetime | _n/a_ | 2010-01-01T00:00:00Z
| trigger_end | Datetime to set as end time for pipeline trigger. | No | Datetime | _n/a_ | 2011-12-31T23:59:59Z
| trigger_frequency | Frequency for the Data Factory pipeline trigger. | No | String | "Monthly" | Monthly
| trigger_interval | Interval value for the Data Factory pipeline trigger. | No | Integer | 1 | 1
| trigger_delay | Delay between Data Factory pipeline triggers, formatted HH:mm:ss | No | String | "02:00:00" | 02:00:00
| window_start_default | Default window start date in the Data Factory pipeline. | No | Date | "2010-01-01" | 2010-01-01
| window_end_default | Default window end date in the Data Factory pipeline. | No | Date | "2010-01-31" | 2010-01-31
|===

==== Processing workloads

|===
| Config field | Description | Required? | Format | Default value | Example value

| pipeline_name | Name of the data pipeline / workload. | Yes | String | _n/a_ | processing_demo
|===

*Additional data source types will be supported in future.*
