---
id: dev_quickstart_data_azure
title: Local Development Quickstart
linkTitle: 3. Local Development Quickstart
weight: 5
keywords:
  - quickstart
  - development
---

This section covers the steps required to start developing an Ensono Stacks Azure Data Platform from your machine:

1. Make sure you have installed the applications in link:./requirements_data_azure.adoc#local-development[local development requirements].
2. Ensure that Poetry is added to your `$PATH`.

== Environment setup

A Makefile has been created to assist with setting up the development environment. Run:

[source,bash]
----
make setup_dev_environment
----

=== (Optional) Azure connection

In order to interact with Azure resources when developing, including running end-to-end tests, you must:

. link:https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli[Sign in to the Azure CLI]
. Set the following environment variables:
    * `AZURE_SUBSCRIPTION_ID`
    * `AZURE_RESOURCE_GROUP_NAME`
    * `AZURE_DATA_FACTORY_NAME`
    * `AZURE_REGION_NAME`
    * `AZURE_STORAGE_ACCOUNT_NAME`
    * `AZURE_CONFIG_ACCOUNT_NAME`

== Running tests

`make` commands are provided to assist with running tests while developing locally. See link:../data_engineering/testing_data_azure.adoc[testing] for further details on these tests.

[TIP]
If you encounter PATH-related issues with Poetry when running the tests, we recommend installing Poetry using
link:https://python-poetry.org/docs/#installing-with-pipx[pipx] rather than the official installer.

=== Unit tests

In order to run unit tests, run the following command:

[source,bash]
----
make test
----

=== End-to-end tests

Running the end-to-end tests will involve executing Data Factory pipelines in Azure. Ensure you have set up the link:#optional-azure-connection[Azure connection] and run:

[source,bash]
----
make test_e2e
----

[TIP]
Running end-to-end tests from your local machine may require additional permissions in Azure. If the tests fail whilst clearing up directories, ensure that you have link:https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor[Storage Blob Data Contributor] access applied to your Azure Active Directory subscription. You may also be required to configure the link:https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security[firewall rules] for the storage account to whitelist your IP address.

=== Code quality checks

link:https://pre-commit.com/[Pre-commit] is used for code quality and linting checks on the project. It can be run using:

[source,bash]
----
make pre_commit
----

== (Optional) PySpark development in Databricks

[NOTE]
.PREREQUISITE
This sub-section assumes that link:./datastacks_deployment_azure.adoc[Datastacks build & deployment] has been completed - if you are working through the _getting started guide_ for the first time, you may skip this section.

When developing with PySpark, you may wish to either:

* Run scripts locally using a local Spark installation, or
* Run scripts on a Databricks cluster, through link:https://learn.microsoft.com/en-us/azure/databricks/repos/[Databricks Repos].

To run scripts within a Databricks cluster, you will need to:

* Install the Datastacks whl file on the cluster, either from:
    ** The latest deployed version in `dbfs:/FileStore/jars/datastacks-latest-py3-none-any.whl`, or
    ** Create a new whl file with the `poetry build` command.
* Add the additional link:../data_engineering/pyspark_utilities.adoc#prerequisites[environment variables] required for PySpark development - the values can be set as per the Data Factory linked service (see link:https://github.com/Ensono/stacks-azure-data/blob/main/de_workloads/shared_resources/data_factory/adf_linked_services.tf[adf_linked_services.tf]).
* Ensure the user has appropriate permissions for Azure resources required.

== Azure Data Factory Development

A core component of the Ensono Stacks Azure data platform is link:https://learn.microsoft.com/en-us/azure/data-factory/[Azure Data Factory], which is used for ingest activities, pipeline orchestration and scheduling. When an instance of Data Factory has been deployed, it's intuitive user interface can be used for reviewing, monitoring and editing resources.

While resources can be edited directly through the UI, the approach used in Stacks is to manage all resources through infrastructure-as-code using Terraform. The allows full CI/CD capabilities and control over changes across environments. Developers may use Data Factory's UI to assist in the development of new resources, and then transpose these into the project repository.

The following resource types will typically be added for new data workloads:

|===
| Resource type | Stacks workload types | Defined in | Notes

| Linked services | Ingest | `data_factory/adf_linked_services.tf` | Refer to Microsoft documentation for up-to-date details on link:https://learn.microsoft.com/en-us/azure/data-factory/connector-overview[connector types supported by Data Factory], and Terraform documentation for adding link:https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_factory_linked_custom_service[custom linked services]. Core linked services are added during deployment of link:./shared_resources_deployment_azure.adoc[shared resources].
| Datasets | Ingest | `data_factory/adf_datasets.tf` | Refer to Terraform documentation for adding link:https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_factory_custom_dataset[custom datasets]. Core datasets are added during deployment of link:./shared_resources_deployment_azure.adoc[shared resources].
| Pipelines | Ingest & Processing | `data_factory/adf_pipelines.tf` | Pipelines are deployed using the Terraform link:https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group_template_deployment[azurerm_resource_group_template_deployment] type. These refer to a JSON file containing the pipeline definition. The pipeline definition JSON can be obtained after creating pipelines interactively through the Data Factory UI. If editing a pipeline in the Data Factory UI, click the `{}` icon to view the underlying JSON - this can then be copied into the workload's JSON file in the project repo (under the `resources` element).
| Triggers | Ingest | `data_factory/adf_triggers.tf` | Refer to Terraform documentation for adding triggers, e.g. link:https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_factory_tumbling_window[tumbling window triggers].
|===

[TIP]
Changes to Data Factory resources directly through the UI will lead to them being overwritten when deployment pipelines are next run. Ensure updates are made within the project repository to ensure updates are not lost.

== Next steps

Once you set up your local development environment, you can continue with the Getting Started tutorial by link:./shared_resources_deployment_azure.adoc[deploying the shared resources].
