---
id: data_engineering_intro_azure
title: Data Engineering Overview
linkTitle: Data Engineering Overview
weight: 1
keywords:
  - ingest
  - silver
  - gold
  - bronze
  - databricks
  - etl
  - medallion
---

Ensono Stacks accelerates the generation of production-ready data engineering workloads and pipelines for a data lakehouse. New data engineering workloads can be generated through the link:./datastacks.adoc[Datastacks] utility, while a range of example data workloads and pipelines are also provided. These workloads cover all stages from ingesting data from source, applying data transformations across data lake layers, ultimately enabling end-user data visualisations and analytics.

## Lakehouse approach

The Ensono Stacks data platform is based upon a _Lakehouse_ architecture. This approach combines the benefits of both data warehouses and data lakes, to provide a platform that is fully scalable, flexible and performant, along with governance and management capabilities. It can support all use cases, from complex machine learning to standard BI reporting and analysis.

== Medallion architecture

The data lake structure and approach used in Ensono Stacks is based upon the link:https://www.databricks.com/glossary/medallion-architecture[medallion architecture] design pattern. This approach delivers a lakehouse solution, with the following structured data transformation layers:

|===
| Data lake layer | Description | Default container name | Data format | Stacks workload type

| Bronze | The initial landing area where data is stored as per its original source, prior to any transformations. | raw | Parquet | link:./ingest_data_azure.adoc[Data ingest]
| Silver | The data has been cleansed, validated, and stored in an optimal format to support downstream analytic use-cases. | staging | Delta | link:./data_processing.adoc[Data processing]
| Gold | Reliable data entities prepared for specific use-cases. These typically combine and aggregate datasets from the silver layer. | conformance | Delta | link:./data_processing.adoc[Data processing]
|===

### Fabric lakehouse

link:https://learn.microsoft.com/en-us/fabric/[Microsoft Fabric] can be used as the lakehouse layer in an Ensono Stacks data platform. Fabric provides an all-in-one analytics experience, with a range of tools for data analysts, data engineers and data scientists. It also closely integrates with Power BI. Full details on getting a Fabric lakehouse up and running with your Ensono Stacks data platform are in the link:../getting_started/fabric_deployment_guide.adoc[getting started] section.

== Datastacks

link:./datastacks.adoc[Datastacks] is a library and CLI built to accelerate the development of data engineering workloads in the data platform. It features a range of utilities to enhance the process for developing and deploying production-ready data pipelines and workloads.

== Quality assurance

Quality and reliability of data workloads are at the core of Ensono Stacks data platform. This is ensured through frameworks for link:./data_quality_azure.adoc[data quality] and link:./testing_data_azure.adoc[automated testing].

== Sample dataset

The example workloads included in the solution are based upon an link:../getting_started/example_data_source.adoc[example Azure SQL data source and dataset]. This data source may optionally be deployed as part of the data platform, to allow full demonstration testing of these example workloads.
