---
id: data_storage_cosmosdb_netcore
title: Data Storage - CosmosDB
linkTitle: Data Storage - CosmosDB
weight: 4
keywords:
  - .net core
  - rest api
  - cqrs
  - template 
  - cosmos 
  - cosmosdb 
  - data storage 
  - azure 
  - storage 
  - configuration 
  - cosmosdbkey
  - documentsearch
  - sql
  - api
  - repository
  - cosmosdb package
  - dependency injection
  - stacks
---

== Summary

This guide provides information on how to use the Ensono Stacks package for CosmosDB document storage.

The focus of this guide is the CosmosDB SQL API, and it assumes that the reader has prior knowledge about link:https://docs.microsoft.com/en-us/azure/cosmos-db/sql-query-getting-started[CosmosDB SQL API] in order to understand the concepts described thereafter.

It is out of scope for this document to explain what CosmosDB is and its respective storage models. Instead, it describes how the package for CosmosDB handles communication with the server. If you don't have enough understanding of CosmosDB concepts, please take some time to review the CosmosDB documentation before you continue.

=== Introduction

Most applications require a persistent data store to keep application data safe and available for use whenever an operation is executed.

The main storage type used in this project is a document store to persist domain data. The storage is provided by Azure Cosmos DB SQL API.

=== The Problem

Monolith applications generally use a single database to store all application data. In general, monolith solutions contain a data layer responsible for handling database communication and data conversion using an ORM library like Entity Framework, NHibernate, or Dapper.

Microservice architecture also requires a data layer. Different from monolith applications, each microservice tends to have its own data layer, generating lots of duplicate data access code spread across multiple repositories.

Duplicate data access code will, at some point, diverge and make each microservice have different features, or the same features with different implementations that, in most cases, will be tightly coupled to the service that created it.

Most of these features created in a data access layer should be well planned, tested, and made available to all services to have a consistent and reliable approach used by every team. This helps reduce the time needed to implement data access logic and avoid the same pitfalls others might face when starting from scratch.

=== The Solution

Azure CosmosDB provides a reliable SDK to communicate with the CosmosDB server, but it still requires some knowledge from the developer to consume it properly, add proper exception handling and logging.

This package provides a single data access layer implementation that is simple and reusable by multiple services, reducing the amount of boilerplate code and keeping a consistent and well-tested library.

The solution is split into two packages: *Ensono.Stacks.Data.Documents* and *Ensono.Stacks.Data.Documents.CosmosDB*.

=== Document Storage

Package *Ensono.Stacks.Data.Documents* contains the interface contracts for data access layers. It defines the operations available and decouples the contracts from the implementation. In the future, if the company decides to use other solutions like MongoDB, Cassandra, or similar, it will be easy to switch between servers without changing the code in services consuming it.

The contracts are defined by two interfaces: *IDocumentStorage* and *IDocumentSearch*:

[source,csharp]
.IDocumentStorage
----
IDocumentStorage<TEntity, in TEntityIdentityType>
        Task<OperationResult<TEntity>> SaveAsync(TEntityIdentityType identifier, string partitionKey, TEntity document, string eTag);
        Task<OperationResult<TEntity>> GetByIdAsync(TEntityIdentityType identifier, string partitionKey);
        Task<OperationResult> DeleteAsync(TEntityIdentityType identifier, string partitionKey);
----

.IDocumentSearch
----
IDocumentSearch<TEntity> Task<OperationResult<IEnumerable<TResult>>> Search<TResult, TOrderKey>(
    Expression<Func<TResult, bool>> searchPredicate,
    Expression<Func<TResult, TOrderKey>> orderPredicate = null,
    bool isAscendingOrder = true,
    string partitionKey = null,
    int pageSize = 20,
    int pageNumber = 1
);
----

The operations were decoupled into two interfaces to:

* Segregate Command operations from Read operations, except for GetById, which is required for commands that need the full structure stored in the database in order to update it.

* Separate concerns: Some implementations might not provide search capabilities, as it might be offloaded to a search engine like ElasticSearch instead.

* Document databases might have separate storage for search, and each document's structure is based on the search pattern, partition, and indexes. In this case, the documents stored in the search database will not match the schema of the original entity stored in the domain database.

*It is expected that the implementations of these interfaces are reused between requests; therefore, they will be registered as singletons to improve performance.*

Applications using Document Storage do not need to be aware of the implementations. You should only need the *Ensono.Stacks.Data.Documents* namespace where it is being used. Doing so will make it easier to mock DB dependencies when doing unit tests.

=== CosmosDB Package

The CosmosDB implementation of the document storage is called *CosmosDbDocumentStorage* and is available in the package *Ensono.Stacks.Data.Documents.CosmosDB*.

The package makes use of the CosmosDB SDK v3, which contains a few improvements compared to the previous version. One notable improvement is the simplicity in the available interfaces. For more details, please check the announcements link:https://azure.microsoft.com/en-gb/blog/azure-cosmos-dotnet-sdk-version-3-0-now-in-public-preview/[here].

A few details you should be aware of when using the CosmosDB package from Stacks:

* The library use naming convention to map the Aggregate to the CosmosDB collection name(previous known as container), the library will map to the entity type name to a CosmosDB collection with same name, in the example below, the entity type is a Menu (like in a restaurant), a collection with same name should exist in the database(provisioned by DevOps).

** The collection name is not configurable because the configuration should be valid for any collection, otherwise multiple configuration would be required per table.

** The package does not create the collection if one does not exist to avoid misconfiguration, which would generate new collections and cause unpredictable behaviors, as the application would continue working but send data to the wrong collection.

** Alternatively, you can add an Attribute *[Table("CollectionName")]* to the entity class name to map your entity to a collection with a different name.

* *CosmosDbDocumentStorage* is registered as a singleton object per collection. This is required to better reuse connections when calling CosmosDB, resulting in better performance.

* CosmosDB requires a partition key to store data. Reusing the ID as a partition key will make it performant and cheap for writes but expensive for reads that do not use the ID as a parameter.

** In case the application is heavy on read operations covering multiple partitions, you should consider a materialized collection partitioned and indexed according to the search parameters used for the query. Doing an across-partition query is an expensive operation and should be avoided.

** Schema design is out of the scope of this guide. For a better design of your data, please watch this video with CosmosDB design patterns.

*Usage:* When the package is used in the data access layer, as in the example of a repository implementation, the package will make the data access logic simple and clean, offloading all the data access logic to the implemented storage.

[source,csharp]
.MenuRepository
----
public class MenuRepository : IMenuRepository
{
    IDocumentStorage<Menu, Guid> documentStorage;

    public MenuRepository(IDocumentStorage<Menu, Guid> documentStorage)
    {
        this.documentStorage = documentStorage;
    }

    public async Task<Menu> GetByIdAsync(Guid id)
    {
        var result = await documentStorage.GetByIdAsync(id, id.ToString());
        return result.Content;
    }

    public async Task<bool> SaveAsync(Menu entity)
    {
        var result = await documentStorage.SaveAsync(entity.Id, entity.Id.ToString(), entity, null);
        return result.IsSuccessful;
    }

    public async Task<bool> DeleteAsync(Guid id)
    {
        var result = await documentStorage.DeleteAsync(id, id.ToString());
        return result.IsSuccessful;
    }
}
----

As seen above, the database implementation shouldn't leak to the repository implementation, making it clean and readable.

Because document stores require a partition key to persist the data, it is required that the caller provides it on every call. In the example above, we are reusing the resource ID as the partition key. Depending on the design, you might choose a different partition key that is only known by the application. Make sure you select a partition key that is known at query time; otherwise, the application will execute cross-partition queries when the partition key is not provided.

=== Dependency Injection

In order to use the right implementation, we need to inject the dependencies into the IoC container before the application starts. The CosmosDB implementation provides an extension method called *AddCosmosDB()* from the namespace *Ensono.Stacks.Data.Documents.CosmosDB.Extensions*. To use it, you just have to make a call to the extension in the dependency registration method of your application, like below:

[source,csharp]
----
public virtual void ConfigureServices(IServiceCollection services)
{
    services.Configure<CosmosDbConfiguration>(context.Configuration.GetSection("CosmosDB"));
    services.AddCosmosDB();
    services.AddSecrets(); // Required for CosmosDB configuration, see below
}
----

The first line will load the configuration from *appsettings.json* and bind it to an instance of *CosmosDbConfiguration*. This will be injected into the CosmosDB constructor as *IOptions<CosmosDbConfiguration>*. Please see the configuration details below.

The second line adds the *CosmosDbDocumentStorage* as a singleton implementation for *IDocumentStorage<TEntity, in TEntityIdentityType>* and *IDocumentSearch<TEntity>*.

The third line adds the dependencies required to resolve the password at runtime. Password management has been described in the docs with more details.

=== Configuration

The Cosmos DB implementation requires three mandatory parameters to work properly: *AccountUri*, *DatabaseName*, and *SecurityKeySecret*. The *CosmosDbConfiguration* represents these settings in the *appsettings.json* as a *CosmosDB* section with the following structure:

[source,json]
----
{
    "CosmosDb": {
        "DatabaseAccountUri": "https://localhost:8081/",
        "DatabaseName": "Stacks",
        "SecurityKeySecret": {
            "Identifier": "COSMOSDBKEY",
            "Source": "Environment"
        }
    }
}
----

* *DatabaseAccountUri*: is the URL for the database account that hosts the databases in Azure.

* *DatabaseName*: is the name of the database used by the application. An account might have multiple databases, but an application generally uses only one.

* *SecurityKeySecret*: represents a secret that is stored elsewhere. The value of this secret will be passed to the SDK as *SecurityKey* for authentication with the CosmosDB account. 

** The secret might be hosted anywhere, within the container or on a remote server. For that reason, we use an identifier that will instruct the secret resolver where to find it. The secret management process is described in the docs with more details.

=== Operation Result

Every operation will return an *OperationResult<T>* response. This is an object that represents the outcome of the operation. In case no exception is thrown, it contains the following details:

* *IsSuccessful*: A Boolean flag that represents the status of the operation, in case it completes without exception.

* *Content*: Depending on the type of the operation, it will return either a bool, an entity object, or a list of entities.

* *Attributes*: A dictionary containing implementation-specific data to be consumed by the application if required. For CosmosDB implementation, it returns the keys *ETag* and *RequestCharge* with their respective values.

=== Unit Tests

The library already has unit tests and integration tests covering the operations implemented. This does not prevent the consuming application from writing their own integration tests.

The tests implemented in the package ensure the package works for the intended scenarios. When applications start consuming it, some design decisions might not serialize properly, causing loss of data. For this reason, it is important that the consuming applications have tests covering at least *Create* and *Read* tests to make sure the contents written to the DB are serialized and de-serialized correctly based on entity structure.
