---
title: Ollama
---

import { Image } from 'astro:assets';
import icon from '../../../assets/icon-ollama.png';

<span title="Label: Pro" data-view-component="true" class="Label Label--api text-uppercase">
  Unstable API
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--version text-uppercase">
  0.8.0
</span>
<span title="Label: Pro" data-view-component="true" class="Label Label--package">
  <a target="_blank" href="https://www.npmjs.com/package/@project-lakechain/ollama-embedding-processor">
    @project-lakechain/ollama-embedding-processor
  </a>
</span>
<span class="language-icon">
  <svg role="img" viewBox="0 0 24 24" width="30" xmlns="http://www.w3.org/2000/svg" style="fill: #3178C6;"><title>TypeScript</title><path d="M1.125 0C.502 0 0 .502 0 1.125v21.75C0 23.498.502 24 1.125 24h21.75c.623 0 1.125-.502 1.125-1.125V1.125C24 .502 23.498 0 22.875 0zm17.363 9.75c.612 0 1.154.037 1.627.111a6.38 6.38 0 0 1 1.306.34v2.458a3.95 3.95 0 0 0-.643-.361 5.093 5.093 0 0 0-.717-.26 5.453 5.453 0 0 0-1.426-.2c-.3 0-.573.028-.819.086a2.1 2.1 0 0 0-.623.242c-.17.104-.3.229-.393.374a.888.888 0 0 0-.14.49c0 .196.053.373.156.529.104.156.252.304.443.444s.423.276.696.41c.273.135.582.274.926.416.47.197.892.407 1.266.628.374.222.695.473.963.753.268.279.472.598.614.957.142.359.214.776.214 1.253 0 .657-.125 1.21-.373 1.656a3.033 3.033 0 0 1-1.012 1.085 4.38 4.38 0 0 1-1.487.596c-.566.12-1.163.18-1.79.18a9.916 9.916 0 0 1-1.84-.164 5.544 5.544 0 0 1-1.512-.493v-2.63a5.033 5.033 0 0 0 3.237 1.2c.333 0 .624-.03.872-.09.249-.06.456-.144.623-.25.166-.108.29-.234.373-.38a1.023 1.023 0 0 0-.074-1.089 2.12 2.12 0 0 0-.537-.5 5.597 5.597 0 0 0-.807-.444 27.72 27.72 0 0 0-1.007-.436c-.918-.383-1.602-.852-2.053-1.405-.45-.553-.676-1.222-.676-2.005 0-.614.123-1.141.369-1.582.246-.441.58-.804 1.004-1.089a4.494 4.494 0 0 1 1.47-.629 7.536 7.536 0 0 1 1.77-.201zm-15.113.188h9.563v2.166H9.506v9.646H6.789v-9.646H3.375z"/></svg>
</span>
<span class="language-icon" style="margin-right: 12px">
  <a target="_blank" href="https://ollama.ai">
    <Image width="24" src={icon} alt="Icon" style="border-radius: 50%" />
  </a>
</span>
<div style="margin-top: 26px"></div>

---

The Ollama embedding middleware enables customers to run [Ollama embedding models](https://ollama.com/library) within their AWS account to create vector embeddings for text and markdown documents.

To orchestrate deployments, this middleware deploys an auto-scaled cluster of CPU or GPU-enabled containers that consume documents from the middleware input queue. The cluster is deployed in the private subnet of the given VPC, and caches the models on an EFS storage to optimize cold-starts.

---

### 🦙 Embedding Documents

To use this middleware, you import it in your CDK stack and specify a VPC in which the processing cluster will be deployed.
You will also need to select the specific embedding model to use.

> ℹ️ The below example shows how to deploy this middleware in a VPC using the [`nomic-embed-text`](https://ollama.com/library/nomic-embed-text) model.

```typescript
import { CacheStorage } from '@project-lakechain/core';
import {
  OllamaEmbeddingProcessor,
  OllamaEmbeddingModel,
  InfrastructureDefinition
} from '@project-lakechain/ollama-embedding-processor';

class Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string) {
    // Sample VPC.
    const vpc = new ec2.Vpc(this, 'Vpc', {});

    // The cache storage.
    const cache = new CacheStorage(this, 'Cache');

    // Creates embeddings for text using the Ollama model.
    const ollama = new OllamaEmbeddingProcessor.Builder()
      .withScope(this)
      .withIdentifier('OllamaEmbeddingProcessor')
      .withCacheStorage(cache)
      .withVpc(vpc)
      .withSource(source) // 👈 Specify a data source
      .withModel(OllamaEmbeddingModel.NOMIC_EMBED_TEXT)
      .withInfrastructure(new InfrastructureDefinition.Builder()
        .withMaxMemory(15 * 1024)
        .withGpus(1)
        .withInstanceType(ec2.InstanceType.of(
          ec2.InstanceClass.G4DN,
          ec2.InstanceSize.XLARGE2
        ))
        .build())
      .build();
  }
}
```

<br />

---

### 🤖 Model Selection

Ollama supports a variety of embedding models, and you can specify the model and optionally the specific tag to use.

> 💁 When no tag is provided, the `latest` tag is automatically used. The example below showcases how to use a specific tag on a model.

```typescript
import { OllamaEmbeddingProcessor, OllamaEmbeddingModel } from '@project-lakechain/ollama-embedding-processor';

const ollama = new OllamaEmbeddingProcessor.Builder()
  .withScope(this)
  .withIdentifier('OllamaEmbeddingProcessor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withModel(OllamaEmbeddingModel.NOMIC_EMBED_TEXT.tag('13b'))
  .withInfrastructure(infrastructure)
  .build();
```

<br />

---

#### Escape Hatch

The `OllamaEmbeddingModel` class provides a quick way to reference existing models, and select a specific tag.
However, as Ollama adds new models, you may be in a situation where a model is not yet referenced by this middleware.

To address this situation, you can manually specify a model definition pointing to the supported Ollama embedding model you wish to run.
You do so by specifying the name of the model in the [Ollama library](https://ollama.com/library), the tag you wish to use, and its supported input and output mime-types.

> 💁 In the example below, we define the `snowflake-arctic-embed` model.

```typescript
import { OllamaEmbeddingProcessor, OllamaEmbeddingModel } from '@project-lakechain/ollama-embedding-processor';

const ollama = new OllamaEmbeddingProcessor.Builder()
  .withScope(this)
  .withIdentifier('OllamaEmbeddingProcessor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withModel(OllamaModel.of('llava', { tag: 'latest' }))
  .withPrompt(prompt)
  .withInfrastructure(infrastructure)
  .build();
```

<br />

---

### ↔️ Concurrency

The cluster of containers deployed by this middleware will auto-scale based on the number of documents that need to be processed.
The cluster scales up to a maximum of 5 instances by default, and scales down to zero when there are no documents to process.

> ℹ️ You can configure the maximum amount of instances that the cluster can auto-scale to by using the `.withMaxConcurrency` method.

```typescript
const ollama = new OllamaEmbeddingProcessor.Builder()
  .withScope(this)
  .withIdentifier('OllamaEmbeddingProcessor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withSource(source)
  .withMaxConcurrency(10) // 👈 Maximum number of instances.
  .withModel(model)
  .withInfrastructure(infrastructure)
  .build();
```

<br />

---

### 📦 Batch Processing

Ollama supports processing documents in batches since [Ollama 0.2.0](https://github.com/ollama/ollama/releases/tag/v0.2.0).
This middleware can take advantage of the new *parallel requests* feature of Ollama to create embeddings for multiple documents in a single request, thus improving
the overall throughput of the processing cluster.

> ℹ️ You can configure the maximum number of documents to process in a single batch by using the `.withBatchSize` method. Note that the maximum batch size is set to 10, and that batching performance depends on the size of the chosen instance.

```typescript
const ollama = new OllamaEmbeddingProcessor.Builder()
  .withScope(this)
  .withIdentifier('OllamaEmbeddingProcessor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withSource(source)
  .withBatchSize(5) // 👈 Maximum batch size.
  .withModel(model)
  .withInfrastructure(infrastructure)
  .build();
```

<br />

---

### 🌉 Infrastructure

Every model requires a specific infrastructure to run optimally.
To ensure the `OllamaEmbeddingProcessor` orchestrates your models using the most optimal instance, memory, and GPU allocation, you need to specify an infrastructure definition.

> 💁 The example below describes the infrastructure suited to run the `nomic-embed-text` model on a GPU instance.

```typescript
const ollama = new OllamaEmbeddingProcessor.Builder()
  .withScope(this)
  .withIdentifier('OllamaEmbeddingProcessor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withModel(OllamaEmbeddingModel.NOMIC_EMBED_TEXT)
  // 👇 Infrastructure definition.
  .withInfrastructure(new InfrastructureDefinition.Builder()
    .withMaxMemory(15 * 1024)
    .withGpus(1)
    .withInstanceType(ec2.InstanceType.of(
      ec2.InstanceClass.G4DN,
      ec2.InstanceSize.XLARGE2
    ))
    .build())
  .build();
```

Below is a description of the fields associated with the infrastructure definition.

| Field | Description |
| ----- | ----------- |
| maxMemory | The maximum RAM in MiB to allocate to the container. |
| gpus | The number of GPUs to allocate to the container (only relevant for GPU instances). |
| instanceType | The EC2 instance type to use for running the containers. |

<br />

---

### 📄 Output

The Ollama embedding middleware does not modify or alter source documents in any way.
It instead enriches the metadata of the documents with a pointer to the vector embedding that were created for the document.

<details>
  <summary>💁 Click to expand example</summary>

  ```json
  {
    "specversion": "1.0",
    "id": "1780d5de-fd6f-4530-98d7-82ebee85ea39",
    "type": "document-created",
    "time": "2023-10-22T13:19:10.657Z",
    "data": {
        "chainId": "6ebf76e4-f70c-440c-98f9-3e3e7eb34c79",
        "source": {
            "url": "s3://bucket/document.txt",
            "type": "text/plain",
            "size": 245328,
            "etag": "1243cbd6cf145453c8b5519a2ada4779"
        },
        "document": {
            "url": "s3://bucket/document.txt",
            "type": "text/plain",
            "size": 245328,
            "etag": "1243cbd6cf145453c8b5519a2ada4779"
        },
        "metadata": {
          "properties": {
            "kind": "text",
            "attrs": {
              "embeddings": {
                "vectors": "s3://cache-storage/ollama-embedding-processor/45a42b35c3225085.json",
                "model": "nomic-embed-text:latest",
                "dimensions": 768
            }
          }
        }
      }
    }
  }
  ```

</details>

<br />

---

### ℹ️ Limits

Embedding models have limits on the number of input tokens they can process.
For more information, you can consult the documentation of the specific model you are using to understand these limits.

> 💁 To limit the size of upstream text documents, we recommend to use a text splitter to chunk text documents before they are passed to this middleware, such as the [Recursive Character Text Splitter](/project-lakechain/text-splitters/recursive-character-text-splitter).

<br />

---

### 🏗️ Architecture

This middleware requires CPU or GPU-enabled instances to run the embedding models. To orchestrate deployments, it deploys an ECS auto-scaled cluster of containers that consume documents from the middleware input queue. The cluster is deployed in the private subnet of the given VPC, and caches the models on an EFS storage to optimize cold-starts.

> ℹ️ The average cold-start for Ollama containers is around 3 minutes when no instances are running.

![Sentence Transformers Architecture](../../../assets/sentence-transformers-architecture.png)

<br />

---

### 🏷️ Properties

<br />

##### Supported Inputs

|  Mime Type  | Description |
| ----------- | ----------- |
| `text/plain` | UTF-8 text documents. |
| `text/markdown` | UTF-8 markdown documents. |

##### Supported Outputs

|  Mime Type  | Description |
| ----------- | ----------- |
| `text/plain` | UTF-8 text documents. |
| `text/markdown` | UTF-8 markdown documents. |

##### Supported Compute Types

| Type  | Description |
| ----- | ----------- |
| `CPU` | This middleware supports CPU compute. |
| `GPU` | This middleware supports GPU compute. |

<br />

---

### 📖 Examples

- [Ollama LanceDB Pipeline](https://github.com/awslabs/project-lakechain/tree/main/examples/simple-pipelines/embedding-pipelines/ollama-lancedb-pipeline) - An example showcasing how to create vector embeddings for text documents using Ollama and store them in a LanceDB embedded database.
