# Remove Langchain from tools

* **Status:** approved
* **Proposer:**  @adamdougal
* **Date:** 2024-04-30
* **Technical Story:** [Make required changes to store image embeddings](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/748)

## Context and Problem Statement

In order to enhance our data querying capabilities to include image-based data, we need to incorporate image embeddings
generated by [Azure Computer Vision](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/how-to/image-retrieval?tabs=csharp#call-the-vectorize-image-api).
Furthermore, we aim to generate image captions using `gpt-4-vision` and store the embeddings of these captions alongside
the image embeddings. This necessitates the introduction of an **additional** field in the index to accommodate multiple
embedding vectors, as demonstrated in the [spike notebook](../../spikes/using-image-data/ai-vision.ipynb).

Currently, we utilise [Langchain](https://python.langchain.com/docs/get_started/quickstart/) to create and query the
Azure AI Search index. However, as of the time of writing, Langchain does not support querying Azure AI Search for
multiple embedding vectors in a single query. Additionally, it lacks support for using Azure Computer Vision to generate
embeddings, as can be seen in the [list of supported embedding models](https://python.langchain.com/docs/integrations/text_embedding/).

The purpose of this ADR is to determine the best approach to incorporate the ability to query image-based data, given
the aforementioned constraints.

## Decision Drivers

* Certainty of delivery
* Speed of delivery
* Zero loss of existing functionality
* Code complexity
* Maintainability

## Considered Options

* Remove Langchain from tools
* Contribute features back to Langchain
* Keep Langchain but not if advanced image processing enabled

## Decision Outcome

Remove Langchain from tools. This gives us full control to deliver this feature. It will also allow us to implement
cutting edge features in future without having to wait for Langchain to be updated.

## Pros and Cons of the Options

### Remove Langchain from tools

For this option, it's proposed to remove the use of Langchain from the following files:
- [AzureSearchHelper.py](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/blob/a1ce1b9854aa5246623be20f5f6778969a37d165/code/backend/batch/utilities/helpers/AzureSearchHelper.py)
- [QuestionAnswerTool.py](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/blob/a1ce1b9854aa5246623be20f5f6778969a37d165/code/backend/batch/utilities/tools/QuestionAnswerTool.py)
- [PostPromptTool.py](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/blob/a1ce1b9854aa5246623be20f5f6778969a37d165/code/backend/batch/utilities/tools/PostPromptTool.py)

The use of Langchain as an orchestrator in the
[LangChainAgent.py](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/blob/a1ce1b9854aa5246623be20f5f6778969a37d165/code/backend/batch/utilities/orchestrator/LangChainAgent.py)
file will be retained.

This change would give us full control over how we create and query the search index.

* Good, because we can implement the ingestion and search to meet our requirements
* Good, because the changes have been proven in [Pull Request #786](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/786)
* Good, because speed of delivery is low and in our control
* Good, because no loss of functionality
* Bad, because we stop using Langchain which makes it easier to switch between different vector stores and llms.
  However, we are very much setup to mainly work with Azure based services already.

### Contribute features back to Langchain

For this option, we would create and contribute back a new integration with Azure Computer Vision
[https://python.langchain.com/docs/contributing/integrations/](https://python.langchain.com/docs/contributing/integrations/).

We would also need to make a change to allow multiple search queries to be passed to Azure AI Search.

* Good, because the changes we would be making have been proven by [https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/786](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/786)
* Good, because minimal changes the CWYDSA code base
* Good, because no loss of functionality
* Bad, because our required changes may not be accepted by the Langchain maintainers
* Bad, there's no control over when this will be able to be delivered


### Keep Langchain but not if advanced image processing enabled

For this option, if advanced image processing is not enabled, we will continue doing what we are doing today. If it is
enabled, we will have a different path in the code base for ingestion and querying.

* Good, because we can implement the ingestion and search to meet our requirements
* Good, because the changes have been proven in [https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/786](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/pull/786)
* Good, because speed of delivery is low and in our control
* Good, because no loss of functionality
* Bad, because it significantly increases the code complexity with multiple paths
