Toggle Menu
Prev Up Next

# `langchain_core.indexing.api`.index

langchain_core.indexing.api.index(docs_source: Union[BaseLoader, Iterable[Document]], record_manager: RecordManager, vector_store: VectorStore, *, batch_size: int = 100, cleanup: Optional[Literal['incremental', 'full', None]] = None, source_id_key: Optional[Union[str, Callable[[Document], str]]] = None, cleanup_batch_size: int = 1000, force_update: bool = False) → IndexingResult

  Index data from the loader into the vector store.

  Indexing functionality uses a manager to keep track of which documents are in the vector store.

  This allows us to keep track of which documents were updated, and which documents were deleted, which documents should be skipped.

  For the time being, documents are indexed using their hashes, and users

    are not able to specify the uid of the document.

  Important

  if auto_cleanup is set to True, the loader should be returning the entire dataset, and not just a subset of the dataset. Otherwise, the auto_cleanup will remove documents that it is not supposed to.

  Parameters

    - docs_source (Union[BaseLoader, Iterable[Document]]) – Data loader or iterable of documents to index.

    - record_manager (RecordManager) – Timestamped set to keep track of which documents were updated.

    - vector_store (VectorStore) – Vector store to index the documents into.

    - batch_size (int) – Batch size to use when indexing.

    - cleanup (Optional[Literal['incremental', 'full', None]]) –

      How to handle clean up of documents. - Incremental: Cleans up all documents that haven’t been updated AND

      > that are associated with source ids that were seen during indexing. Clean up is done continuously during indexing helping to minimize the probability of users seeing duplicated content.

      - Full: Delete all documents that have not been returned by the loader

          during this run of indexing. Clean up runs after all documents have been indexed. This means that users may see duplicated content during indexing.

      - None: Do not delete any documents.

    - source_id_key (Optional[Union[str, Callable[[Document], str]]]) – Optional key that helps identify the original source of the document.

    - cleanup_batch_size (int) – Batch size to use when cleaning up documents.

    - force_update (bool) – Force update documents even if they are present in the record manager. Useful if you are re-indexing with updated embeddings.

  Returns

    Indexing result which contains information about how many documents were added, updated, deleted, or skipped.

  Return type

    IndexingResult

# Examples using index

- indexing.md