Toggle Menu
Prev Up Next

# `langchain_experimental.text_splitter`.SemanticChunker

class langchain_experimental.text_splitter.SemanticChunker(embeddings: Embeddings, buffer_size: int = 1, add_start_index: bool = False, breakpoint_threshold_type: Literal['percentile', 'standard_deviation', 'interquartile'] = 'percentile', breakpoint_threshold_amount: Optional[float] = None, number_of_chunks: Optional[int] = None, sentence_split_regex: str = '(?<=[.?!])\\s+')

  Split the text based on semantic similarity.

  Taken from Greg Kamradt’s wonderful notebook: https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb

  All credits to him.

  At a high level, this splits into sentences, then groups into groups of 3 sentences, and then merges one that are similar in the embedding space.

  Methods

  `__init__`(embeddings[, buffer_size, ...]) |
  `atransform_documents`(documents, **kwargs) | Asynchronously transform a list of documents.
  `create_documents`(texts[, metadatas]) | Create documents from a list of texts.
  `split_documents`(documents) | Split documents.
  `split_text`(text) |
  `transform_documents`(documents, **kwargs) | Transform sequence of documents by splitting them.

  Parameters

    - embeddings (Embeddings) –

    - buffer_size (int) –

    - add_start_index (bool) –

    - breakpoint_threshold_type (Literal['percentile', 'standard_deviation', 'interquartile']) –

    - breakpoint_threshold_amount (Optional[float]) –

    - number_of_chunks (Optional[int]) –

    - sentence_split_regex (str) –

  __init__(embeddings: Embeddings, buffer_size: int = 1, add_start_index: bool = False, breakpoint_threshold_type: Literal['percentile', 'standard_deviation', 'interquartile'] = 'percentile', breakpoint_threshold_amount: Optional[float] = None, number_of_chunks: Optional[int] = None, sentence_split_regex: str = '(?<=[.?!])\\s+')
    Parameters

      - embeddings (Embeddings) –

      - buffer_size (int) –

      - add_start_index (bool) –

      - breakpoint_threshold_type (Literal['percentile', 'standard_deviation', 'interquartile']) –

      - breakpoint_threshold_amount (Optional[float]) –

      - number_of_chunks (Optional[int]) –

      - sentence_split_regex (str) –

  async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]

    Asynchronously transform a list of documents.

    Parameters

      - documents (Sequence[Document]) – A sequence of Documents to be transformed.

      - kwargs (Any) –

    Returns

      A list of transformed Documents.

    Return type

      Sequence[Document]

  create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) → List[Document]

    Create documents from a list of texts.

    Parameters

      - texts (List[str]) –

      - metadatas (Optional[List[dict]]) –

    Return type

      List[Document]

  split_documents(documents: Iterable[Document]) → List[Document]

    Split documents.

    Parameters

      documents (Iterable[Document]) –

    Return type

      List[Document]

  split_text(text: str) → List[str]
    Parameters

      text (str) –

    Return type

      List[str]

  transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]

    Transform sequence of documents by splitting them.

    Parameters

      - documents (Sequence[Document]) –

      - kwargs (Any) –

    Return type

      Sequence[Document]

# Examples using SemanticChunker

- This is a long document we can split up.