Toggle Menu
Prev Up Next

# `langchain_text_splitters.character`.CharacterTextSplitter

class langchain_text_splitters.character.CharacterTextSplitter(separator: str = '\n\n', is_separator_regex: bool = False, **kwargs: Any)

  Splitting text that looks at characters.

  Create a new TextSplitter.

  Methods

  `__init__`([separator, is_separator_regex]) | Create a new TextSplitter.
  `atransform_documents`(documents, **kwargs) | Asynchronously transform a list of documents.
  `create_documents`(texts[, metadatas]) | Create documents from a list of texts.
  `from_huggingface_tokenizer`(tokenizer, **kwargs) | Text splitter that uses HuggingFace tokenizer to count length.
  `from_tiktoken_encoder`([encoding_name, ...]) | Text splitter that uses tiktoken encoder to count length.
  `split_documents`(documents) | Split documents.
  `split_text`(text) | Split incoming text and return chunks.
  `transform_documents`(documents, **kwargs) | Transform sequence of documents by splitting them.

  Parameters

    - separator (str) –

    - is_separator_regex (bool) –

    - kwargs (Any) –

  __init__(separator: str = '\n\n', is_separator_regex: bool = False, **kwargs: Any) → None

    Create a new TextSplitter.

    Parameters

      - separator (str) –

      - is_separator_regex (bool) –

      - kwargs (Any) –

    Return type

      None

  async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]

    Asynchronously transform a list of documents.

    Parameters

      - documents (Sequence[Document]) – A sequence of Documents to be transformed.

      - kwargs (Any) –

    Returns

      A list of transformed Documents.

    Return type

      Sequence[Document]

  create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) → List[Document]

    Create documents from a list of texts.

    Parameters

      - texts (List[str]) –

      - metadatas (Optional[List[dict]]) –

    Return type

      List[Document]

  classmethod from_huggingface_tokenizer(tokenizer: Any, **kwargs: Any) → TextSplitter

    Text splitter that uses HuggingFace tokenizer to count length.

    Parameters

      - tokenizer (Any) –

      - kwargs (Any) –

    Return type

      TextSplitter

  classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → TS

    Text splitter that uses tiktoken encoder to count length.

    Parameters

      - encoding_name (str) –

      - model_name (Optional[str]) –

      - allowed_special (Union[Literal['all'], ~typing.AbstractSet[str]]) –

      - disallowed_special (Union[Literal['all'], ~typing.Collection[str]]) –

      - kwargs (Any) –

    Return type

      TS

  split_documents(documents: Iterable[Document]) → List[Document]

    Split documents.

    Parameters

      documents (Iterable[Document]) –

    Return type

      List[Document]

  split_text(text: str) → List[str]

    Split incoming text and return chunks.

    Parameters

      text (str) –

    Return type

      List[str]

  transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]

    Transform sequence of documents by splitting them.

    Parameters

      - documents (Sequence[Document]) –

      - kwargs (Any) –

    Return type

      Sequence[Document]

# Examples using CharacterTextSplitter

- # Automatically restart kernel after installs so that your environment can access the new packages

- # if you plan to use bson serialization, install also:

- %pip install –upgrade –quiet surrealdb langchain langchain-community

- Create a bes instance and index docs.

- Create collection if running for the first time. If the collection

- Dependencies {#dependencies}

- DocumentDB connection string

- Download the content

- Establishing a connection to the database is facilitated through the singlestoredb Python connector.

- Get an OpenAI token: https://platform.openai.com/account/api-keys

- Helper function for printing docs

- Here we useimport getpass

- Here we want to make sure the answer is minimally relevant

- If using the default Docker installation, use this instantiation instead:

- Load the document and split it into chunks

- Map reduce example

- Metadata {#metadata}

- OR

- OpenAI

- Option 1: use an OpenAI account

- Pip install necessary package

- Pip install necessary package {#pip-install-necessary-package}

- Pip install necessary packages

- Refresh is required for server use

- Retrievers

- Run tests with shell:

- Set env var OPENAI_API_KEY or load from a .env file

- Set up the OpenAI Environment Variables

- Setup {#setup}

- Swapping the `ByteStore` {#swapping-the-bytestore}

- This is a long document we can split up.

- This will only get documents for Ankush

- To make the caching really obvious, lets use a slower model.

- Uncomment this to install psychicapi if you don’t already have it installed

- Use Meilisearch vector store to store texts & associated embeddings as vector

- VDMS

- Vector stores

- Wait until the cluster is ready for use.

- You need to install sqlite-vss as a dependency.

- add texts

- adding_memory_chain_multiple_inputs.md

- analyticdb.md

- awadb.md

- baiduvectordb.md

- clickhouse.md

- conversational_retrieval_agents.md

- cosine: distance metric

- create cluster and add texts

- databricks_vector_search.md

- default metric is angular

- docs[0].metadata[“id”] == “id:testapp:testapp::32”

- document_comparison_toolkit.md

- drop first if index already exists

- duckdb.md

- ecloud_vector_search.md

- epsilla.md

- for example

- from langchain_community.embeddings.openai import OpenAIEmbeddings

- hologres.md

- import

- indexing.md

- initialize marqo

- install package

- lancedb.md

- openai

- or install latest:

- or shorter

- os.environ[“LANGCHAIN_TRACING_V2”] = “true”

- output length: 4

- pinecone.md

- qdrant.md

- replace

- scann.md

- tigris.md

- tiledb.md

- typesense.md

- use directly a `where_str` to delete

- usearch.md

- vectorstore.md

- with pip

- xata.md