Toggle Menu
Prev Up Next

# `langchain_community.document_loaders.parsers.language.language_parser`.LanguageParser

class langchain_community.document_loaders.parsers.language.language_parser.LanguageParser(language: Optional[Literal['cpp', 'go', 'java', 'kotlin', 'js', 'ts', 'php', 'proto', 'python', 'rst', 'ruby', 'rust', 'scala', 'swift', 'markdown', 'latex', 'html', 'sol', 'csharp', 'cobol', 'c', 'lua', 'perl']] = None, parser_threshold: int = 0)

  Parse using the respective programming language syntax.

  Each top-level function and class in the code is loaded into separate documents. Furthermore, an extra document is generated, containing the remaining top-level code that excludes the already segmented functions and classes.

  This approach can potentially improve the accuracy of QA models over source code.

  The supported languages for code parsing are:

  - C: “c” (*)

  - C++: “cpp” (*)

  - C#: “csharp” (*)

  - COBOL: “cobol”

  - Go: “go” (*)

  - Java: “java” (*)

  - JavaScript: “js” (requires package esprima)

  - Kotlin: “kotlin” (*)

  - Lua: “lua” (*)

  - Perl: “perl” (*)

  - Python: “python”

  - Ruby: “ruby” (*)

  - Rust: “rust” (*)

  - Scala: “scala” (*)

  - TypeScript: “ts” (*)

  Items marked with (*) require the packages tree_sitter and tree_sitter_languages. It is straightforward to add support for additional languages using tree_sitter, although this currently requires modifying LangChain.

  The language used for parsing can be configured, along with the minimum number of lines required to activate the splitting based on syntax.

  If a language is not explicitly specified, LanguageParser will infer one from filename extensions, if present.

  Examples

          from langchain_community.document_loaders.generic import GenericLoader
          from langchain_community.document_loaders.parsers import LanguageParser

          loader = GenericLoader.from_filesystem(
              "./code",
              glob="**/*",
              suffixes=[".py", ".js"],
              parser=LanguageParser()
          )
          docs = loader.load()

      Example instantiations to manually select the language:

      .. code-block:: python

          loader = GenericLoader.from_filesystem(
              "./code",
              glob="**/*",
              suffixes=[".py"],
              parser=LanguageParser(language="python")
          )

      Example instantiations to set number of lines threshold:

      .. code-block:: python

          loader = GenericLoader.from_filesystem(
              "./code",
              glob="**/*",
              suffixes=[".py"],
              parser=LanguageParser(parser_threshold=200)
          )

  Language parser that split code using the respective language syntax.

  Parameters

    - language (Optional[Language]) – If None (default), it will try to infer language from source.

    - parser_threshold (int) – Minimum lines needed to activate parsing (0 by default).

  Methods

  `__init__`([language, parser_threshold]) | Language parser that split code using the respective language syntax.
  `lazy_parse`(blob) | Lazy parsing interface.
  `parse`(blob) | Eagerly parse the blob into a document or documents.

  __init__(language: Optional[Literal['cpp', 'go', 'java', 'kotlin', 'js', 'ts', 'php', 'proto', 'python', 'rst', 'ruby', 'rust', 'scala', 'swift', 'markdown', 'latex', 'html', 'sol', 'csharp', 'cobol', 'c', 'lua', 'perl']] = None, parser_threshold: int = 0)

    Language parser that split code using the respective language syntax.

    Parameters

      - language (Optional[Literal['cpp', 'go', 'java', 'kotlin', 'js', 'ts', 'php', 'proto', 'python', 'rst', 'ruby', 'rust', 'scala', 'swift', 'markdown', 'latex', 'html', 'sol', 'csharp', 'cobol', 'c', 'lua', 'perl']]) – If None (default), it will try to infer language from source.

      - parser_threshold (int) – Minimum lines needed to activate parsing (0 by default).

  lazy_parse(blob: Blob) → Iterator[Document]

    Lazy parsing interface.

    Subclasses are required to implement this method.

    Parameters

      blob (Blob) – Blob instance

    Returns

      Generator of documents

    Return type

      Iterator[Document]

  parse(blob: Blob) → List[Document]

    Eagerly parse the blob into a document or documents.

    This is a convenience method for interactive development environment.

    Production applications should favor the lazy_parse method instead.

    Subclasses should generally not over-ride this parse method.

    Parameters

      blob (Blob) – Blob instance

    Returns

      List of documents

    Return type

      List[Document]

# Examples using LanguageParser

- Code for: class MyClass:

- Set env var OPENAI_API_KEY or load from a .env file