Toggle Menu
Prev Up Next

# `langchain_community.document_loaders.spider`.SpiderLoader

class langchain_community.document_loaders.spider.SpiderLoader(url: str, *, api_key: Optional[str] = None, mode: Literal['scrape', 'crawl'] = 'scrape', params: Optional[dict] = {'return_format': 'markdown'})

  Load web pages as Documents using Spider AI.

  Must have the Python package spider-client installed and a Spider API key. See https://spider.cloud for more.

  Initialize with API key and URL.

  Parameters

    - url (str) – The URL to be processed.

    - api_key (Optional[str]) – The Spider API key. If not specified, will be read from env

    - SPIDER_API_KEY. (var) –

    - mode (Literal['scrape', 'crawl']) – The mode to run the loader in. Default is “scrape”. Options include “scrape” (single page) and “crawl” (with deeper crawling following subpages).

    - params (Optional[dict]) – Additional parameters for the Spider API.

  Methods

  `__init__`(url, *[, api_key, mode, params]) | Initialize with API key and URL.
  `alazy_load`() | A lazy loader for Documents.
  `aload`() | Load data into Document objects.
  `lazy_load`() | Load documents based on the specified mode.
  `load`() | Load data into Document objects.
  `load_and_split`([text_splitter]) | Load Documents and split into chunks.

  __init__(url: str, *, api_key: Optional[str] = None, mode: Literal['scrape', 'crawl'] = 'scrape', params: Optional[dict] = {'return_format': 'markdown'})

    Initialize with API key and URL.

    Parameters

      - url (str) – The URL to be processed.

      - api_key (Optional[str]) – The Spider API key. If not specified, will be read from env

      - SPIDER_API_KEY. (var) –

      - mode (Literal['scrape', 'crawl']) – The mode to run the loader in. Default is “scrape”. Options include “scrape” (single page) and “crawl” (with deeper crawling following subpages).

      - params (Optional[dict]) – Additional parameters for the Spider API.

  async alazy_load() → AsyncIterator[Document]

    A lazy loader for Documents.

    Return type

      AsyncIterator[Document]

  async aload() → List[Document]

    Load data into Document objects.

    Return type

      List[Document]

  lazy_load() → Iterator[Document]

    Load documents based on the specified mode.

    Return type

      Iterator[Document]

  load() → List[Document]

    Load data into Document objects.

    Return type

      List[Document]

  load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]

    Load Documents and split into chunks. Chunks are returned as Documents.

    Do not override this method. It should be considered to be deprecated!

    Parameters

      text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

    Returns

      List of Documents.

    Return type

      List[Document]