Toggle Menu
Prev Up Next

# `langchain_experimental.agents.agent_toolkits.spark.base`.create_spark_dataframe_agent

langchain_experimental.agents.agent_toolkits.spark.base.create_spark_dataframe_agent(llm: BaseLLM, df: Any, callback_manager: Optional[BaseCallbackManager] = None, prefix: str = '\nYou are working with a spark dataframe in Python. The name of the dataframe is `df`.\nYou should use the tools below to answer the question posed of you:', suffix: str = '\nThis is the result of `print(df.first())`:\n{df}\n\nBegin!\nQuestion: {input}\n{agent_scratchpad}', input_variables: Optional[List[str]] = None, verbose: bool = False, return_intermediate_steps: bool = False, max_iterations: Optional[int] = 15, max_execution_time: Optional[float] = None, early_stopping_method: str = 'force', agent_executor_kwargs: Optional[Dict[str, Any]] = None, **kwargs: Any) → AgentExecutor

  Construct a Spark agent from an LLM and dataframe.

  Parameters

    - llm (BaseLLM) –

    - df (Any) –

    - callback_manager (Optional[BaseCallbackManager]) –

    - prefix (str) –

    - suffix (str) –

    - input_variables (Optional[List[str]]) –

    - verbose (bool) –

    - return_intermediate_steps (bool) –

    - max_iterations (Optional[int]) –

    - max_execution_time (Optional[float]) –

    - early_stopping_method (str) –

    - agent_executor_kwargs (Optional[Dict[str, Any]]) –

    - kwargs (Any) –

  Return type

    AgentExecutor

# Examples using create_spark_dataframe_agent

- in apache-spark root directory. (tested here with “spark-3.4.0-bin-hadoop3 and later”)