Toggle Menu
Prev Up Next

# `langchain_community.callbacks.argilla_callback`.ArgillaCallbackHandler

class langchain_community.callbacks.argilla_callback.ArgillaCallbackHandler(dataset_name: str, workspace_name: Optional[str] = None, api_url: Optional[str] = None, api_key: Optional[str] = None)

  Callback Handler that logs into Argilla.

  Parameters

    - dataset_name (str) – name of the FeedbackDataset in Argilla. Note that it must exist in advance. If you need help on how to create a FeedbackDataset in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html.

    - workspace_name (Optional[str]) – name of the workspace in Argilla where the specified FeedbackDataset lives in. Defaults to None, which means that the default workspace will be used.

    - api_url (Optional[str]) – URL of the Argilla Server that we want to use, and where the FeedbackDataset lives in. Defaults to None, which means that either ARGILLA_API_URL environment variable or the default will be used.

    - api_key (Optional[str]) – API Key to connect to the Argilla Server. Defaults to None, which means that either ARGILLA_API_KEY environment variable or the default will be used.

  Raises

    - ImportError – if the argilla package is not installed.

    - ConnectionError – if the connection to Argilla fails.

    - FileNotFoundError – if the FeedbackDataset retrieval from Argilla fails.

  Examples

      >>> from langchain_community.llms import OpenAI
      >>> from langchain_community.callbacks import ArgillaCallbackHandler
      >>> argilla_callback = ArgillaCallbackHandler(
      ...     dataset_name="my-dataset",
      ...     workspace_name="my-workspace",
      ...     api_url="http://localhost:6900",
      ...     api_key="argilla.apikey",
      ... )
      >>> llm = OpenAI(
      ...     temperature=0,
      ...     callbacks=[argilla_callback],
      ...     verbose=True,
      ...     openai_api_key="API_KEY_HERE",
      ... )
      >>> llm.generate([
      ...     "What is the best NLP-annotation tool out there? (no bias at all)",
      ... ])
      "Argilla, no doubt about it."

  Initializes the ArgillaCallbackHandler.

  Parameters

    - dataset_name (str) – name of the FeedbackDataset in Argilla. Note that it must exist in advance. If you need help on how to create a FeedbackDataset in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html.

    - workspace_name (Optional[str]) – name of the workspace in Argilla where the specified FeedbackDataset lives in. Defaults to None, which means that the default workspace will be used.

    - api_url (Optional[str]) – URL of the Argilla Server that we want to use, and where the FeedbackDataset lives in. Defaults to None, which means that either ARGILLA_API_URL environment variable or the default will be used.

    - api_key (Optional[str]) – API Key to connect to the Argilla Server. Defaults to None, which means that either ARGILLA_API_KEY environment variable or the default will be used.

  Raises

    - ImportError – if the argilla package is not installed.

    - ConnectionError – if the connection to Argilla fails.

    - FileNotFoundError – if the FeedbackDataset retrieval from Argilla fails.

  Attributes

  `BLOG_URL` |
  `DEFAULT_API_URL` |
  `ISSUES_URL` |
  `REPO_URL` |
  `ignore_agent` | Whether to ignore agent callbacks.
  `ignore_chain` | Whether to ignore chain callbacks.
  `ignore_chat_model` | Whether to ignore chat model callbacks.
  `ignore_llm` | Whether to ignore LLM callbacks.
  `ignore_retriever` | Whether to ignore retriever callbacks.
  `ignore_retry` | Whether to ignore retry callbacks.
  `raise_error` |
  `run_inline` |

  Methods

  `__init__`(dataset_name[, workspace_name, ...]) | Initializes the ArgillaCallbackHandler.
  `on_agent_action`(action, **kwargs) | Do nothing when agent takes a specific action.
  `on_agent_finish`(finish, **kwargs) | Do nothing
  `on_chain_end`(outputs, **kwargs) | If either the parent_run_id or the run_id is in self.prompts, then log the outputs to Argilla, and pop the run from self.prompts.
  `on_chain_error`(error, **kwargs) | Do nothing when LLM chain outputs an error.
  `on_chain_start`(serialized, inputs, **kwargs) | If the key input is in inputs, then save it in self.prompts using either the parent_run_id or the run_id as the key.
  `on_chat_model_start`(serialized, messages, *, ...) | Run when a chat model starts running.
  `on_llm_end`(response, **kwargs) | Log records to Argilla when an LLM ends.
  `on_llm_error`(error, **kwargs) | Do nothing when LLM outputs an error.
  `on_llm_new_token`(token, **kwargs) | Do nothing when a new token is generated.
  `on_llm_start`(serialized, prompts, **kwargs) | Save the prompts in memory when an LLM starts.
  `on_retriever_end`(documents, *, run_id[, ...]) | Run when Retriever ends running.
  `on_retriever_error`(error, *, run_id[, ...]) | Run when Retriever errors.
  `on_retriever_start`(serialized, query, *, run_id) | Run when Retriever starts running.
  `on_retry`(retry_state, *, run_id[, parent_run_id]) | Run on a retry event.
  `on_text`(text, **kwargs) | Do nothing
  `on_tool_end`(output[, observation_prefix, ...]) | Do nothing when tool ends.
  `on_tool_error`(error, **kwargs) | Do nothing when tool outputs an error.
  `on_tool_start`(serialized, input_str, **kwargs) | Do nothing when tool starts.

  __init__(dataset_name: str, workspace_name: Optional[str] = None, api_url: Optional[str] = None, api_key: Optional[str] = None) → None

    Initializes the ArgillaCallbackHandler.

    Parameters

      - dataset_name (str) – name of the FeedbackDataset in Argilla. Note that it must exist in advance. If you need help on how to create a FeedbackDataset in Argilla, please visit https://docs.argilla.io/en/latest/tutorials_and_integrations/integrations/use_argilla_callback_in_langchain.html.

      - workspace_name (Optional[str]) – name of the workspace in Argilla where the specified FeedbackDataset lives in. Defaults to None, which means that the default workspace will be used.

      - api_url (Optional[str]) – URL of the Argilla Server that we want to use, and where the FeedbackDataset lives in. Defaults to None, which means that either ARGILLA_API_URL environment variable or the default will be used.

      - api_key (Optional[str]) – API Key to connect to the Argilla Server. Defaults to None, which means that either ARGILLA_API_KEY environment variable or the default will be used.

    Raises

      - ImportError – if the argilla package is not installed.

      - ConnectionError – if the connection to Argilla fails.

      - FileNotFoundError – if the FeedbackDataset retrieval from Argilla fails.

    Return type

      None

  on_agent_action(action: AgentAction, **kwargs: Any) → Any

    Do nothing when agent takes a specific action.

    Parameters

      - action (AgentAction) –

      - kwargs (Any) –

    Return type

      Any

  on_agent_finish(finish: AgentFinish, **kwargs: Any) → None

    Do nothing

    Parameters

      - finish (AgentFinish) –

      - kwargs (Any) –

    Return type

      None

  on_chain_end(outputs: Dict[str, Any], **kwargs: Any) → None

    If either the parent_run_id or the run_id is in self.prompts, then log the outputs to Argilla, and pop the run from self.prompts. The behavior differs if the output is a list or not.

    Parameters

      - outputs (Dict[str, Any]) –

      - kwargs (Any) –

    Return type

      None

  on_chain_error(error: BaseException, **kwargs: Any) → None

    Do nothing when LLM chain outputs an error.

    Parameters

      - error (BaseException) –

      - kwargs (Any) –

    Return type

      None

  on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) → None

    If the key input is in inputs, then save it in self.prompts using either the parent_run_id or the run_id as the key. This is done so that we don’t log the same input prompt twice, once when the LLM starts and once when the chain starts.

    Parameters

      - serialized (Dict[str, Any]) –

      - inputs (Dict[str, Any]) –

      - kwargs (Any) –

    Return type

      None

  on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any

    Run when a chat model starts running.

    ATTENTION: This method is called for chat models. If you’re implementing

      a handler for a non-chat model, you should use on_llm_start instead.

    Parameters

      - serialized (Dict[str, Any]) –

      - messages (List[List[BaseMessage]]) –

      - run_id (UUID) –

      - parent_run_id (Optional[UUID]) –

      - tags (Optional[List[str]]) –

      - metadata (Optional[Dict[str, Any]]) –

      - kwargs (Any) –

    Return type

      Any

  on_llm_end(response: LLMResult, **kwargs: Any) → None

    Log records to Argilla when an LLM ends.

    Parameters

      - response (LLMResult) –

      - kwargs (Any) –

    Return type

      None

  on_llm_error(error: BaseException, **kwargs: Any) → None

    Do nothing when LLM outputs an error.

    Parameters

      - error (BaseException) –

      - kwargs (Any) –

    Return type

      None

  on_llm_new_token(token: str, **kwargs: Any) → None

    Do nothing when a new token is generated.

    Parameters

      - token (str) –

      - kwargs (Any) –

    Return type

      None

  on_llm_start(serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) → None

    Save the prompts in memory when an LLM starts.

    Parameters

      - serialized (Dict[str, Any]) –

      - prompts (List[str]) –

      - kwargs (Any) –

    Return type

      None

  on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any

    Run when Retriever ends running.

    Parameters

      - documents (Sequence[Document]) –

      - run_id (UUID) –

      - parent_run_id (Optional[UUID]) –

      - kwargs (Any) –

    Return type

      Any

  on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any

    Run when Retriever errors.

    Parameters

      - error (BaseException) –

      - run_id (UUID) –

      - parent_run_id (Optional[UUID]) –

      - kwargs (Any) –

    Return type

      Any

  on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any

    Run when Retriever starts running.

    Parameters

      - serialized (Dict[str, Any]) –

      - query (str) –

      - run_id (UUID) –

      - parent_run_id (Optional[UUID]) –

      - tags (Optional[List[str]]) –

      - metadata (Optional[Dict[str, Any]]) –

      - kwargs (Any) –

    Return type

      Any

  on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any

    Run on a retry event.

    Parameters

      - retry_state (RetryCallState) –

      - run_id (UUID) –

      - parent_run_id (Optional[UUID]) –

      - kwargs (Any) –

    Return type

      Any

  on_text(text: str, **kwargs: Any) → None

    Do nothing

    Parameters

      - text (str) –

      - kwargs (Any) –

    Return type

      None

  on_tool_end(output: Any, observation_prefix: Optional[str] = None, llm_prefix: Optional[str] = None, **kwargs: Any) → None

    Do nothing when tool ends.

    Parameters

      - output (Any) –

      - observation_prefix (Optional[str]) –

      - llm_prefix (Optional[str]) –

      - kwargs (Any) –

    Return type

      None

  on_tool_error(error: BaseException, **kwargs: Any) → None

    Do nothing when tool outputs an error.

    Parameters

      - error (BaseException) –

      - kwargs (Any) –

    Return type

      None

  on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None

    Do nothing when tool starts.

    Parameters

      - serialized (Dict[str, Any]) –

      - input_str (str) –

      - kwargs (Any) –

    Return type

      None

# Examples using ArgillaCallbackHandler

- Argilla

- argilla.md