Toggle Menu
Prev Up Next

# `langchain.evaluation.schema`.StringEvaluator

class langchain.evaluation.schema.StringEvaluator

  Grade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.

  Attributes

  `evaluation_name` | The name of the evaluation.
  `requires_input` | Whether this evaluator requires an input string.
  `requires_reference` | Whether this evaluator requires a reference label.

  Methods

  `__init__`() |
  `aevaluate_strings`(*, prediction[, ...]) | Asynchronously evaluate Chain or LLM output, based on optional input and label.
  `evaluate_strings`(*, prediction[, reference, ...]) | Evaluate Chain or LLM output, based on optional input and label.

  __init__()
  async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict

    Asynchronously evaluate Chain or LLM output, based on optional input and label.

    Parameters

      - prediction (str) – The LLM or chain prediction to evaluate.

      - reference (Optional[str], optional) – The reference label to evaluate against.

      - input (Optional[str], optional) – The input to consider during evaluation.

      - **kwargs – Additional keyword arguments, including callbacks, tags, etc.

    Returns

      The evaluation results containing the score or value.

    Return type

      dict

  evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict

    Evaluate Chain or LLM output, based on optional input and label.

    Parameters

      - prediction (str) – The LLM or chain prediction to evaluate.

      - reference (Optional[str], optional) – The reference label to evaluate against.

      - input (Optional[str], optional) – The input to consider during evaluation.

      - **kwargs – Additional keyword arguments, including callbacks, tags, etc.

    Returns

      The evaluation results containing the score or value.

    Return type

      dict

# Examples using StringEvaluator

- The perplexity is much higher since LangChain was introduced after ‘gpt-2’ was released and because it is never used in the following context.