Toggle Menu
Prev Up Next

# `langchain.evaluation.parsing.base`.JsonValidityEvaluator

class langchain.evaluation.parsing.base.JsonValidityEvaluator(**kwargs: Any)

  Evaluate whether the prediction is valid JSON.

  This evaluator checks if the prediction is a valid JSON string. It does not

    require any input or reference.

  Parameters

    kwargs (Any) –

  requires_input

    Whether this evaluator requires an input string. Always False.

    Type

      bool

  requires_reference

    Whether this evaluator requires a reference string. Always False.

    Type

      bool

  evaluation_name

    The name of the evaluation metric. Always “json”.

    Type

      str

  Examples

      >>> evaluator = JsonValidityEvaluator()
      >>> prediction = '{"name": "John", "age": 30, "city": "New York"}'
      >>> evaluator.evaluate(prediction)
      {'score': 1}

      >>> prediction = '{"name": "John", "age": 30, "city": "New York",}'
      >>> evaluator.evaluate(prediction)
      {'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes'}

  Attributes

  `evaluation_name` | The name of the evaluation.
  `requires_input` | Whether this evaluator requires an input string.
  `requires_reference` | Whether this evaluator requires a reference label.

  Methods

  `__init__`(**kwargs) |
  `aevaluate_strings`(*, prediction[, ...]) | Asynchronously evaluate Chain or LLM output, based on optional input and label.
  `evaluate_strings`(*, prediction[, reference, ...]) | Evaluate Chain or LLM output, based on optional input and label.

  __init__(**kwargs: Any) → None
    Parameters

      kwargs (Any) –

    Return type

      None

  async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict

    Asynchronously evaluate Chain or LLM output, based on optional input and label.

    Parameters

      - prediction (str) – The LLM or chain prediction to evaluate.

      - reference (Optional[str], optional) – The reference label to evaluate against.

      - input (Optional[str], optional) – The input to consider during evaluation.

      - **kwargs – Additional keyword arguments, including callbacks, tags, etc.

    Returns

      The evaluation results containing the score or value.

    Return type

      dict

  evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict

    Evaluate Chain or LLM output, based on optional input and label.

    Parameters

      - prediction (str) – The LLM or chain prediction to evaluate.

      - reference (Optional[str], optional) – The reference label to evaluate against.

      - input (Optional[str], optional) – The input to consider during evaluation.

      - **kwargs – Additional keyword arguments, including callbacks, tags, etc.

    Returns

      The evaluation results containing the score or value.

    Return type

      dict

# Examples using JsonValidityEvaluator

- Equivalently