Package com.google.cloud.aiplatform.v1beta1

Interface EvaluationParserConfig.CustomCodeParserConfigOrBuilder

All Superinterfaces:
com.google.protobuf.MessageLiteOrBuilder, com.google.protobuf.MessageOrBuilder
All Known Implementing Classes:
EvaluationParserConfig.CustomCodeParserConfig, EvaluationParserConfig.CustomCodeParserConfig.Builder
Enclosing class:
EvaluationParserConfig
public static interface EvaluationParserConfig.CustomCodeParserConfigOrBuilder extends com.google.protobuf.MessageOrBuilder

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    getParsingFunction()
    Required.
    com.google.protobuf.ByteString
    getParsingFunctionBytes()
    Required.
    boolean
    hasParsingFunction()
    Required.

    Methods inherited from interface com.google.protobuf.MessageLiteOrBuilder

    isInitialized

    Methods inherited from interface com.google.protobuf.MessageOrBuilder

    findInitializationErrors, getAllFields, getDefaultInstanceForType, getDescriptorForType, getField, getInitializationErrorString, getOneofFieldDescriptor, getRepeatedField, getRepeatedFieldCount, getUnknownFields, hasField, hasOneof

  • Method Details

    • hasParsingFunction

      boolean hasParsingFunction()
       Required. Python function for parsing results. The function should be
 defined within this string.

 The function takes a list of strings (LLM responses) and should return
 either a list of dictionaries (for rubrics) or a single dictionary
 (for a metric result).

 Example function signature:
 def parse(responses: list[str]) -> list[dict[str, Any]] | dict[str, Any]:

 When parsing rubrics, return a list of dictionaries, where each
 dictionary represents a Rubric.
 Example for rubrics:
 [
 {
 "content": {"property": {"description": "The response is
 factual."}},
 "type": "FACTUALITY",
 "importance": "HIGH"
 },
 {
 "content": {"property": {"description": "The response is
 fluent."}},
 "type": "FLUENCY",
 "importance": "MEDIUM"
 }
 ]

 When parsing critique results, return a dictionary representing a
 MetricResult.
 Example for a metric result:
 {
 "score": 0.8,
 "explanation": "The model followed most instructions.",
 "rubric_verdicts": [...]
 }

 ... code for result extraction and aggregation
      optional string parsing_function = 1 [(.google.api.field_behavior) = REQUIRED];
      Returns:
      Whether the parsingFunction field is set.

    • getParsingFunction

      String getParsingFunction()
       Required. Python function for parsing results. The function should be
 defined within this string.

 The function takes a list of strings (LLM responses) and should return
 either a list of dictionaries (for rubrics) or a single dictionary
 (for a metric result).

 Example function signature:
 def parse(responses: list[str]) -> list[dict[str, Any]] | dict[str, Any]:

 When parsing rubrics, return a list of dictionaries, where each
 dictionary represents a Rubric.
 Example for rubrics:
 [
 {
 "content": {"property": {"description": "The response is
 factual."}},
 "type": "FACTUALITY",
 "importance": "HIGH"
 },
 {
 "content": {"property": {"description": "The response is
 fluent."}},
 "type": "FLUENCY",
 "importance": "MEDIUM"
 }
 ]

 When parsing critique results, return a dictionary representing a
 MetricResult.
 Example for a metric result:
 {
 "score": 0.8,
 "explanation": "The model followed most instructions.",
 "rubric_verdicts": [...]
 }

 ... code for result extraction and aggregation
      optional string parsing_function = 1 [(.google.api.field_behavior) = REQUIRED];
      Returns:
      The parsingFunction.

    • getParsingFunctionBytes

      com.google.protobuf.ByteString getParsingFunctionBytes()
       Required. Python function for parsing results. The function should be
 defined within this string.

 The function takes a list of strings (LLM responses) and should return
 either a list of dictionaries (for rubrics) or a single dictionary
 (for a metric result).

 Example function signature:
 def parse(responses: list[str]) -> list[dict[str, Any]] | dict[str, Any]:

 When parsing rubrics, return a list of dictionaries, where each
 dictionary represents a Rubric.
 Example for rubrics:
 [
 {
 "content": {"property": {"description": "The response is
 factual."}},
 "type": "FACTUALITY",
 "importance": "HIGH"
 },
 {
 "content": {"property": {"description": "The response is
 fluent."}},
 "type": "FLUENCY",
 "importance": "MEDIUM"
 }
 ]

 When parsing critique results, return a dictionary representing a
 MetricResult.
 Example for a metric result:
 {
 "score": 0.8,
 "explanation": "The model followed most instructions.",
 "rubric_verdicts": [...]
 }

 ... code for result extraction and aggregation
      optional string parsing_function = 1 [(.google.api.field_behavior) = REQUIRED];
      Returns:
      The bytes for parsingFunction.