Verification API

Complete API reference for verifying LLM operations in tests.

Overview

Verification methods assert that the mocked LLM was called with expected parameters. Use these to verify your code interacts with the LLM correctly.

Module: embabel-agent-test Class: EmbabelMockitoIntegrationTest Import: import com.embabel.agent.test.integration.EmbabelMockitoIntegrationTest;

Text Generation Verification

verifyGenerateText (Simple)

Verify LLM text generation with prompt matching.

protected void verifyGenerateText(
    Predicate<String> promptMatcher
)

Parameters:

• promptMatcher - Predicate to match against the prompt text

Throws: AssertionError if verification fails

Usage:

verifyGenerateText(prompt -> prompt.contains("summarize"));

When to use: Most common case - verify based on prompt content only.

verifyGenerateText (With Interaction)

Verify LLM text generation with both prompt and interaction matching.

protected void verifyGenerateText(
    Predicate<String> promptMatcher,
    Predicate<LlmInteraction> llmInteractionMatcher
)

Parameters:

• promptMatcher - Predicate to match against the prompt text
• llmInteractionMatcher - Predicate to match LlmInteraction details

Throws: AssertionError if verification fails

Usage:

verifyGenerateText(
    prompt -> prompt.contains("analyze"),
    interaction -> interaction.getModel().equals("gpt-4")
);

When to use: When you need to verify both prompt and model configuration.

verifyGenerateTextMatching (Predicate)

Verify text generation with prompt matching (alternative signature).

protected void verifyGenerateTextMatching(
    Predicate<String> promptMatcher
)

Parameters:

• promptMatcher - Predicate to match against the prompt text

Throws: AssertionError if verification fails

Usage:

verifyGenerateTextMatching(prompt -> prompt.startsWith("Generate"));

When to use: Alternative to verifyGenerateText with same functionality.

verifyGenerateTextMatching (With Specific Interaction)

Verify text generation with specific LlmInteraction instance.

protected void verifyGenerateTextMatching(
    Predicate<String> promptMatcher,
    LlmInteraction llmInteraction
)

Parameters:

• promptMatcher - Predicate to match against the prompt text
• llmInteraction - Expected LlmInteraction instance

Throws: AssertionError if verification fails

Usage:

LlmInteraction expectedInteraction = new LlmInteraction(...);
verifyGenerateTextMatching(
    prompt -> prompt.contains("text"),
    expectedInteraction
);

When to use: When you have a specific interaction instance to match against.

Object Creation Verification

verifyCreateObject (Simple)

Verify LLM object creation with prompt and class matching.

protected <T> void verifyCreateObject(
    Predicate<String> promptMatcher,
    Class<T> outputClass
)

Type Parameters:

• T - The type of object being created

Parameters:

• promptMatcher - Predicate to match against the prompt text
• outputClass - Expected output class type

Throws: AssertionError if verification fails

Usage:

verifyCreateObject(
    prompt -> prompt.contains("extract"),
    Person.class
);

When to use: Most common case - verify based on prompt and output type.

verifyCreateObject (With Interaction)

Verify LLM object creation with prompt, class, and interaction matching.

protected <T> void verifyCreateObject(
    Predicate<String> promptMatcher,
    Class<T> outputClass,
    Predicate<LlmInteraction> llmInteractionMatcher
)

Type Parameters:

• T - The type of object being created

Parameters:

• promptMatcher - Predicate to match against the prompt text
• outputClass - Expected output class type
• llmInteractionMatcher - Predicate to match LlmInteraction details

Throws: AssertionError if verification fails

Usage:

verifyCreateObject(
    prompt -> prompt.contains("extract"),
    Person.class,
    interaction -> interaction.getTemperature() == 0.0
);

When to use: When you need to verify prompt, type, and model configuration.

verifyCreateObjectMatching (ArgumentMatcher)

Verify object creation with custom ArgumentMatcher for complex validation.

protected <T> void verifyCreateObjectMatching(
    Predicate<String> promptMatcher,
    Class<T> outputClass,
    ArgumentMatcher<LlmInteraction> llmInteractionMatcher
)

Type Parameters:

• T - The type of object being created

Parameters:

• promptMatcher - Predicate to match against the prompt text
• outputClass - Expected output class type
• llmInteractionMatcher - ArgumentMatcher for complex LlmInteraction validation

Throws: AssertionError if verification fails

Usage:

verifyCreateObjectMatching(
    prompt -> prompt.contains("data"),
    DataObject.class,
    interaction -> interaction.getToolGroups().size() > 2
);

When to use: When you need complex validation logic for interactions.

verifyCreateObjectMatchingMessages

Verify object creation with message list matching.

protected <T> void verifyCreateObjectMatchingMessages(
    ArgumentMatcher<List<Message>> promptMatcher,
    Class<T> outputClass,
    ArgumentMatcher<LlmInteraction> llmInteractionMatcher
)

Type Parameters:

• T - The type of object being created

Parameters:

• promptMatcher - ArgumentMatcher for matching message lists
• outputClass - Expected output class type
• llmInteractionMatcher - ArgumentMatcher for LlmInteraction validation

Throws: AssertionError if verification fails

Usage:

verifyCreateObjectMatchingMessages(
    messages -> messages.size() == 1 && messages.get(0).getRole().equals("user"),
    Result.class,
    interaction -> interaction.getModel().startsWith("gpt")
);

When to use: When you need to verify the complete message structure.

Interaction Verification

verifyNoInteractions

Verify that no interactions occurred with the mocked LLM.

protected void verifyNoInteractions()

Throws: AssertionError if any interactions occurred

Usage:

verifyNoInteractions();

When to use: To verify code path didn't use LLM (e.g., cached results).

verifyNoMoreInteractions

Verify that no additional interactions occurred beyond already verified ones.

protected void verifyNoMoreInteractions()

Throws: AssertionError if unverified interactions exist

Usage:

verifyGenerateText(p -> p.contains("test"));
verifyNoMoreInteractions();

When to use: To ensure no unexpected LLM calls were made.

Argument Capture

capturePrompt

Create an ArgumentCaptor for capturing prompt strings.

protected ArgumentCaptor<String> capturePrompt()

Returns: ArgumentCaptor<String> for capturing prompts

Usage:

ArgumentCaptor<String> captor = capturePrompt();
verify(llmOperations).generateText(captor.capture(), any());
String capturedPrompt = captor.getValue();
assertTrue(capturedPrompt.contains("expected"));

When to use: When you need to inspect the actual prompt text.

captureLlmInteraction

Create an ArgumentCaptor for capturing LlmInteraction instances.

protected ArgumentCaptor<LlmInteraction> captureLlmInteraction()

Returns: ArgumentCaptor<LlmInteraction> for capturing interactions

Usage:

ArgumentCaptor<LlmInteraction> captor = captureLlmInteraction();
// ... perform verification ...
LlmInteraction interaction = captor.getValue();
assertEquals("gpt-4", interaction.getModel());
assertEquals(0.7, interaction.getTemperature());

When to use: When you need to inspect interaction details.

captureOutputClass

Create an ArgumentCaptor for capturing output class types.

protected <T> ArgumentCaptor<Class<T>> captureOutputClass()

Type Parameters:

• T - The type of class being captured

Returns: ArgumentCaptor<Class<T>> for capturing class types

Usage:

ArgumentCaptor<Class<?>> captor = captureOutputClass();
// ... perform verification ...
Class<?> capturedClass = captor.getValue();
assertEquals(Person.class, capturedClass);

When to use: When you need to verify the output class type.

Complete Examples

Example 1: Simple Verification

@Test
void testSimpleVerification() {
    myAgent.process("input");

    verifyGenerateText(prompt -> prompt.contains("input"));
}

Example 2: Verify Multiple Calls

@Test
void testMultipleCalls() {
    myAgent.multiStep();

    verifyGenerateText(p -> p.contains("step1"));
    verifyGenerateText(p -> p.contains("step2"));
    verifyNoMoreInteractions();
}

Example 3: Verify with Interaction Details

@Test
void testWithInteraction() {
    myAgent.preciseAnalysis();

    verifyGenerateText(
        p -> p.contains("analyze"),
        i -> i.getModel().equals("gpt-4") && i.getTemperature() == 0.0
    );
}

Example 4: Verify Object Creation

@Test
void testObjectCreation() {
    myAgent.extractData();

    verifyCreateObject(
        p -> p.contains("extract"),
        Person.class
    );
}

Example 5: Capture and Inspect

@Test
void testCapture() {
    myAgent.process("test input");

    ArgumentCaptor<LlmInteraction> captor = captureLlmInteraction();
    verifyGenerateText(p -> true);

    LlmInteraction interaction = captor.getValue();
    assertEquals("gpt-4", interaction.getModel());
    assertTrue(interaction.getMaxTokens() > 0);
}

Example 6: Verify No Interactions

@Test
void testCachedPath() {
    myAgent.getCached();

    verifyNoInteractions();
}

Example 7: Complex Predicate

@Test
void testComplexPredicate() {
    myAgent.complexOperation();

    verifyGenerateText(prompt ->
        prompt.contains("analyze") &&
        prompt.length() > 50 &&
        prompt.toLowerCase().contains("data")
    );
}

Verification Modes

Mockito provides verification modes that can be used with these methods:

import static org.mockito.Mockito.*;

// Verify called exactly once (default)
verifyGenerateText(p -> true);

// Verify called N times
verify(llmOperations, times(2)).generateText(any(), any());

// Verify never called
verify(llmOperations, never()).generateText(any(), any());

// Verify called at least once
verify(llmOperations, atLeastOnce()).generateText(any(), any());

// Verify called at most N times
verify(llmOperations, atMost(3)).generateText(any(), any());

Note: For direct Mockito verification on llmOperations, you'll need to import Mockito's verify and use argThat() with predicates.

Related APIs

• Stubbing API - Stub LLM responses
• Verification Guide - Detailed guide with patterns

Types

Predicate<T>

@FunctionalInterface
interface Predicate<T> {
    boolean test(T t);
}

Functional interface for matching conditions.

ArgumentCaptor<T>

class ArgumentCaptor<T> {
    T getValue();
    List<T> getAllValues();
}

Mockito utility for capturing arguments.

ArgumentMatcher<T>

@FunctionalInterface
interface ArgumentMatcher<T> {
    boolean matches(T argument);
}

Custom argument matching interface.

LlmInteraction

class LlmInteraction {
    String getModel();
    Double getTemperature();
    Integer getMaxTokens();
    List<String> getToolGroups();
}

Configuration for LLM interaction.

Message

interface Message {
    String getContent();
    String getRole();
}

Chat message interface.