CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/maven-com-embabel-agent--embabel-agent-test-support

Multi-module test support framework for Embabel Agent applications providing integration testing, mock AI services, and test configuration utilities

Overview
Eval results
Files

verifying-llm-usage.mddocs/guides/

Verifying LLM Usage Guide

Step-by-step guide for verifying that your code correctly interacts with the LLM.

What is Verification?

Verification allows you to assert that the LLM was called with expected parameters. This ensures your code constructs correct prompts and uses the LLM appropriately.

Prerequisites

  • Test class extends EmbabelMockitoIntegrationTest
  • Code has been executed that may call the LLM
  • Understanding of predicates and assertions

Basic Verification Pattern

Step 1: Execute Your Code

myAgent.process("user input");

Step 2: Verify the LLM Call

// Verify text generation
verifyGenerateText(prompt -> prompt.contains("expected"));

// Verify object creation
verifyCreateObject(prompt -> prompt.contains("expected"), OutputClass.class);

If verification fails, an AssertionError is thrown with details about the mismatch.

Verifying Text Generation

Simple Verification

@Test
void testLlmWasCalled() {
    // Execute code
    myAgent.summarize(document);

    // Verify LLM was called with prompt containing "summarize"
    verifyGenerateText(prompt -> prompt.contains("summarize"));
}

Multiple Verifications

@Test
void testMultipleLlmCalls() {
    // Execute code that makes multiple LLM calls
    myAgent.multiStepProcess();

    // Verify first call
    verifyGenerateText(p -> p.contains("step 1"));

    // Verify second call
    verifyGenerateText(p -> p.contains("step 2"));

    // Ensure no additional calls were made
    verifyNoMoreInteractions();
}

Complex Predicate Verification

@Test
void testComplexVerification() {
    myAgent.analyzeData(dataset);

    // Verify with multiple conditions
    verifyGenerateText(prompt ->
        prompt.contains("analyze") &&
        prompt.contains(dataset.getName()) &&
        prompt.length() > 100
    );
}

Verifying Object Creation

Simple Object Verification

@Test
void testObjectExtraction() {
    // Execute code that creates object from LLM
    myAgent.extractPerson(text);

    // Verify LLM was asked to create Person object
    verifyCreateObject(
        prompt -> prompt.contains("extract person"),
        Person.class
    );
}

Multiple Object Verifications

@Test
void testMultipleObjects() {
    // Execute code that extracts multiple objects
    myAgent.extractData(text);

    // Verify person extraction
    verifyCreateObject(p -> p.contains("person"), Person.class);

    // Verify address extraction
    verifyCreateObject(p -> p.contains("address"), Address.class);
}

Advanced Verification Techniques

Verify with Interaction Details

Verify both prompt and model configuration:

@Test
void testWithInteractionDetails() {
    myAgent.preciseModeAnalysis();

    // Verify prompt and model settings
    verifyGenerateText(
        prompt -> prompt.contains("precise"),
        interaction ->
            interaction.getModel().equals("gpt-4") &&
            interaction.getTemperature() == 0.0
    );
}

Verify No LLM Calls

Ensure code path doesn't use LLM:

@Test
void testCachedPath() {
    // Execute cached code path
    myAgent.getCachedResult("key");

    // Verify no LLM interactions occurred
    verifyNoInteractions();
}

Verify No Additional Calls

Ensure only expected calls were made:

@Test
void testOnlyExpectedCalls() {
    myAgent.singleOperation();

    // Verify the expected call
    verifyGenerateText(p -> p.contains("operation"));

    // Verify no other calls were made
    verifyNoMoreInteractions();
}

Capturing and Inspecting Arguments

Capture LLM Interaction

@Test
void testCaptureInteraction() {
    myAgent.process("input");

    // Capture the LlmInteraction
    ArgumentCaptor<LlmInteraction> captor = captureLlmInteraction();
    verifyGenerateText(p -> true);

    // Inspect captured interaction
    LlmInteraction interaction = captor.getValue();
    assertEquals("gpt-4", interaction.getModel());
    assertEquals(0.7, interaction.getTemperature());
    assertTrue(interaction.getMaxTokens() > 0);
}

Capture Prompt

@Test
void testCapturePrompt() {
    myAgent.generateText("user query");

    // Capture the actual prompt
    ArgumentCaptor<String> promptCaptor = capturePrompt();
    // Use Mockito directly for capturing
    verify(llmOperations).generateText(
        argThat(messages -> {
            promptCaptor.capture();
            return true;
        }),
        any()
    );

    // Inspect captured prompt
    String actualPrompt = promptCaptor.getValue();
    assertTrue(actualPrompt.contains("user query"));
    assertTrue(actualPrompt.contains("system instructions"));
}

Multiple Captures

@Test
void testMultipleCaptures() {
    myAgent.processMultiple();

    ArgumentCaptor<LlmInteraction> captor = captureLlmInteraction();

    // Verify multiple calls and capture all
    verify(llmOperations, times(2)).generateText(any(), captor.capture());

    // Inspect all captured interactions
    List<LlmInteraction> allInteractions = captor.getAllValues();
    assertEquals(2, allInteractions.size());

    allInteractions.forEach(interaction -> {
        assertNotNull(interaction.getModel());
    });
}

Kotlin Examples

Simple Verification (Kotlin)

@Test
fun `test LLM was called`() {
    myAgent.summarize(document)

    // Kotlin lambda syntax
    verifyGenerateText { it.contains("summarize") }
}

Object Verification (Kotlin)

@Test
fun `test object extraction`() {
    myAgent.extractPerson(text)

    verifyCreateObject({ it.contains("extract") }, Person::class.java)
}

No Interactions (Kotlin)

@Test
fun `test cached path`() {
    myAgent.getCached()

    verifyNoInteractions()
}

Common Verification Patterns

Pattern 1: Verify Call Count

// Verify called exactly once (default)
verifyGenerateText(p -> p.contains("text"));

// Verify called N times using Mockito directly
verify(llmOperations, times(3)).generateText(any(), any());

// Verify never called
verify(llmOperations, never()).generateText(any(), any());

Pattern 2: Verify Order

@Test
void testCallOrder() {
    InOrder inOrder = inOrder(llmOperations);

    myAgent.sequentialProcess();

    // Verify calls happened in order
    inOrder.verify(llmOperations).generateText(
        argThat(messages -> /* first call predicate */),
        any()
    );
    inOrder.verify(llmOperations).generateText(
        argThat(messages -> /* second call predicate */),
        any()
    );
}

Pattern 3: Verify Any Call

@Test
void testAnyCall() {
    myAgent.process();

    // Verify LLM was called with any prompt
    verifyGenerateText(p -> true);
}

Pattern 4: Partial Match

@Test
void testPartialMatch() {
    myAgent.search("query");

    // Verify prompt contains expected parts
    verifyGenerateText(p ->
        p.contains("search") &&
        p.contains("query")
    );
}

Troubleshooting

Verification Fails

Problem: Verification throws AssertionError even though code seems correct.

Solutions:

  1. Check predicate: Ensure predicate matches actual prompt

    // Debug: Print actual prompt
verifyGenerateText(p -> {
    System.out.println("Actual prompt: " + p);
    return p.contains("expected");
});

  2. Check operation type: Ensure verifying correct operation (text vs object)

    // Wrong: Code calls createObject but verifying generateText
verifyGenerateText(p -> true);  // Fails

// Correct
verifyCreateObject(p -> true, Person.class);

  3. Check call count: Code might call LLM multiple times

    // Use Mockito's times() to verify count
verify(llmOperations, times(2)).generateText(any(), any());

Unexpected Calls

Problem: verifyNoMoreInteractions() fails with unexpected calls.

Solution: Use captors to see what calls were made:

ArgumentCaptor<List> messageCaptor = ArgumentCaptor.forClass(List.class);
verify(llmOperations, atLeastOnce()).generateText(
    messageCaptor.capture(),
    any()
);

// Inspect all captured calls
messageCaptor.getAllValues().forEach(messages -> {
    System.out.println("Call: " + messages);
});

Wrong Verification Method

Problem: Using wrong verification method for operation type.

Solution:

Your Code CallsUse This Verification
llmOperations.generateText(...)verifyGenerateText()
llmOperations.createObject(...)verifyCreateObject()

Best Practices

  1. Verify After Stub: Always pair stubs with verification

    whenGenerateText(p -> p.contains("test")).thenReturn("result");
myAgent.test();
verifyGenerateText(p -> p.contains("test"));  // Verify it was used

  2. Use Specific Predicates: Match exactly what you expect

    // Good: Specific
verifyGenerateText(p ->
    p.contains("user: Alice") &&
    p.contains("action: analyze")
);

// Bad: Too vague
verifyGenerateText(p -> p.length() > 0);

  3. Verify No Unexpected Calls: End tests with verifyNoMoreInteractions() when appropriate

    verifyGenerateText(p -> p.contains("expected"));
verifyNoMoreInteractions();  // Ensure nothing else was called

  4. Test Negative Cases: Verify LLM is NOT called when it shouldn't be

    @Test
void testCachingWorks() {
    myAgent.getCached("key");
    verifyNoInteractions();  // Cache hit, no LLM call
}

  5. Use Constants: Define reusable predicates

    private static final Predicate<String> CONTAINS_ANALYZE =
    p -> p.contains("analyze");

@Test
void test() {
    myAgent.analyze();
    verifyGenerateText(CONTAINS_ANALYZE);
}

Combining with Assertions

Verification confirms LLM was called correctly, but also assert on results:

@Test
void testComplete() {
    // Stub
    whenGenerateText(p -> p.contains("summarize"))
        .thenReturn("Summary");

    // Execute
    String result = myAgent.summarize(doc);

    // Verify LLM call
    verifyGenerateText(p -> p.contains("summarize"));

    // Assert result
    assertEquals("Summary", result);
    assertNotNull(result);
    assertTrue(result.length() > 0);
}

Next Steps

  • Stubbing Guide - Learn to stub LLM calls
  • Testing Patterns - See complete patterns
  • Verification API - Complete API reference

Related Topics

tessl i tessl/maven-com-embabel-agent--embabel-agent-test-support@0.3.0

docs

guides

spring-test-setup.md

stubbing-llm-calls.md

testing-embeddings.md

testing-options-converters.md

verifying-llm-usage.md

index.md

tile.json