or run

tessl search
Log in

Version

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
npmpkg:npm/langsmith@0.4.x

docs

guides

anti-patterns.mddecision-trees.mderror-handling.mdevaluation-comparative.mdevaluation.mdindex.mdintegration-patterns.mdquick-reference.mdsetup.mdtesting.mdtracing.mdworkflows.md
index.md
tile.json

tessl/npm-langsmith

tessl install tessl/npm-langsmith@0.4.3

TypeScript client SDK for the LangSmith LLM tracing, evaluation, and monitoring platform.

testing.mddocs/guides/

Testing

LangSmith provides seamless integration with Jest and Vitest testing frameworks, enabling test-driven evaluation workflows.

Overview

The testing integrations extend familiar testing APIs with LangSmith-specific features:

  • Automatic Dataset Creation: Tests create dataset examples automatically
  • Custom Matchers: LLM-specific assertions for semantic similarity
  • Feedback Logging: Track evaluation metrics during test execution
  • Evaluator Wrappers: Create reusable evaluation logic
  • Experiment Tracking: Test runs create experiments in LangSmith

When to use testing integration:

  • Build test suites for LLM applications
  • Create evaluation datasets through testing
  • Regression testing for model outputs
  • CI/CD integration with quality gates

Choose Your Framework

LangSmith provides identical APIs for both Jest and Vitest.

Jest Integration

Use if you're already using Jest or prefer its ecosystem.

import { test, expect } from "langsmith/jest";

test(
  "greeting generation",
  {
    input: { name: "Alice" },
    expected: { greeting: "Hello, Alice!" }
  },
  async (input) => {
    return { greeting: `Hello, ${input.name}!` };
  }
);

Complete Jest documentation

Vitest Integration

Use if you're using Vite/Vitest or want faster test execution.

import { test, expect } from "langsmith/vitest";

test(
  "summarize text correctly",
  {
    input: { text: "Long document..." },
    expected: { summary: "Summary" }
  },
  async (input) => {
    return await summarizeText(input.text);
  }
);

Vitest requires reporter configuration:

// vitest.config.ts
import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    reporters: ["default", "langsmith/vitest/reporter"]
  }
});

Complete Vitest documentation

Comparison: Jest vs Vitest

FeatureJestVitest
Test APIIdenticalIdentical
Custom MatchersYesYes
Reporter ConfigNot requiredRequired
PerformanceGoodFaster
ES ModulesRequires configNative
TypeScriptRequires ts-jestNative

Recommendation:

  • Use Jest if you already use Jest or have Jest-based CI/CD pipelines
  • Use Vitest if you use Vite, want faster tests, or prefer native ES modules

Shared Features

Both frameworks provide:

Custom Matchers

// Relative closeness (normalized edit distance)
expect(output).toBeRelativeCloseTo("Expected text", { threshold: 0.8 });

// Absolute closeness (raw edit distance)
expect(output).toBeAbsoluteCloseTo("Expected text", { threshold: 5 });

// Semantic similarity (embeddings-based)
expect(output).toBeSemanticCloseTo("Expected meaning", { threshold: 0.85 });

// Custom evaluators
expect(output).evaluatedBy(customEvaluator);

Feedback Logging

import { test, logFeedback, logOutputs } from "langsmith/jest";  // or vitest

test(
  "with feedback",
  { input: { text: "test" } },
  async (input) => {
    const result = await process(input.text);

    logFeedback({
      key: "quality",
      score: 0.95,
      comment: "High quality"
    });

    logOutputs({ step1: result });

    return result;
  }
);

Reusable Evaluators

import { wrapEvaluator } from "langsmith/jest";  // or vitest

const customEvaluator = wrapEvaluator((args) => {
  const { input, output, expected } = args;
  return {
    key: "custom_metric",
    score: calculateScore(output, expected),
    comment: "Evaluation comment"
  };
});

test(
  "with custom evaluator",
  {
    input: "test",
    evaluators: [customEvaluator]
  },
  async (input) => {
    const result = await process(input);
    expect(result).evaluatedBy(customEvaluator);
    return result;
  }
);

Quick Examples

Basic Test

import { test } from "langsmith/jest";  // or vitest

test(
  "test name",
  {
    input: { data: "input" },
    expected: { result: "output" }
  },
  async (input) => {
    return { result: processData(input.data) };
  }
);

With Dataset Integration

test(
  "translation quality",
  {
    input: { text: "Hello", lang: "es" },
    expected: "Hola",
    datasetName: "translation-tests",
    projectName: "translation-eval"
  },
  async (input) => {
    return await translate(input.text, input.lang);
  }
);

Test Suite

import { describe, test } from "langsmith/jest";  // or vitest

describe("LLM Response Generation", () => {
  test(
    "should generate greeting",
    {
      input: { name: "Bob" },
      expected: { message: "Hello, Bob!" }
    },
    async (input) => {
      return { message: `Hello, ${input.name}!` };
    }
  );

  test(
    "should generate farewell",
    {
      input: { name: "Bob" },
      expected: { message: "Goodbye, Bob!" }
    },
    async (input) => {
      return { message: `Goodbye, ${input.name}!` };
    }
  );
});

Related Documentation