Skip to main content

How to return structured data from a model

It is often useful to have a model return output that matches some specific schema. One common use-case is extracting data from arbitrary text to insert into a traditional database or use with some other downstrem system. This guide will show you a few different strategies you can use to do this.

Prerequisites

This guide assumes familiarity with the following concepts:

The .withStructuredOutput() methodโ€‹

There are several strategies that models can use under the hood. For some of the most popular model providers, including Anthropic, Google VertexAI, Mistral, and OpenAI LangChain implements a common interface that abstracts away these strategies called .withStructuredOutput.

By invoking this method (and passing in JSON schema or a Zod schema) the model will add whatever model parameters + output parsers are necessary to get back structured output matching the requested schema. If the model supports more than one way to do this (e.g., function calling vs JSON mode) - you can configure which method to use by passing into that method.

Letโ€™s look at some examples of this in action! Weโ€™ll use Zod to create a simple response schema.

Pick your chat model:

Install dependencies

yarn add @langchain/openai 

Add environment variables

OPENAI_API_KEY=your-api-key

Instantiate the model

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
model: "gpt-3.5-turbo",
temperature: 0
});
import { z } from "zod";

const joke = z.object({
setup: z.string().describe("The setup of the joke"),
punchline: z.string().describe("The punchline to the joke"),
rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});

const structuredLlm = model.withStructuredOutput(joke);

await structuredLlm.invoke("Tell me a joke about cats");
{
setup: "Why don't cats play poker in the wild?",
punchline: "Too many cheetahs.",
rating: 7
}

One key point is that though we set our Zod schema as a variable named joke, Zod is not able to access that variable name, and therefore cannot pass it to the model. Though it is not required, we can pass a name for our schema in order to give the model additional context as to what our schema represents, improving performance:

const structuredLlm = model.withStructuredOutput(joke, { name: "joke" });

await structuredLlm.invoke("Tell me a joke about cats");
{
setup: "Why don't cats play poker in the wild?",
punchline: "Too many cheetahs!",
rating: 7
}

The result is a JSON object.

We can also pass in an OpenAI-style JSON schema dict if you prefer not to use Zod. This object should contain three properties:

  • name: The name of the schema to output.
  • description: A high level description of the schema to output.
  • parameters: The nested details of the schema you want to extract, formatted as a JSON schema dict.

In this case, the response is also a dict:

const structuredLlm = model.withStructuredOutput({
name: "joke",
description: "Joke to tell user.",
parameters: {
title: "Joke",
type: "object",
properties: {
setup: { type: "string", description: "The setup for the joke" },
punchline: { type: "string", description: "The joke's punchline" },
},
required: ["setup", "punchline"],
},
});

await structuredLlm.invoke("Tell me a joke about cats", { name: "joke" });
{
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!"
}

If you are using JSON Schema, you can take advantage of other more complex schema descriptions to create a similar effect.

You can also use tool calling directly to allow the model to choose between options, if your chosen model supports it. This involves a bit more parsing and setup. See this how-to guide for more details.

Specifying the output method (Advanced)โ€‹

For models that support more than one means of outputting data, you can specify the preferred one like this:

const structuredLlm = model.withStructuredOutput(joke, {
method: "json_mode",
name: "joke",
});

await structuredLlm.invoke(
"Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys"
);
{
setup: "Why don't cats play poker in the jungle?",
punchline: "Too many cheetahs!"
}

In the above example, we use OpenAIโ€™s alternate JSON mode capability along with a more specific prompt.

For specifics about the model you choose, peruse its entry in the API reference pages.

(Advanced) Raw outputsโ€‹

LLMs arenโ€™t perfect at generating structured output, especially as schemas become complex. You can avoid raising exceptions and handle the raw output yourself by passing includeRaw: true. This changes the output format to contain the raw message output and the parsed value (if successful):

const joke = z.object({
setup: z.string().describe("The setup of the joke"),
punchline: z.string().describe("The punchline to the joke"),
rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});

const structuredLlm = model.withStructuredOutput(joke, {
includeRaw: true,
name: "joke",
});

await structuredLlm.invoke("Tell me a joke about cats");
{
raw: AIMessage {
lc_serializable: true,
lc_kwargs: {
content: "",
tool_calls: [
{
name: "joke",
args: [Object],
id: "call_0pEdltlfSXjq20RaBFKSQOeF"
}
],
invalid_tool_calls: [],
additional_kwargs: { function_call: undefined, tool_calls: [ [Object] ] },
response_metadata: {}
},
lc_namespace: [ "langchain_core", "messages" ],
content: "",
name: undefined,
additional_kwargs: {
function_call: undefined,
tool_calls: [
{
id: "call_0pEdltlfSXjq20RaBFKSQOeF",
type: "function",
function: [Object]
}
]
},
response_metadata: {
tokenUsage: { completionTokens: 33, promptTokens: 88, totalTokens: 121 },
finish_reason: "stop"
},
tool_calls: [
{
name: "joke",
args: {
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!",
rating: 7
},
id: "call_0pEdltlfSXjq20RaBFKSQOeF"
}
],
invalid_tool_calls: [],
usage_metadata: { input_tokens: 88, output_tokens: 33, total_tokens: 121 }
},
parsed: {
setup: "Why was the cat sitting on the computer?",
punchline: "Because it wanted to keep an eye on the mouse!",
rating: 7
}
}

Prompting techniquesโ€‹

You can also prompt models to outputting information in a given format. This approach relies on designing good prompts and then parsing the output of the models. This is the only option for models that donโ€™t support .with_structured_output() or other built-in approaches.

Using JsonOutputParserโ€‹

The following example uses the built-in JsonOutputParser to parse the output of a chat model prompted to match a the given JSON schema. Note that we are adding format_instructions directly to the prompt from a method on the parser:

import { JsonOutputParser } from "@langchain/core/output_parsers";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
name: string;
height_in_meters: number;
};

type People = {
people: Person[];
};

const formatInstructions = `Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.
`;

// Set up a parser
const parser = new JsonOutputParser<People>();

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
[
"system",
"Answer the user query. Wrap the output in `json` tags\n{format_instructions}",
],
["human", "{query}"],
]).partial({
format_instructions: formatInstructions,
});

Letโ€™s take a look at what information is sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Wrap the output in `json` tags
Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.

Human: Anna is 23 years old and she is 6 feet tall

And now letโ€™s invoke it:

const chain = prompt.pipe(model).pipe(parser);

await chain.invoke({ query });
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }

For a deeper dive into using output parsers with prompting techniques for structured output, see this guide.

Custom Parsingโ€‹

You can also create a custom prompt and parser with LangChain Expression Language (LCEL), using a plain function to parse the output from the model:

import { AIMessage } from "@langchain/core/messages";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
name: string;
height_in_meters: number;
};

type People = {
people: Person[];
};

const schema = `{{ people: [{{ name: "string", height_in_meters: "number" }}] }}`;

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
[
"system",
`Answer the user query. Output your answer as JSON that
matches the given schema: \`\`\`json\n{schema}\n\`\`\`.
Make sure to wrap the answer in \`\`\`json and \`\`\` tags`,
],
["human", "{query}"],
]).partial({
schema,
});

/**
* Custom extractor
*
* Extracts JSON content from a string where
* JSON is embedded between ```json and ``` tags.
*/
const extractJson = (output: AIMessage): Array<People> => {
const text = output.content as string;
// Define the regular expression pattern to match JSON blocks
const pattern = /```json(.*?)```/gs;

// Find all non-overlapping matches of the pattern in the string
const matches = text.match(pattern);

// Process each match, attempting to parse it as JSON
try {
return (
matches?.map((match) => {
// Remove the markdown code block syntax to isolate the JSON string
const jsonStr = match.replace(/```json|```/g, "").trim();
return JSON.parse(jsonStr);
}) ?? []
);
} catch (error) {
throw new Error(`Failed to parse: ${output}`);
}
};

Here is the prompt sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Output your answer as JSON that
matches the given schema: ```json
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}
```.
Make sure to wrap the answer in ```json and ``` tags
Human: Anna is 23 years old and she is 6 feet tall

And hereโ€™s what it looks like when we invoke it:

import { RunnableLambda } from "@langchain/core/runnables";

const chain = prompt
.pipe(model)
.pipe(new RunnableLambda({ func: extractJson }));

await chain.invoke({ query });
[
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }
]

Next stepsโ€‹

Now youโ€™ve learned a few methods to make a model output structured data.

To learn more, check out the other how-to guides in this section, or the conceptual guide on tool calling.


Was this page helpful?


You can also leave detailed feedback on GitHub.