{ "date": "Aug 7, 2024", "title": "Construction Derby: Structured Data Generation with JSON Mode", "company": "Snowglobe", "images": [ "https://framerusercontent.com/images/pXl4tOQx1tL5L5dMOtPEzZ2xv8.png?width=2000&height=919" ], "author_note": "This article was written prior to the [public availability of Structured Outputs in OpenAI](https://openai.com/index/introducing-structured-outputs-in-the-api/) and was originally scheduled for release on the day of the announcement. The information remains relevant for open source models, but there is [more to consider now with OpenAI's offering](https://x.com/guardrails_ai/status/1821934378383274477).", "introduction": "When making orange juice one can choose from a fair variety of extraction methods. Each option has differences in yield, quality of output, effort required, and failure modes, among other things. Deconstructing an orange into its constituent particles represents a significant investment of effort and a substantial likelihood of a sternly worded letter from the local nuclear regulatory agency [1](/content/blog/json-mode-all-i-want-is-structured-data#user-content-fn-1/index.html), but yields an arguably pure product. Placing an untouched orange in a glass is arguably the minimal amount of effort but leads to a number of philosophical questions about the definition of orange juice. This could be considered “full pulp”. Nearly all of engineering is choosing between tradeoffs. Picking structured data generation options is no exception.", "evaluation_note": "Natural language is messy, and being able to perform a comparison that's fair, informative, and tractable for a short project is its own challenge. The Language-Independent Named Entity Recognition task has a well publicized methodology for evaluating the performance of models. We make two simplifying assumptions for our evaluation: first, for the NER task, the label of the recovered entity is ignored. Second, we perform a fuzzy match between the thresholded embeddings of the strings.", "evaluation_examples": [ { "description": "These two nearly-identical sets have a score of 0.963:", "table": [ { "Score": 0.963, "Simpsons": {"The Simpsons": 0.8551, "Springfield": 0.4576, "Homer": 0.6699, "Marge": 0.638} } ] }, { "description": "These two completely different examples have a score of 0.0:", "table": [ { "Score": 0.0, "Jimmy Hendricks": {"Rick Astley": 0.2891, "Lancashire": 0.1819, "England": 0.0992} } ] } ], "dataset_construction": "To best cover the gamut of data qualities, ranging from “near perfect” to “acceptable”, we sampled from three places: a hand-written artificial dataset made specifically to be easy to parse into structured data, a small set of randomly chosen Wikipedia pages, and a random selection of Guardian news articles. The “golden” dataset gives us an upper-bounds for what can be expected for a model. Names are unambiguous, quantities are well known, and events are distinct.", "plain_pod_prompting": "We begin our benchmarking with the simplest thing that could possibly work: asking nicely. Plain-old-prompting solicits a model for JSON output and provides a schema for consideration. The benefits of this approach are fairly numerous: there are no limitations on the choice of LLM providers and there are no additional parameters to typo in the API call.", "conclusions": "GPT-4 Turbo remains the leader of the pack for structured data extraction, though at $5.00 per million input tokens and $15 per million output tokens, it’s also the most expensive and highest latency option. Our overall recommendation is GPT-4o Mini for the combination of high quality output and low latency in processing." }