Introduction

Synthetic data generation has advanced significantly, with larger language models now being leveraged for knowledge distillation to produce high-quality training examples. It enables the creation of domain-specific datasets that can be used to finetune smaller or specialised models while preserving accuracy, lowering computational costs, and reducing existing data requirements.

For instance, this approach enables efficient finetuning of GPT-4o, Llama etc. for specific use cases with just a few synthetic examples, or training specialized ModernBERT models for classification tasks within your agentic workflow - achieving better performance on downstream tasks while requiring only a fraction of the computing resources. This is particularly valuable for high-throughput, latency-sensitive applications like LLM routing or real-time classification tasks, especially in agentic workflows where multiple models are used in sequence - each round of model inference compounds the efficiency benefits and reduces overall latency.

---

---

---

Building Synthetic Datasets: A Practical Example

We believe that intent recognition is a fundamental component of smart, AI-powered user experiences. That’s why we’re showcasing its potential through synthetic data generation. You can explore a minimal, straightforward example on Google Colab, which demonstrates how synthetic data can be generated for intent recognition.

Tools and libraries make things easier by obfuscating some parts of the execution, but at the same time, they may make it harder to understand what happens behind the scenes. For that reason, we wanted to create the simplest and cleanest example possible, making it accessible even to those with less experience in the synthetic data space.

In a nutshell, in this code:

1. We install dependencies, instructor for structured generation, openai to be used as a client for generation, and some other helpers
2. Define the intents or labels we want to create queries for
3. Use Pydantic class to define our output structure
4. Create generate_queries function which takes in intent (str) and outputs 20 queries for intent, also define generate_dataset function which takes list of intents and generates queries for each and outputs list of dicts which contains query, text and some metadata
5. We turn our raw generated dataset into Huggingface Dataset and push it to Huggingface

Output dataset can be found here

We challenge you to replicate this using Curator or Distilabel and let us know which one you preferred and why.

---

Looking Forward: The Promise of Small Language Models

There seems to be a trend emerging: the shift toward smaller, more efficient and specialized models. We believe the future lies in using LLM generated data to train smaller focused models that can be deployed efficiently at scale (or even locally).

In our next article, we'll explore how organizations are successfully replacing large, general-purpose models with smaller, specialized ones trained on synthetic data. We'll examine real-world cases where this approach has led to significant improvements in both performance and operational efficiency.

Have you experimented with synthetic data or small language models in your work? We'd love to hear about your experiences. Share your insights in the comments below or join our community discussion about the future of AI model development.

Appendix: Can I Use Commercial Models for Synthetic Data?

In short, if you’re using a commercial LLM like Claude, be aware that while you generally own the text it generates, the Terms of Service often prohibit using that output to build a competing language model. Please consult your company's legal team to avoid potential pitfalls. It’s also worth considering an open-source LLM with a more permissive license that explicitly allows you to reuse outputs for training. It’s always better to play by the rules than to find yourself on the wrong side of a TOS.

---

Further Reading

For a deeper dive into these topics, check out these excellent resources:

1. "Fine-tune classifier with ModernBERT in 2025" - Comprehensive guide on utilizing ModernBERT for classification tasks
2. "Finally, a replacement for BERT: Introducing ModernBERT" - Simon Willison's detailed exploration of ModernBERT
3. "Fine-tune ModernBERT for RAG with Synthetic Data" - HuggingFace guide on combining RAG and synthetic data
4. "Synthetic data & Smol models in 2024" - Comprehensive presentation on the evolution of synthetic data and small models
5. "Synthetic Data Generation with Instructor" - Tutorial on using Instructor library for structured synthetic data generation
6. "Distilabel: Advanced Data Generation Guide" - Detailed guide on scalable data generation with Distilabel