When a small open-weight model performs surprisingly well, one explanation tends to dominate the conversation: it must have distilled a larger U.S. frontier model.

Knowledge distillation—training a smaller “student” model to reproduce the behavior of a larger “teacher”—is real, useful and increasingly common. But it has also become a recurring accusation against open-weight models developed by Chinese labs. When one of these models approaches the performance of a proprietary U.S. system, critics are quick to assume that it copied, distilled or simply “stole” from a more capable competitor.

Some of those suspicions may be justified. Training-data provenance is often opaque, and model developers rarely disclose every source they use. But similarity is not proof of imitation, and imitation is not the only explanation for rapid progress. Treating every unexpected performance gain as evidence of distillation ignores the many other ways a model can become more capable.

Models can improve through better data, better tokenization, more efficient use of training compute, architectural changes, stronger post-training and increasingly sophisticated compression techniques. A smaller model may perform better not because it copied a larger one, but because it was trained more intelligently.

These distinctions will only become harder to trace as AI takes a larger role in generating data, evaluating answers and optimizing training pipelines. Depending on whom you ask, genuinely self-improving AI may be months away or still years in the future. Either way, the boundary between human-directed development, synthetic-data training, model-to-model learning and autonomous improvement is already beginning to blur.

Distillation can be effective, but it is far from the only option.

This article explores the other ways language models can improve in quality, efficiency and size—from data curation and compute allocation to architecture design, post-training and compression.

What does “optimizing an LLM” mean?

Before comparing techniques, it is helpful to define what is being optimized. “Better” can mean several different things:

• Higher factual accuracy
• Better reasoning or coding ability
• Better instruction following
• Lower hallucination rates
• Better performance in a specialized domain
• Lower latency
• Lower inference cost
• A smaller model checkpoint
• Lower memory consumption
• Better performance per parameter
• Better performance per unit of training compute

These goals are related, but they are not identical.

For example, quantization can make a model much smaller in memory without changing its parameter count. Retrieval-augmented generation can improve factual accuracy without changing the model weights. A Mixture-of-Experts architecture can reduce the computation used per token while increasing the model’s total number of stored parameters.

It is therefore useful to divide LLM optimization into six broad categories:

1. Data optimization
2. Pretraining and compute optimization
3. Architecture optimization
4. Post-training
5. Model compression
6. Inference-time and system-level improvements

1. Start with the data

One of the most important lessons from recent LLM research is that more data is not automatically better.

A trillion tokens of duplicated, low-quality or irrelevant text may be less useful than a smaller but carefully selected dataset. Research projects such as DataComp-LM and FineWeb have demonstrated that data filtering, deduplication and mixture design can substantially affect model performance.[1][2]

However, data optimization should not be understood as simply “deleting bad data.” The real objective is to increase the amount of useful learning signal available for every unit of training compute.

Clean low-quality and malformed content

Raw web data contains enormous amounts of material that may provide little training value, including:

• HTML boilerplate and navigation menus
• Spam and keyword-stuffed pages
• Repetitive templates
• Broken or incorrectly encoded text
• Scraped search results
• Machine-generated garbage
• Extremely short or incoherent documents
• Pages containing mostly advertisements
• Low-quality machine translations
• Documents written in the wrong language

These documents still consume tokens and computation during training. Filtering them allows the model to spend more of its training budget on useful language, knowledge and reasoning patterns.

Common filtering methods include:

• Handwritten heuristics
• Language-identification systems
• Perplexity-based scoring
• Spam and toxicity classifiers
• Document-quality classifiers
• Educational-value classifiers
• LLM-based quality scoring

DataComp-LM, for example, found model-based filtering to be an important part of assembling a high-quality training set.[1] FineWeb also documented how different filtering choices affected downstream model performance.[2]

Do not over-filter

Filtering creates its own risks. If the definition of “high quality” is too narrow, the dataset may lose:

• Minority languages and dialects
• Informal conversation
• Creative writing
• Rare technical knowledge
• Unusual formatting
• Source code
• Mathematical notation
• Long-tail cultural information

A good dataset should not contain only formal textbook prose. It should balance quality with diversity and coverage.

Remove duplicate content

Deduplication is another standard part of modern LLM data pipelines.

Duplicates can appear at multiple levels:

• Identical documents
• Slightly modified copies of documents
• Repeated paragraphs
• Quoted articles
• Website templates
• Mirrors of the same repository
• Reposted discussions
• Repeated code files

Deduplication offers several benefits:

• Less training compute is wasted
• The effective diversity of the dataset increases
• The risk of memorizing repeated passages decreases
• Extremely duplicated viewpoints have less influence
• Benchmark contamination becomes easier to control

Exact duplicates are relatively easy to identify. Near-duplicates are harder because two pages may contain the same content with different formatting, advertisements or small wording changes. Techniques such as MinHash, locality-sensitive hashing and embedding similarity are commonly used to detect them.

Decontaminate evaluation benchmarks

Evaluation contamination occurs when benchmark questions or answers appear in the training data.

A contaminated model may appear to reason correctly when it has actually memorized the test material. This can lead developers to overestimate the effect of a new architecture or training method.

Decontamination pipelines may search for:

• Exact benchmark questions
• Benchmark answers
• Paraphrased questions
• Explanations of benchmark solutions
• Online discussions containing benchmark content
• Translations or reformatted versions of test examples

Decontamination does not necessarily make the model more capable, but it makes measurements of capability more reliable.

Optimize the data mixture

A pretraining dataset is normally assembled from many sources, such as:

• General web pages
• Books
• News
• Encyclopedias
• Scientific literature
• Source code
• Mathematics
• Question-answer data
• Conversations
• Legal or financial documents
• Multilingual text

The proportion assigned to each source can have a major effect on the final model.

If code is underrepresented, coding performance may suffer. If technical documents dominate, the model may become less capable with everyday conversation. If one language is overrepresented, other languages may receive inefficient tokenization and weaker representation.

Data-mixture optimization can involve:

• Upsampling valuable but scarce sources
• Downsampling abundant low-quality sources
• Balancing languages and domains
• Monitoring per-source validation loss
• Adjusting the mixture during training
• Selecting different mixtures for different training stages

The key idea is that training data should be treated as a portfolio, not a single undifferentiated pile of text.

Preserve diversity and long-tail knowledge

Quality scores alone often favor documents that resemble the classifier’s preferred examples. This can create a large dataset containing many polished but conceptually similar documents.

Diversity-aware selection attempts to cover more of the available knowledge space with fewer tokens. Methods include:

• Semantic clustering
• Topic balancing
• Representative sampling
• Domain balancing
• Language balancing
• Diversity-aware deduplication
• Preserving rare document categories

A diverse dataset may contain a small amount of imperfect text because it introduces concepts, writing styles or domains not found elsewhere.

Use curriculum learning

Curriculum learning changes the order or composition of data during training.

Possible strategies include:

• Starting with clean general text
• Introducing technical material later
• Moving from simple to difficult examples
• Adding more mathematics or code in later stages
• Training on short sequences before long sequences
• Following general pretraining with domain-specific pretraining

Curriculum learning does not always guarantee improvements. Its effectiveness depends on the model, objective and target capabilities. Nevertheless, data ordering gives developers another way to extract more value from a fixed token budget.

Generate high-quality synthetic data

Synthetic data can be especially valuable when natural examples are scarce or when answers can be automatically verified.

Common applications include:

• Mathematics problems
• Programming exercises
• Logical reasoning
• Tool-use demonstrations
• Structured extraction
• Rare languages
• Instruction-following examples

The most useful synthetic data often has an external correctness check. For example:

• Code can be tested with unit tests
• Mathematical answers can be checked with a solver
• SQL queries can be executed against a database
• Structured output can be checked against a schema
• Formal proofs can be checked with a proof assistant
• Tool calls can be tested in a simulator

The phi-1 research showed how carefully curated “textbook-quality” and synthetic data could help a relatively small model achieve strong coding results.[3]

Synthetic data is not necessarily distillation. If examples are generated from rules, simulators, formal systems or the model’s own outputs and then independently verified, the process is better described as synthetic-data training.

The boundary becomes less clear when a small model is trained specifically to imitate answers generated by a stronger model. That process is often considered black-box distillation.

Synthetic data also needs quality control. Research on recursive training has shown that repeatedly training on uncurated model-generated data can distort the learned distribution and contribute to “model collapse.”[4]

Build a better tokenizer

The tokenizer determines how text is divided into the units processed by the model.

A poor tokenizer may split ordinary words into many fragments, especially in underrepresented languages. This increases:

• Sequence length
• Training cost
• Inference cost
• Context-window consumption
• Difficulty learning certain concepts

Tokenizer optimization can include:

• Training on the intended language mixture
• Allocating vocabulary more fairly across languages
• Preserving common code patterns
• Representing mathematical notation efficiently
• Supporting byte-level fallback
• Reducing unnecessary token fragmentation

For narrow-domain models, reducing the vocabulary can also shrink the embedding and output layers. However, aggressive vocabulary reduction may damage multilingual, code or general-purpose capabilities.

2. Allocate training compute more intelligently

Model quality depends not only on how many parameters the model has, but also on how many tokens it sees during training.

Earlier LLM development often focused on building the largest model possible. The Chinchilla scaling-law research demonstrated that many large models were undertrained: they had too many parameters relative to the number of training tokens.[5]

In that study, a 70-billion-parameter model trained on more data outperformed substantially larger models trained with a comparable compute budget.

This leads to a powerful non-distillation strategy:

Instead of building the largest model possible, build a smaller model and train it properly.

A smaller, well-trained model can provide:

• Better performance for its size
• Lower inference costs
• Lower memory requirements
• Easier fine-tuning
• Faster deployment
• Higher request throughput

The correct balance depends on:

• Parameter count
• Number of training tokens
• Data quality
• Training compute
• Expected inference volume

A production system with millions of future requests may deliberately spend more compute during pretraining to produce a smaller and cheaper inference model.

Improve the training objective

Most decoder-only LLMs use next-token prediction, but the training objective can be adapted to expected use cases.

Additional objectives may include:

• Fill-in-the-middle prediction for code
• Span corruption
• Prefix language modeling
• Retrieval-conditioned generation
• Tool-use prediction
• Long-context training
• Structured-output objectives
• Multi-task training
• Outcome-verified reasoning

For example, fill-in-the-middle training helps a code model complete missing sections inside existing files rather than only generating code from left to right.

The best training objective is not necessarily the most general objective. It is the one that produces the capabilities required at deployment.

Tune the training process

Many LLM improvements come from relatively ordinary engineering decisions:

• Learning-rate selection
• Warmup duration
• Learning-rate decay
• Batch size
• Gradient clipping
• Weight decay
• Optimizer configuration
• Initialization
• Sequence packing
• Mixed-precision stability
• Checkpoint selection
• Loss normalization

These choices can significantly affect convergence and final quality.

Training-efficiency technologies such as FlashAttention do not directly change the model’s intelligence, but they reduce the cost of exact attention.[6] The saved computation can be invested in more tokens, longer sequences or more experiments.

Continue pretraining on a target domain

An existing base model can be improved for a particular domain by continuing next-token training on specialized material.

Examples include:

• Legal documents
• Medical literature
• Financial reports
• Scientific papers
• A specific programming language
• Internal company terminology
• A new natural language

This is commonly called continued pretraining or domain-adaptive pretraining.

It can add specialized vocabulary and knowledge without distillation. However, training too heavily on a narrow domain can cause catastrophic forgetting or reduce general capabilities. A common mitigation is to mix some general-domain data into the continued-pretraining corpus.

3. Use a more efficient architecture

Architecture determines how effectively parameters and computation are converted into capability.

Optimize depth and width

Two models with the same number of parameters can perform differently depending on how those parameters are allocated across:

• Number of layers
• Hidden dimension
• Feed-forward dimension
• Number of attention heads
• Head dimension
• Vocabulary embeddings

Small-scale architecture experiments can identify better configurations before a full training run.

Use efficient Transformer components

Modern LLMs commonly incorporate components designed to improve quality, stability or efficiency, including:

• RMSNorm
• Gated feed-forward networks such as SwiGLU
• Rotary positional representations
• Grouped-query attention
• Multi-query attention
• Sliding-window attention
• Local and global attention combinations
• Improved residual scaling
• Long-context stabilization methods

Not all of these reduce checkpoint size. For example, grouped-query attention primarily reduces the memory required for the attention KV cache during inference.

This illustrates an important distinction: model-size optimization and runtime-memory optimization are different problems.

Use Mixture of Experts carefully

A Mixture-of-Experts, or MoE, architecture contains multiple expert networks but activates only a subset for each token.

This can provide more model capacity without using every parameter for every input. Switch Transformer demonstrated how sparse expert activation could increase capacity while controlling computation.[7]

MoE models can offer:

• Better quality per active computation
• Specialized experts
• Greater total capacity
• Efficient scaling

However, they also introduce:

• Routing complexity
• Load-balancing problems
• Communication overhead
• Training instability
• Large storage requirements

An MoE model may have relatively few active parameters per token while still having a very large checkpoint. MoE should therefore be described as compute-efficient or sparsely activated—not necessarily small.

Share or factorize parameters

Parameter count can be reduced by sharing weights between layers or factorizing large embedding matrices.

Possible techniques include:

• Cross-layer weight sharing
• Tying input and output embeddings
• Factorized vocabulary embeddings
• Reusing Transformer blocks
• Sharing attention projections

ALBERT is a well-known example of parameter reduction through factorized embeddings and cross-layer parameter sharing.[8]

These methods directly reduce unique parameter count, although excessive sharing can limit capacity.

4. Improve behavior through post-training

Pretraining teaches broad language patterns and knowledge. Post-training teaches the model how to behave as an assistant or solve particular classes of tasks.

Supervised fine-tuning

Supervised fine-tuning, or SFT, trains the model on curated prompt-response examples.

It can improve:

• Instruction following
• Conversation
• Reasoning presentation
• Structured output
• Tool use
• Safety behavior
• Domain-specific tasks

Dataset quality is especially important. A relatively small set of correct, diverse and carefully reviewed examples can be more useful than millions of inconsistent responses.

Useful SFT data should cover:

• Easy and difficult instructions
• Different writing styles
• Multi-step tasks
• Ambiguous requests
• Refusal scenarios
• Output-format constraints
• Long-context tasks
• Tool-use examples

Preference optimization

Supervised examples show the model what a good answer looks like. Preference data shows which of two answers is better.

Preference optimization can improve:

• Helpfulness
• Conciseness
• Tone
• Instruction adherence
• Safety
• User-perceived quality

Methods include:

• Reinforcement learning from human feedback
• Reinforcement learning from AI feedback
• Direct Preference Optimization
• Reward-model optimization
• Rejection sampling
• Rule-based or constitutional feedback

Direct Preference Optimization simplifies preference learning by avoiding the standard reinforcement-learning stage used in traditional RLHF pipelines.[9]

Preference optimization does not automatically improve factual knowledge. If annotators prefer confident, polished answers, the model may learn to sound better without becoming more correct. Preference labels should therefore include factuality and uncertainty when possible.

Use outcome and process supervision

For tasks with verifiable answers, the model can receive feedback based on actual outcomes.

Examples include:

• Whether code passes tests
• Whether a numerical answer is correct
• Whether a database query executes successfully
• Whether a tool call achieves the intended result
• Whether output follows a schema
• Whether a proof is formally valid

The model can be trained on final outcomes or on the quality of intermediate steps.

This is particularly valuable because correctness signals can be produced without using a stronger teacher model.

Merge specialized models

Multiple fine-tuned checkpoints can sometimes be combined using:

• Weight averaging
• Task-vector arithmetic
• Spherical interpolation
• Layer-wise merging
• Conflict-aware merging

Model merging may combine capabilities without training a new student model. However, results can be unpredictable when the source models have very different training histories or objectives.

5. Compress the model without distillation

When the objective is specifically to reduce model size, the main non-distillation options are quantization, pruning, low-rank factorization and parameter sharing.[10]

Quantization

Quantization stores model weights using fewer bits.

For example:

• FP32 to FP16 or BF16
• FP16 to INT8
• INT8 to INT4
• More aggressive 3-bit, 2-bit or binary representations

Quantization can be applied to:

• Weights
• Activations
• The attention KV cache
• Selected layers at different precisions

The main benefits are:

• Smaller checkpoints
• Lower memory usage
• Potentially faster inference
• Ability to run larger models on fewer devices

GPTQ demonstrated 3-bit and 4-bit post-training weight quantization with limited degradation in its evaluated settings.[11] AWQ uses activation information to identify and protect important weights during low-bit quantization.[12]

Quantization does not reduce the number of parameters. It reduces the number of bits used to store and process them. Actual speed improvements depend on whether the target hardware has efficient low-precision kernels.

For most deployment projects, quantization is the first compression technique worth testing.

Pruning

Pruning removes weights or model structures considered less important.

Unstructured pruning

Individual weights are set to zero.

This can achieve high sparsity while preserving quality, but it does not always produce real speed improvements. Ordinary GPUs are optimized for dense matrix multiplication, so specialized sparse formats and kernels may be required.

SparseGPT reported pruning large language models to at least 50% sparsity with limited accuracy loss in its tested settings.[13]

Structured pruning

Structured pruning removes complete components, such as:

• Attention heads
• Feed-forward channels
• Layers
• Experts
• Hidden dimensions

It is more likely to produce practical speed improvements on standard hardware, but it may cause more quality loss than unstructured pruning.

Semi-structured pruning

Semi-structured methods use hardware-friendly sparsity patterns, such as keeping two of every four weights. This provides a compromise between flexibility and deployability.

Pruning is often followed by recovery training. That is not distillation unless the recovery objective explicitly uses a teacher model’s outputs or representations.

Low-rank factorization

A large matrix can sometimes be approximated by two smaller matrices:

[ W \approx AB ]

If the intermediate rank is sufficiently small, this reduces both parameter count and computation.

Low-rank factorization works best when the original matrices contain redundant structure. Aggressive factorization can damage model quality, so recovery fine-tuning is often required.

This should not be confused with LoRA.

LoRA adds small trainable low-rank adapters while preserving the original model. It reduces the cost of fine-tuning, but it does not normally shrink the underlying base model.

Remove layers or dimensions

A model can also be compressed by removing:

• Entire Transformer layers
• Attention heads
• Feed-forward dimensions
• Vocabulary entries
• Experts
• Embedding dimensions

Removing full structures produces real dense-model reductions, but it is more destructive than simply quantizing weights. Recovery training is normally needed.

Dynamic techniques such as early exiting or layer skipping can reduce active computation without reducing the stored checkpoint.

Train a compact model from scratch

Instead of shrinking a large existing model, developers can design a smaller model from the beginning.

A strong compact-model recipe may include:

1. A carefully selected parameter count
2. A high-quality, deduplicated dataset
3. More training tokens per parameter
4. An efficient tokenizer
5. A well-tested architecture
6. High-quality supervised fine-tuning
7. Outcome or preference optimization
8. Quantization for deployment

This is arguably the cleanest non-distillation path to a small, capable model.

6. Improve the complete LLM system

Some of the most effective quality improvements do not require changing the model weights at all.

Retrieval-augmented generation

Retrieval-augmented generation, or RAG, searches an external knowledge source and provides relevant documents to the model.

RAG can improve:

• Factual accuracy
• Access to recent information
• Domain-specific knowledge
• Citation quality
• Knowledge provenance
• Ease of updating information

The original RAG research found that retrieval-augmented models produced more specific and factual language than a parametric-only baseline on knowledge-intensive tasks.[14]

RAG allows the model to rely less on memorizing every fact in its parameters. However, its quality depends on the entire retrieval pipeline:

• Document parsing
• Chunking
• Embeddings
• Search
• Reranking
• Context selection
• Citation verification

Poor retrieval can make a good model perform worse by supplying irrelevant or contradictory evidence.

Give the model tools

Models can be connected to:

• Search engines
• Calculators
• Code interpreters
• Databases
• Business APIs
• Symbolic solvers
• Compilers
• Formal verification systems

A smaller model with the right tools can outperform a much larger unsupported model on specific tasks. The model does not need to memorize a multiplication algorithm perfectly if it can call a calculator, or memorize changing inventory information if it can query a database.

Spend more compute at inference time

Inference-time scaling allows the system to spend additional computation on difficult questions.

Methods include:

• Sampling multiple answers
• Self-consistency voting
• Best-of-(N) selection
• Iterative critique and revision
• Search over reasoning paths
• Verifier-based ranking
• Adaptive compute allocation

Self-consistency has been shown to improve some reasoning tasks by sampling multiple reasoning paths and selecting the most consistent answer.[15]

Other research has found that well-allocated test-time computation can allow a smaller model to outperform a much larger model on suitable problems.[16]

The trade-off is higher latency and inference cost. Test-time scaling is therefore most useful when it can be applied selectively to difficult or high-value requests.

Which methods should be tried first?

For most teams, a practical non-distillation optimization sequence would be:

1. Define target capabilities and trustworthy evaluations.
2. Clean and deduplicate the training corpus.
3. Remove benchmark contamination.
4. Optimize the domain and language mixture.
5. Preserve diversity and long-tail knowledge.
6. Select an appropriate tokenizer.
7. Choose a compute-efficient model size and token budget.
8. Tune training hyperparameters on smaller experiments.
9. Apply high-quality supervised fine-tuning.
10. Use preference or outcome-based optimization.
11. Quantize the model for deployment.
12. Evaluate structured pruning or factorization if further compression is required.
13. Add retrieval, tools and test-time verification where appropriate.

The highest-return areas are usually:

• Better data
• Better data mixtures
• More appropriate training-token allocation
• High-quality post-training
• Quantization
• Retrieval and tool use

Conclusion

Knowledge distillation is only one way to produce an efficient language model.

A model can become smaller or more capable by:

• Removing duplicated and low-value data
• Selecting a better balance of training sources
• Preserving diversity
• Using verified synthetic data
• Improving tokenization
• Training a smaller model for more tokens
• Choosing a more efficient architecture
• Applying supervised and preference-based post-training
• Quantizing or pruning the weights
• Using retrieval, tools and inference-time search

The central lesson is that LLM capability does not come from parameter count alone.

A smaller model can compete with a larger model when it receives better data, a more appropriate training budget, a better architecture and stronger post-training. At the system level, retrieval and tools can further reduce the amount of knowledge and computation that must be stored inside the model itself.

The goal should not simply be to make an LLM smaller. It should be to maximize useful capability per parameter, per token, per byte and per unit of compute.

References and further reading

1. Li et al. — “DataComp-LM: In Search of the Next Generation of Training Sets for Language Models”
   https://arxiv.org/abs/2406.11794
2. Penedo et al. — “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”
   https://arxiv.org/abs/2406.17557
3. Gunasekar et al. — “Textbooks Are All You Need”
   https://arxiv.org/abs/2306.11644
4. Shumailov et al. — “AI Models Collapse When Trained on Recursively Generated Data”
   https://www.nature.com/articles/s41586-024-07566-y
5. Hoffmann et al. — “Training Compute-Optimal Large Language Models”
   https://arxiv.org/abs/2203.15556
6. Dao et al. — “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”
   https://arxiv.org/abs/2205.14135
7. Fedus, Zoph and Shazeer — “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”
   https://arxiv.org/abs/2101.03961
8. Lan et al. — “ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations”
   https://openreview.net/forum?id=H1eA7AEtvS
9. Rafailov et al. — “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model”
   https://arxiv.org/abs/2305.18290
10. Zhu et al. — “A Survey on Model Compression for Large Language Models”
    https://arxiv.org/abs/2308.07633
11. Frantar et al. — “GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers”
    https://arxiv.org/abs/2210.17323
12. Lin et al. — “AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration”
    https://arxiv.org/abs/2306.00978
13. Frantar and Alistarh — “SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot”
    https://arxiv.org/abs/2301.00774
14. Lewis et al. — “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”
    https://arxiv.org/abs/2005.11401
15. Wang et al. — “Self-Consistency Improves Chain of Thought Reasoning in Language Models”
    https://arxiv.org/abs/2203.11171
16. Snell et al. — “Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters”
    https://arxiv.org/abs/2408.03314
17. Wang et al. — “Self-Instruct: Aligning Language Models with Self-Generated Instructions”
    https://arxiv.org/abs/2212.10560
18. Penedo et al. — “The RefinedWeb Dataset for Falcon LLM”
    https://arxiv.org/abs/2306.01116