PRX Part 4: Our Data Strategy
Photoroom's PRX data strategy reveals that in pretraining, broad and diverse 'dirty' data teaches the model far more about the visual world than a meticulously cleaned set; aesthetics is a fine-tuning concern.
- Pretraining data is about coverage and diversity; over-filtering for aesthetics hurts concept and composition learning
- The data pipeline mixes public and internal datasets, re-captioned with a VLM for accurate long descriptions
- Clear division: pretraining handles breadth, fine-tuning injects taste and polish
- Streamable corpus design enables efficient training without per-image perfection
In the AI world, we often hear a golden rule: 'garbage in, garbage out.' So many teams spend enormous effort cleaning, filtering, and selecting high-quality samples when building training data. But Photoroom's PRX team took a counterintuitive path: they deliberately kept images that weren't 'perfect,' arguing that premature aesthetic obsession can ruin a model.
Origins: A candid data pipeline retrospective
This article, 'PRX Part 4: Our Data Strategy' on the Hugging Face blog, is the fourth in Photoroom's series. After covering model architecture, training design, and a 24-hour speedrun, they finally pull back the curtain on the most unglamorous yet critical part: the data. They don't claim to have built the 'best dataset in the universe'; instead, they openly share guiding principles, pitfalls, and what they'd do differently. This hands-on experience is far more valuable to visual model builders than many academic papers.
Breakdown: Three core principles behind the pipeline
Diversity over aesthetics: Pretraining teaches the model 'what the world looks like.' The team makes it clear that coverage is the goal at this stage—exposing the model to enough visual concepts, objects, scenes, lighting, and compositions. Even if many images are ordinary snapshots or slightly compressed, together they form a rich distribution. Filtering for beauty too early narrows the model's worldview; missed concepts and compositions can never be recovered later. Their mantra: 'Pretraining is for breadth; fine-tuning is for taste.' This division liberates dataset construction from one-size-fits-all perfectionism.
Mixing data sources and standing on the shoulders of others. PRX's data comes from a blend of public and internal datasets. Instead of starting from scratch, they leveraged existing quality filters, deduplication, NSFW detection, etc. Sources arrived in various shapes—some with images, others with metadata and baseline captions—which they unified with a custom toolchain. This looks like a shortcut but is pragmatic: quickly assembling a usable pretraining corpus for a 7B model is far more efficient than spending months polishing a perfect dataset.
Re-captioning with a VLM to accurately describe everything. The team learned that long captions that accurately capture everything in the image are what matter most for pretraining. They used a vision-language model to regenerate captions, ensuring the visual-language alignment isn't skewed by sloppy alt-text. This hints at a trend: synthetic captions are becoming a standard way to boost data quality in the AI era.
Trend insight: The separation of pretraining and fine-tuning duties is becoming consensus
Photoroom's practice isn't isolated. From GPT-4 to Stable Diffusion iterations, we see more teams splitting training into two distinct phases: first, massive, diverse, even noisy data to build foundational capabilities; second, small, meticulously curated data to inject style, safety, and alignment. This separation makes data workflows more scalable and reduces early cleaning costs. More profoundly, it challenges our intuition that data must be clean—sometimes, keeping a bit of 'dirt' makes the model more robust.
Practical value: What to consider if you're training models
- Don't overshoot on data quality too early. In pretraining, ensure coverage first. Even at petabyte scale, use fast, automated tools to crudely filter harmful content rather than investing human effort in sample-level perfection.
- Invest in caption quality. A good image with a bad label hurts far more than a mediocre image with a precise, long description. Consider batch re-captioning with a VLM—it might be the highest-ROI data improvement.
- Build reproducible, streamable pipelines. PRX emphasizes a 'streamable corpus,' meaning the pipeline directly outputs a format consumed by the training framework without intermediate storage. This saves precious time and storage costs at scale.
Unexpected: Dirty data isn't always bad
The most striking insight: excessive aesthetic filtering damages model capability. The logic: the real world is noisy, messy, and full of average-looking images. If a model never sees these 'imperfections,' it can't understand the full distribution of the world. This doesn't mean no quality control at all, but rather deferring 'denoising' to the right stage. It may explain why some models trained on coarse data like CommonCrawl generalize better on downstream tasks—they've seen a more authentic world.
Analysis by BitByAI · Read original