Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Nicholas Roberts, Ahmed Ahmed, Suhas Kotha, Percy Liang, Yu Wang +2

Synthetic rewrites, where a language model rewrites a natural document e.g. into a target style, are now a standard ingredient of pretraining corpora. However, critical aspects of their use remain poorly understood, such as how much compute should be spent rewriting training data and whether a model trained on rewrites can outperform the rewriter. We study these two questions by proposing a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites. We fit this scaling law in two settings. In the compute-limited setting, natural data is plentiful and rewrites come from a third-party rewriter trained with more compute than our largest run. We find that the estimated optimal rewrite fraction increases with compute, from under $10\%$ at $3\times10^{18}$ FLOPs to $32\%$ at $10^{22}$ FLOPs. Our scaling forecast extrapolates $33\times$ in FLOPs to train a 22B-parameter model, whose final loss is within $1.1\%$ of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the…