[Lecture 11, slide 11] Correct the interpretation of the red optimal-batch curve

[LOGO127]

Hi, and thank you for making the lecture materials public!

Location

lecture_11.pdf, slide 11, “Optimal batch”

The slide currently says:

Red line attempts to identify minimum loss points for each y-value — this is the “optimal batch size” for a model size / dataset size combination.

I believe this reverses what is being held fixed and what is being minimized.

Interpretation in the MiniCPM paper

In MiniCPM, Section 3.2, the vertical columns are fixed-batch training trajectories. For each target loss, the authors collect equal-loss points across batch sizes, fit a parabola, and connect the minima of those parabolas with the red line.

Because the y-axis is token consumption, each red point is the batch size that reaches a fixed target loss using the fewest training tokens. In notation, the construction is

$$ B_{\mathrm{opt}}(L)=\arg\min_B D(B;L), $$

where $D(B;L)$ is the number of tokens required for batch size $B$ to reach loss $L$. It is not

$$ B_{\mathrm{opt}}(D)=\arg\min_B L(B,D), $$

i.e. selecting the lowest loss at each fixed y-value/token budget.

This also matches the paper's next step: plotting the selected optimal batch sizes against their corresponding target losses. The paper's Appendix A.2 further explains that, under its fixed-GPU setting, the objective is to minimize token consumption rather than jointly minimize token consumption and optimization steps.

Suggested wording

Red line connects the minima of parabolic fits to equal-loss points. For each target loss, it identifies the batch size that reaches that loss using the fewest training tokens. This is the token-efficient optimal batch size for a fixed model size and target loss.

It may also be clearer to replace “dataset size” here with “target loss” or “training stage,” since the y-axis is cumulative tokens processed rather than necessarily the size of a unique dataset.

Thanks for considering this clarification!