Ai! Jack!v2026.04.01-f12
← 홈HF Daily Papers

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

원문 보기

AI 분석

•기여: 수학 추론의 On-Policy Self-Distillation에서 발생하는 추론 경로 축소를 세 가지 조절 축으로 통합해 설명하는 비판적 문헌 검토 제시
•방법: 토큰별 가중치로 신호를 어디에 적용하는지, 교사에게 어떤 특권 정보를 제공하는지, 교사와 지도 강도가 언제 변하는지에 따라 기존 연구를 구조화
•시사점: 더 큰 교사 없이도 추가 정보를 받은 동일 모델로 조밀한 학습 신호를 만들 수 있지만, 정보 비대칭이 편향과 추론 붕괴를 악화할 수 있음을 정리. 논문마다 다른 용어를 연결하고 합의된 사실과 논쟁 중인 주제를 구분
•한계: 새로운 알고리즘이나 실험 결과를 제시하지 않는 리뷰이며, 분석 범위가 수학 추론으로 제한됨

검증 정보

출처
huggingface.co
피드
HF Daily Papers
수집/분석 기준
2026. 9. 8. 오전 9:00
원문 확인
원문 열기

본문

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

하나의 증상, 세 가지 조절 축: On-Policy Self-Distillation 비판적 검토 — Ai! Jack! — Ai! Jack!