Tag: 知识蒸馏 (2 articles)

Multiverse Computing introduces a memory-efficient knowledge distillation technique that uses cached top-K logits and a fused chunked KL loss to train on long contexts with just a single GPU, drastically reducing costs.

Ben Thompson proposes US legislation to clarify that training data collection is fair use and to ban terms that forbid distillation, countering Chinese open-source model competition and revealing the hypocrisy in AI policies.