How Big of a Problem Is Distillation?
Not everyone agrees that distillation is bad for a few different reasons.
One response has been to point out that distillation can help one AI model catch up to another, but it will never help that model surpass the model that it is distilling. From this perspective, it’s not a tactic that will actually lead to the innovation that any AI company needs to demonstrate in order to get ahead of the competition.
Another perspective argues that distillation is inherently good, since it encourages competition by functioning to prevent any single AI lab from getting too far ahead of the rest.
Nevertheless, this new accusation from Anthropic takes the ongoing conversation about AI into a new direction.
