News
The AI Seminar is a weekly meeting at the University of Alberta where researchers interested in artificial intelligence (AI) can share their research. Presenters include both local speakers from the University of Alberta and visitors from other institutions. Topics can be related in any way to artificial intelligence, from foundational theoretical work to innovative applications of AI techniques to new fields and problems.
On March 17, Yuqiao Wen — PhD student at the University of Alberta — presented “f-Divergence Minimization for Sequential Knowledge Distillation" at the AI Seminar.
Knowledge distillation (KD) is the process of transferring knowledge from a large model to a small one. It has gained increasing attention in the natural language processing community, driven by the demands of compressing ever-growing language models.
In this work, Wen proposes an f-distill framework, which formulates sequential knowledge distillation as minimizing a generalized f-divergence function. He proposes four distilling variants under his team's framework and shows that existing SeqKD and ENGINE approaches are approximations of their f-distill methods. He further derives step-wise decomposition for their f-distill, reducing intractable sequence-level divergence to word-level losses that can be computed in a tractable manner.
Experiments across four datasets show that the methods outperform existing KD approaches, and that symmetric distilling losses can better force the student to learn from the teacher distribution.
Watch the full presentation below:
Want to learn how you can kick-start your AI career? Find out more about Amii's Career Accelerator to find out more.
Mar 26th 2024
News
In this month's episode, Alona talks about how ChatGPT changed the public’s perception of what AI language models can do, instantly making most previous benchmarks seem out of date, and the excitement and intensity of working in a fast-moving field like AI.
Mar 18th 2024
News
Google.org announces new research grants to support critical AI research in Canada focused on areas such as sustainability and the responsible development of AI. The grant will provide a total of $2.7 million in grant funding to Amii, the Canadian Institute for Advanced Research (CIFAR) and the International Center of Expertise of Montreal on AI (CEIMIA).
Mar 11th 2024
News
Amii CEO, Cam Linke, shares insight and advice for executives in a rapidly changing AI landscape.
Looking to build AI capacity? Need a speaker at your event?