Large language models (LLMs) like those used by Meta FAIR excel at rapid text generation but struggle with complex reasoning tasks that require deliberate thought and planning, akin to System 2 thinking in cognitive science.
System 2 techniques, designed to enhance LLMs’ reasoning abilities, involve prompting the model to generate intermediate steps toward problem-solving. While effective, these techniques come with a computational cost and latency, making them impractical for real-time applications.
In response to these challenges, Meta FAIR researchers have introduced “System 2 distillation,” a novel approach aimed at teaching LLMs complex tasks without the need for intermediate reasoning steps.
Drawing inspiration from human cognitive processes where repeated practice moves tasks from System 2 to System 1 (automatic, intuitive thinking), System 2 distillation seeks to embed System 2 reasoning directly into LLMs’ fast, System 1-like text generation capabilities.
Typically, distillation in machine learning involves training a smaller model (the “student”) using outputs from a larger model (the “teacher”). However, System 2 distillation innovatively utilizes the LLM’s own System 2 reasoning outputs as teaching examples for refining its System 1 response.
This process involves prompting the model with complex tasks using System 2 techniques, verifying responses for consistency, discarding intermediate steps, and fine-tuning the model based on correct answers.
The effectiveness of System 2 distillation was evaluated across various reasoning tasks using different System 2 prompting techniques such as Chain-of-Thought, System 2 Attention, Rephrase and Respond, and Branch-Solve-Merge.
The results demonstrated significant improvements in performance on these tasks compared to traditional System 1 generation methods. Moreover, distilled models showed enhanced speed and reduced computational overhead since they could skip the intermediate reasoning steps.
Despite its successes, System 2 distillation isn’t universally applicable. Tasks requiring highly complex reasoning, such as certain types of mathematical problem-solving, proved challenging to distill effectively into the LLM’s System 1-like framework.
This indicates that while System 2 distillation can optimize performance for many tasks, some tasks may inherently require the slower, more deliberate processing characteristic of System 2.
Looking ahead, further research is needed to explore how System 2 distillation performs on smaller LLM models and its broader impact beyond the specific tasks used in training. Additionally, considerations around potential biases and generalizability in LLM benchmarks highlight ongoing challenges in deploying distilled models effectively in real-world applications.
System 2 distillation represents a promising advancement in optimizing LLM capabilities for complex reasoning tasks, aligning with human cognitive processes to enhance efficiency and performance in AI applications.
