LLM Router for Solving Math Problems

Main Goal

Develop an intelligent router that analyzes the difficulty and type of a query to dynamically allocate the optimal AI model.

• To objectively measure accuracy, it is limited to verifiable domains such as math and coding.
• Thinking (reasoning) vs. Non-Thinking (non-reasoning)

A naive approach would be to call an API (either "thinking" or "fast") and if the API returns an incorrect answer, label it and then use supervised learning to refine the router.

Open to other approaches, following the prior literature on LLM Routers.

Router Literature

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (2023-5)

https://arxiv.org/pdf/2305.05176

Core Idea

• 싸고 빠른 모델부터 순차 실행 → 각 단계에서 심판(judge)이 품질 임계점($\theta$) 통과 여부 판정 → 통과 시 정지

Method