RouteJudge: Preference-Based Evaluation of LLM Routers under Pluralistic User Preferences

Abstract

RouteJudge proposes a preference-based evaluation framework for LLM routers that accounts for pluralistic user preferences. Unlike conventional evaluation methods that assume a single optimization target, our framework accommodates diverse and potentially conflicting user requirements in routing decisions.

Role: Second author — contributed to full-stack application development (RouteJudge) and evaluation pipeline.

Status: Accepted at ICML 2026 Workshop