Xiaohongshu Recommender System Evaluation Agent: A Context-Driven LLM-as-Judge Practice

Xiaohongshu has built a context-driven recommendation evaluation agent. It treats each recommendation request as the minimum unit of evaluation, dividing tasks between deterministic rules and LLM-as-Judge to perform multi-dimensional semantic judgments on the content users saw at that moment. The system uses a three-tier scoring scale (0/1/2) and fixes input snapshots and evaluation versions to ensure results are comparable, replayable, and drill-down capable for specific requests or individual items. The team is also exploring LLM-as-User continuous user simulation, currently positioned as an offline pre-screening tool before launch, not replacing online A/B testing.