Personalized Benchmarking: Evaluating LLMs by Individual Preferences

Develop a framework that re-ranks standard LLM benchmarks based on user-provided preference weights. This allows researchers to understand model performance through the lens of specific user archetypes rather than flat averages.

Suggested repo: pref-eval

"Benchmarks that actually care about your specific preferences."

Estimated effort: 40h