Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models

Build an open-source evaluation framework that allows developers to run the SemanticQA benchmark against any local LLM via an API. This helps developers quantify how well their fine-tuned models handle idiomatic expressions and complex noun compounds.

Suggested repo: semantic-eval

"Stop guessing if your model understands idioms—test it with SemanticQA."

Estimated effort: 20h