https://samuelepunzo.github.io/authors/z.-prins/Z. PrinsHugoBlox Kit (https://hugoblox.com)en-usTue, 31 Mar 2026 00:00:00 +0000https://samuelepunzo.github.io/publications/ppl_metric/Tue, 31 Mar 2026 00:00:00 +0000https://samuelepunzo.github.io/publications/ppl_metric/<p>This paper introduces a simple, principled interpretability framework based on <strong>token-level perplexity</strong> to test whether LLMs rely on linguistically relevant cues when solving benchmarks.</p> <p>Key contributions:</p> <ul> <li>A hypothesis-driven analysis method using perplexity distributions over minimal sentence pairs</li> <li>No reliance on unstable feature-attribution techniques</li> <li>Experiments on controlled linguistic benchmarks with several open-weight LLMs</li> <li>Evidence that linguistically important tokens influence but never fully explain model behavior, revealing reliance on non-linguistic heuristics</li> </ul>