Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
What it is
This paper presents Foyer, a self-hosted personal inbox agent, and a rigorous measurement protocol for evaluating email routing systems. It compares nine systems, including classical baselines and generative LLMs, on 739 real emails to determine the most effective approach for classifying message type, urgency, and actionability.
The problem it solves
The lack of reliable ground truth data for evaluating personal email agents, as stored labels are often generated by the system itself, leading to unmeasured error rates.
What you could use it for
- Evaluating the performance of personal email triage agents
- Benchmarking classical machine learning models against LLMs for text classification
- Designing robust evaluation protocols for AI agents with self-generated labels
Why it matters
Provides empirical evidence that simple classical models can outperform or match LLMs for specific email routing tasks, potentially reducing computational costs and improving reliability in personal AI assistants.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.