Building a Better LiteParse Skill with Evals

Why giving an agent a tool is no longer enough Recently, the LlamaIndex team published a highly instructive engineering case study detailing how to optimize Claude Agent's LiteParse PDF-parsing skill through systematic evaluations. Many developers assume that simply hooking up an external parsing library to a large language model completes the integration. In reality, agents often behave like inexperienced interns. Even when equipped with powerful external tools, they repeatedly call the same file, blindly trigger OCR on natively digital reports, and dump high-resolution page screenshots directly into the context window. This behavior does not just cause latency spikes and token bill explosions; it frequently triggers context overflow and degrades downstream reasoning quality. This case is worth discussing now because it pinpoints a critical bottleneck in current agent deployment. Stacking toolchains is no longer the technical barrier. Teaching agents to use tools with discipline, cost-awareness, and state management is the true dividing line between a fragile prototype and a production-ready system.

Deconstructing anti-patterns and precision tuning Instead of blindly tweaking prompts or chasing higher benchmark scores, the team approached the problem forensically. By running standardized benchmarks and collecting structured JSONL interaction traces, they dissected the agent's actual runtime behavior. They identified several classic money-burning anti-patterns. For instance, the agent invoked the parse command up to nine times for a single document within one session. It defaulted to running OCR even for PDFs with native text layers, effectively doubling processing time. It also abused wildcard grep searches, injecting twenty to thirty thousand characters of raw text into the conversation at once. The solution was pragmatic: surgical refinement of the skill instructions based on trace analysis. They embedded mandatory pre-checks into the prompt, enforced a do not re-parse cached content rule, and strictly capped the character output per tool call. After just a few iteration cycles, parsing speed improved dramatically, while token consumption and hallucination rates dropped significantly. It is also worth noting that because document parsing is inherently I/O bound, wrapping the local CLI as a skill proved far more practical than forcing it into an MCP server architecture that lacks native file upload support.

The deeper trend: agent engineering is entering a trace-driven era This case reveals a fundamental shift in how we build AI agents. The development paradigm is moving away from intuition-based prompt engineering toward data-driven trace optimization. Previously, we relied on manual experience to craft system prompts. Today, success depends heavily on observability and automated evaluation loops. A skill is no longer a static markdown manual; it has become a strategic layer that requires continuous load testing, bad-pattern monitoring, and rapid iteration. Markdown is effectively becoming the agent's operational handbook, but what truly determines its reliability is the evaluation pipeline running behind it. The industry is learning that you cannot manage what you do not measure, and agent traces are rapidly becoming the new source of truth for system optimization.

Practical takeaways for developers If you are building agents that handle long documents or complex workflows, stop pouring all your effort into model fine-tuning or chasing the latest benchmark scores. First, ensure your agent's full interaction traces are structured and persistently stored. Second, run periodic scripts to scan for the most expensive calls, pinpointing redundant operations and context pollution. Third, hard-code strong constraints directly into your tool-calling logic. Implement state checks, enforce result truncation, and design graceful fallback strategies. This methodology will directly slash your cloud infrastructure bills while simultaneously boosting system stability. It turns agent optimization from a guessing game into a repeatable engineering process.

The counter-intuitive reality: the bottleneck is not intelligence, it is tool misuse We instinctively blame poor agent performance on the underlying model's lack of reasoning power. However, real-world debugging shows that a massive portion of performance degradation stems from agents lacking cost awareness and proper state management. Implementing evaluation guardrails and usage discipline often yields a significantly higher return on investment than blindly upgrading to the next flagship model. The future of agent development will not be won by raw computational intelligence alone. It will be dominated by teams that master system-level engineering discipline, precise cost control, and iterative trace analysis. Agents are maturing from clever conversational interfaces into structured software components, and their success will depend entirely on the rigor of the frameworks we build around them.