AnythingLLM tests MCP tool reranking

The idea is solid, but it is less of a novel trick than a direction the MCP ecosystem is already moving toward. The real tradeoff is not just reranker latency; it is adding another decision layer that can fail silently, be hard to debug, and become its own maintenance burden.

• –Tool catalogs get expensive fast, so retrieval or reranking makes sense once you have lots of overlapping tools, but small tool sets probably do not justify the extra orchestration.
• –Accuracy depends heavily on tool metadata quality, schema freshness, and how well the reranker understands the task, not just on the reranker model itself.
• –The hidden cost is observability: when the wrong tool is selected, it is harder to tell whether the problem was intent detection, retrieval, ranking, or the tool description.
• –Latency can be acceptable, but end-to-end task success matters more than top-k retrieval metrics; a fast wrong tool is still a failed tool call.
• –A hybrid setup is usually safer: coarse filtering, reranking, a small fallback set, and strong evals on real tool-use tasks.