What AI says about ToolBeHonest

ToolBeHonest is a benchmark and dataset for diagnosing hallucinations in large language models that are augmented with external tools for real-world tasks. It employs a multi-level diagnostic framework with tasks like solvability detection, solution planning, and missing-tool analysis, using about 700 manually annotated samples across seven scenarios to test tool-enabled LLMs under varying tool availability. The project provides a leaderboard, a paper, and data resources to evaluate and benchmark the reliability of tool use in LLMs.