🔍 Read the full analysis: CUGA Vs. Hermes Vs. Deep Agents: A Practical Guide To Choosing on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
IBM reports that CUGA completed 73 of 90 AppWorld tasks, compared with 42 for Hermes and 27 for Deep Agents, with all three using GLM-5.3-flash. The result shows a performance gap on this particular benchmark, but does not establish an overall ranking or show how the systems perform in production or enforce policies.
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks, ahead of Hermes with 42 and Deep Agents with 27, when all three systems used the same open-weight model, GLM-5.3-flash. The comparison suggests that the software coordinating an AI agent can affect results on a defined workload, but it does not establish which harness performs best overall.
According to IBM, the evaluation ran the three harnesses on the same 90 tasks across 30 scenarios, using the Harbor evaluation framework and one attempt per task. IBM reports task completion rates of 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. On a separate scenario measure, which counted a scenario only if all three of its variants passed, CUGA passed 20 of 30 scenarios, Hermes eight and Deep Agents two.
The systems did not use identical configurations. CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP and LangChain middleware, including tool selection, context summarization, a to-do list, retries and call limits. IBM says it also applied two fixes to Deep Agents so malformed model replies would not end a run. The experiment setup is publicly available in IBM’s Harbor fork, called harbor-eval, according to the source.
Why Harness Design Affects Scores
The results make the harness a factor worth testing separately from the underlying model. An agent harness shapes how a task is divided, which tools are available and whether actions or results are checked. In a workflow involving several applications, those choices can influence whether the model completes each step successfully.
For organizations comparing agents, the practical lesson is to test the model, harness and workload together, rather than treating a model benchmark as a complete forecast. The reported gap is substantial within this test, but it is not proof that CUGA will outperform Hermes or Deep Agents on personal assistance, research or other tasks. The benchmark also did not measure policy enforcement, an important requirement for systems handling business data or transactions.
Popular right now
Dyson V8 Cyclone Cordless Vacuum
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Dreame H12 Pro Wet and Dry Vacuum Mop
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Use Cases
The systems are designed with different applications in mind. The source describes CUGA, short for ConfigUrable Generalist Agent, as IBM’s open-source harness for enterprise work across systems such as customer relationship management and payments. It describes Hermes from Nous Research as intended for always-on personal assistants and Deep Agents from LangChain as suited to longer multi-step work, including research. The comparison is therefore better read as a test of these particular configurations than as a general contest among products.
AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM characterizes it as a proxy for some capabilities relevant to enterprise automation, such as reading API documentation, paging through results and changing state across apps. But the benchmark uses consumer apps; it does not directly show how the agents would perform on real enterprise systems. The source also cites a separate directory analysis in which GLM-5.2’s SWE-bench Pro result changed from 23% to 52% with a different harness. That is a separate comparison, not part of the AppWorld experiment.
“This is not a verdict on Hermes or Deep Agents.”
— IBM
What the AppWorld Test Cannot Show
The reported comparison covers one model, one 90-task sample and one benchmark, with a single attempt per task. IBM says the harness configurations differed, so the experiment does not isolate every implementation difference. The supplied material does not show how results would change with other models, repeated runs, different task sets or production workloads.
The test did not use policies and therefore provides no evidence about policy enforcement. IBM separately describes CUGA features for applying rules through the harness, but their effectiveness is not established by these scores. The source also gives no comparative figures for cost, latency, reliability at larger scale or deployment outcomes, and provides no timeline for independent replication or a broader comparison.
Evaluate Agents on Real Workflows
Teams considering these systems can start with the publicly available harbor-eval setup, then test agents against workflows representative of their own operations. A useful evaluation would match the models and tools planned for deployment, include repeated or varied tasks, and track both completion rates and consistency.
Organizations would also need separate checks for access controls, approval steps and data boundaries, alongside performance in their own environments. No broader study or independent replication timeline is specified in the supplied material, so the next evidence will depend on further testing by IBM or other evaluators.
Key Questions
Which harness completed the most AppWorld tasks?
IBM reports that CUGA completed 73 of 90 tasks. Hermes completed 42 and Deep Agents completed 27 in the reported test.
Did all three systems use the same model?
Yes. According to IBM, all three used GLM-5.3-flash and were evaluated on the same tasks. Their harness configurations differed.
Does this prove CUGA is the best agent?
No. The result covers one benchmark, one model and one set of 90 tasks. It does not establish an overall ranking or predict performance on other kinds of work.
Did the experiment test safety policies or enterprise deployment?
No. The supplied account says the runs did not use policies and that AppWorld uses consumer apps. The scores do not establish policy enforcement or performance on real enterprise systems.
Primary source: Hugging Face · via ThorstenMeyerAI.com
