Key takeaways

• Hermes 3 70B Instruct is a Transformer-based generalist language model, finetuned from Llama-3.1 70B, designed for enhanced agentic capabilities, roleplaying, reasoning, and conversation.
• The model achieved 100% accuracy on the 'AI2 Reasoning Challenge - Easy' dataset, demonstrating strong performance in elementary science reasoning and instruction following with no observed toxicity.
• Its significant limitations include struggles with multi-step reasoning, complex mathematical problems, and factual knowledge retrieval in niche domains, frequently leading to hallucination or inaccurate responses.
• The model scored 0% accuracy on the AIME 2025 Mathematical Competitive Programming benchmark, indicating significant deficiencies in advanced mathematical problem-solving, with common failures in algebraic errors, incorrect theorem application, and combinatorial miscounting.
• The model's lowest recorded performance was 17.1% accuracy on the 'SimpleQA' benchmark, revealing a considerable weakness in recalling specific or complex factual knowledge across various domains.

Technical specifications

Context length

131k tokens

Max output

—

Architecture

Transformer

Modalities

text->text

Parameters

~70B

Open weights

No

Licence

Mixed

Regions

USA

Vendor

Nous Research

Provider model key

nousresearch/hermes-3-llama-3.1-70b