Key takeaways

• Hermes 3 405B Instruct is a frontier-level, full-parameter finetune of Llama-3.1 405B, designed for advanced reasoning, roleplay, and function calling.
• It is a Transformer-based model with 405 billion parameters and a 12000-token context length, aiming to solve complex tasks requiring sophisticated language understanding and generation.
• The model demonstrates strong capability in recalling factual information and applying general knowledge across many academic domains, achieving 79.24% accuracy on MMLU benchmarks. It excels at generating detailed step-by-step mathematical solutions with low toxicity.
• Despite its strengths, the model struggles significantly with nuanced moral judgments and social understanding, making ethically questionable answers in 'moral_scenarios'. It also exhibits difficulty in precise factual understanding or complex causal reasoning in some subjects.
• The model shows a 0% accuracy rate on the AIME 2024 math benchmark, failing due to frequent calculation errors, logical inconsistencies, and misinterpretation of problem constraints, despite attempting detailed verbose solutions.

Technical specifications

Context length

12k tokens

Max output

—

Architecture

Transformer

Modalities

text->text

Parameters

~405B

Open weights

No

Licence

Mixed

Regions

USA

Vendor

Nous Research

Provider model key

nousresearch/hermes-3-llama-3.1-405b