Key takeaways

• Llama 3.1 Nemotron Ultra 253B v1 is a Transformer-based LLM optimized for advanced reasoning, human-interactive chat, RAG, and tool-calling tasks, derived from Meta’s Llama-3.1-405B-Instruct.
• The model was customized using Neural Architecture Search (NAS) to enhance efficiency, reduce memory usage, and improve inference latency, supporting a context length of up to 128K tokens.
• The model demonstrates strong performance in fundamental scientific reasoning, achieving 100% accuracy on the AI2 Reasoning Challenge - Easy dataset, but can be verbose in its responses.
• A significant limitation was observed in the Accounting Audit dataset, where it achieved only 13.33% accuracy, struggling with statistical sampling, auditing standards, and misinterpreting question wording.
• The model exhibits inconsistent performance, excelling in easy science questions but struggling with nuanced concepts in auditing, often providing verbose 'think' sections that did not lead to correct answers in complex scenarios.

Technical specifications

Context length

131k tokens

Max output

—

Architecture

Transformer

Modalities

text->text

Parameters

~175B

Open weights

Yes

Licence

NVIDIA Open Model

Regions

USA

Vendor

Nvidia

Provider model key

nvidia/llama-3.1-nemotron-ultra-253b-v1