Llama-3.3-Nemotron-Super-49B-v1.5 is a 49B-parameter, English-centric reasoning/chat model derived from Meta’s Llama-3.3-70B-Instruct with a 128K context. It’s post-trained for agentic workflows (RAG, tool calling) via SFT across math, code, science, and multi-turn chat, followed by multiple RL stages; Reward-aware Preference Optimization (RPO) for alignment, RL with Verifiable Rewards (RLVR) for step-wise reasoning, and iterative DPO to refine tool-use behavior. A distillation-driven Neural Architecture Search (“Puzzle”) replaces some attention blocks and varies FFN widths to shrink memory footprint and improve throughput, enabling single-GPU (H100/H200) deployment while preserving instruction following and CoT quality. reasoning/chat model derived from Meta’s Llama-3.3-70B-Instruct with a 128K context. It’s post-trained for agentic workflows (RAG, tool calling) via SFT across math, code, science, and multi-turn chat, followed by multiple RL stages; Reward-aware Preference Optimization (RPO) for alignment, RL with Verifiable Rewards (RLVR) for step-wise reasoning, and iterative DPO to refine tool-use behavior. A distillation-driven Neural Architecture Search (“Puzzle”) replaces some attention blocks and varies FFN widths to shrink memory footprint and improve throughput, enabling single-GPU (H100/H200) deployment while preserving instruction following and CoT quality
Evaluations
3
across 3 benchmarks
Latency
2m 17s
Context length
131k
tokens
Cost
$0.10 · $0.40
input · output per 1M tokens
Key takeaways
Llama 3.3 Nemotron Super 49B V1.5 is a 49B-parameter, English-centric reasoning/chat model derived from Meta’s Llama-3.3-70B-Instruct, designed for agentic workflows like RAG and tool calling with a 128K context window.
The model utilizes a distillation-driven Neural Architecture Search called 'Puzzle' to optimize memory and throughput, enabling single-GPU (H100/H200) deployment while preserving instruction following and CoT quality.
It excels in algebraic problem-solving, number theory, and advanced SQL features like CTEs, window functions, and JSONB operations. However, it struggles with complex combinatorics, precise string manipulation (especially regex), complex conditional logic in SQL, and multi-step intricate reasoning.
The model achieved 73.3% accuracy on the AIME 2024 benchmark for advanced mathematics but scored 0% on combinatorics questions. It achieved 25.33% accuracy on the BIRD-CRITIC benchmark for SQL generation.
Common failure modes include providing incorrect numerical answers despite partially correct reasoning in math, inability to derive or apply complex geometric relationships, and errors in complex combinatorial reasoning. For SQL, failures often involve incorrect string manipulation functions, misinterpretation of NULL values, and logical errors in complex conditional statements.
