Llama-3.3-Nemotron-Super-49B-v1.5 is a 49B-parameter, English-centric reasoning/chat model derived from Meta’s Llama-3.3-70B-Instruct with a 128K context. It’s post-trained for agentic workflows (RAG, tool calling) via SFT across math, code, science, and multi-turn chat, followed by multiple RL stages; Reward-aware Preference Optimization (RPO) for alignment, RL with Verifiable Rewards (RLVR) for step-wise reasoning, and iterative DPO to refine tool-use behavior. A distillation-driven Neural Architecture Search (“Puzzle”) replaces some attention blocks and varies FFN widths to shrink memory footprint and improve throughput, enabling single-GPU (H100/H200) deployment while preserving instruction following and CoT quality. reasoning/chat model derived from Meta’s Llama-3.3-70B-Instruct with a 128K context. It’s post-trained for agentic workflows (RAG, tool calling) via SFT across math, code, science, and multi-turn chat, followed by multiple RL stages; Reward-aware Preference Optimization (RPO) for alignment, RL with Verifiable Rewards (RLVR) for step-wise reasoning, and iterative DPO to refine tool-use behavior. A distillation-driven Neural Architecture Search (“Puzzle”) replaces some attention blocks and varies FFN widths to shrink memory footprint and improve throughput, enabling single-GPU (H100/H200) deployment while preserving instruction following and CoT quality
Evaluations
3
across 3 benchmarks
Latency
2m 17s
Context length
131k
tokens
Cost
$0.10 · $0.40
input · output per 1M tokens
Key takeaways
Llama 3.3 Nemotron Super 49B V1.5 is an English-centric reasoning/chat model with 49 billion parameters, designed for agentic workflows like RAG and tool calling, with a large 128K context window.
The model is derived from Meta's Llama-3.3-70B-Instruct and underwent extensive post-training, including Reward-aware Preference Optimization (RPO) for alignment, RL with Verifiable Rewards (RLVR) for reasoning, and iterative DPO for refining tool-use behavior. A Neural Architecture Search named 'Puzzle' was used to optimize memory and throughput for single-GPU deployment.
It shows strong foundational SQL knowledge, including advanced features like CTEs, window functions, and JSONB operations. However, it struggles with precise string manipulation (especially regex), complex conditional logic, and optimizing performance-critical SQL scenarios. In mathematical problem-solving, it excels in algebra, geometry, and number theory, but fails significantly in combinatorics and highly complex geometry problems.
The model achieved 73.3% accuracy on the AIME 2024 benchmark for advanced mathematics and 25.33% accuracy on the BIRD-CRITIC benchmark for SQL generation. While it demonstrates strong factual recall in auditing and consistent instruction following, it struggles with nuanced interpretations and multi-step logical deductions in both SQL and math.
Common failure modes include providing incorrect numerical answers despite partially correct reasoning in math, inability to derive or apply complex geometric relationships, and errors in complex combinatorial reasoning. For SQL, failures often involve incorrect string manipulation functions, misinterpretation of NULL values in aggregations, improper application of window functions, and logical errors in complex conditional statements.
