Key takeaways

• OpenAI o3-mini is a cost-efficient, Transformer-based reasoning model optimized for STEM domains, supporting function calling, structured outputs, and flexible 'reasoning effort' modes for balancing speed and accuracy.
• The model exhibits strong logical reasoning abilities, achieving 99.8% accuracy on the 'Formal Logic Extended' dataset, particularly excelling in evaluating complex nested logical expressions with consistent true/false outputs.
• A major limitation is the model's poor performance (7.78% accuracy) on general scientific knowledge tests like 'Humanity's Last Exam', where it frequently confabulates, gives inconsistent responses, and struggles with specific output formats and complex calculations.
• The model struggles with logical and inferential reasoning, probabilistic thinking, chaining sequential operations, and adhering to output formats, often leading to confabulation and incorrect answers that violate scientific facts.
• Performance analysis shows significant instability across prompts with minimal differences and inconsistency for identical questions, suggesting that additional training under current conditions is unlikely to lead to improvement.

Technical specifications

Context length

32k tokens

Max output

—

Architecture

Transformer

Modalities

text->text

Parameters

~12B

Open weights

No

Licence

Proprietary

Regions

USA

Vendor

OpenAI

Provider model key

openai/o3-mini