o3 AI Evaluation | LayerLens - Benchmark Results & Performance

Model

o3 by OpenAI

text+image->text

Free credits included. No credit card required.

o3 is a well-rounded and powerful model across domains. It sets a new standard for math, science, coding, and visual reasoning tasks. It also excels at technical writing and instruction-following. Use it to think through multi-step problems that involve analysis across text, code, and images.

Evaluations

across 14 benchmarks

Latency

10.8s

Context length

200k

tokens

Cost

$2 · $8

input · output per 1M tokens

Key takeaways

o3 by OpenAI is proposed as a powerful, Transformer-based model for multi-step problem-solving across text, code, and images, excelling in technical writing and instruction-following for complex analytical tasks.
The model demonstrates strong performance in math and science, achieving 96% accuracy on the MATH-500 dataset with high readability (74.37) and low toxicity (0.0049) in general responses.
A significant limitation of o3 is its struggle with complex calculations and nuanced scientific reasoning in biology and medicine, evidenced by only 16.7% accuracy on 'Humanity's Last Exam' dataset.
Key failure modes include incorrect answer selection despite well-reasoned explanations, inability to perform necessary calculations, and miscalibration between confidence and correctness, particularly in quantitative biological and medical problems.
The model is proprietary, with a context length of 200,000 tokens, and its internal training methodologies beyond its Transformer architecture are not detailed.

Technical specifications

Context length: 200k tokens
Max output: —
Architecture: Transformer
Modalities: text+image->text
Parameters: ~175B
Open weights: No
Licence: Proprietary
Regions: USA
Vendor: OpenAI
Provider model key: openai/o3