Skip to main content

Gemini 1.5 Pro (May '24)

GoogleGemini

Description

Gemini 1.5 Pro is a mid-size multimodal model optimized for a wide range of reasoning tasks. It can process large amounts of data at once, including 2 hours of video, 19 hours of audio, codebases with 60,000 lines of code, or 2,000 pages of text.

Release Date
2024-05-15
Parameters
Context Length
Modalities

Capability Radar

24
general
22
coding
32
reasoning
27
scienceest.
29
agents
80
multimodal

Science uses a reasoning proxy when dedicated science benchmarks are unavailable.

Rankings

Domain#RankScoreSource
Code Ranking322
25.0
AA
General Ranking369
30.0
AA
Math Reasoning238
37.0
AA
Multimodal Ranking37
79.0
LS
Reasoning4
93.0
LS
Science393
28.0
AA

Benchmark Scores (LLM Stats)

Biology

GPQA59.1%SR

Code

HumanEval84.1%SR

Finance

MMLU85.9%SR
MMLU-Pro75.8%SR

General

Natural2Code85.4%SR
MRCR82.6%SR
MMMU65.9%SR
Vibe-Eval53.9%SR

Healthcare

WMT2375.1%SR

Language

FLEURS93.3%SR
BIG-Bench Hard89.2%SR

Math

GSM8k90.8%SR
MGSM87.5%SR
MATH86.5%SR
DROP74.9%SR
MathVista68.1%SR
FunctionalMATH64.6%SR
PhysicsFinals63.9%SR
HiddenMath52.0%SR
AMC_2022_2346.4%SR

Multimodal

Video-MME78.6%SR

Reasoning

HellaSwag93.3%SR

Safety

XSTest98.8%SR

AA Evaluation Indices

Coding Index
19.8
Intelligence Index
6.3
Math 500
0.7
Mmlu Pro
0.7
Gpqa
0.4
Scicode
0.3
Livecodebench
0.2
Aime
0.1
Hle
0.0

LLM Stats Category Scores

Safety
100
Speech To Text
90
Language
80
Legal
80
Long Context
80
Math
80
Reasoning
80
Finance
80
Healthcare
80
Code
80
Multimodal
70
General
70
Vision
70
Physics
60
Biology
60
Chemistry
60

Pricing

Input PriceFree
Output PriceFree
Blended Price (3:1)Free

Speed

Tokens/sec0.0
Time to First Token0.00s
Time to Answer0.00s

Provider Price Ranking

No provider data available

External Sources