MIMIC Clinical Decision Making

This leaderboard shows current scores of models on the MIMIC Clinical Decision Making (MIMIC-CDM) and MIMIC Clinical Decision Making Full Information (MIMIC-CDM-FI) datasets. The dataset can be found here. The code used to run the models can be found here.

Model	Average ⬆️	Appendicits	Cholecystitis	Diverticulitis	Pancreatitis	#Params (B)	Max Sequence Length	Bits
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	70.48	89.9	65.7	66.1	60.2	70	8192	16

Model	Average ⬆️	Appendicits	Cholecystitis	Diverticulitis	Pancreatitis	#Params (B)	Max Sequence Length	Bits
google/gemma-3-27b-it	70.48	89.9	65.7	66.1	60.2	70	8192	16
google/medgemma-27b-text-it	69.1	89.1	58	61.1	68.2	70	8192	16
google/gemma-2-27b	68.65	85.9	70.2	58.8	59.7	70	8192	8
google/gemma-3-12b-it	66.2	86.1	59.4	61.1	58.2	70	8192	16
meta-llama/Meta-Llama-3-70B-Instruct	63.32	84.6	59	54.5	55.2	70	8192	4
meta-llama/Llama-3.3-70B-Instruct	62.42	84.2	56.8	58	50.7	70	8192	4
meta-llama/Meta-Llama-3.1-70B-Instruct	60.45	84.7	51.7	55.6	49.8	70	8192	4
TheBloke/Llama2-70B-OASST-SFT-v10-GPTQ	54.9	82	48	45.5	44.1	70	4096	4
TheBloke/WizardLM-70B-V1.0-GPTQ	53.92	78.4	47.4	44.4	45.5	70	4096	4
google/medgemma-4b-it	53.05	67.7	61.6	37	45.9	70	8192	16
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	51.6	78.8	41.2	46.3	40.1	70	8192	4
google/gemma-3-4b-it	50.8	64.3	55.7	36.2	47	70	8192	16
TheBloke/Llama-2-70B-Chat-GPTQ	45.45	74.6	33.8	34.2	39.2	70	4096	4
aaditya/OpenBioLLM-Llama3-70B	35.52	60.7	29.5	17.1	34.8	70	8192	4
google/gemma-3-1b-it	21.45	45.2	15.9	12.8	11.9	70	8192	16

Model	Average ⬆️	Appendicits	Cholecystitis	Diverticulitis	Pancreatitis	#Params (B)	Max Sequence Length	Bits
meta-llama/Llama-3.3-70B-Instruct 16bit	83.85	97.9	82.6	72	82.9	70	8192	16
google/gemma-2-27b 4bit	81.32	98.1	86.1	71.2	69.9	70	8192	4
google/gemma-2-27b 16bit	80.97	98.1	86.7	69.6	69.5	70	8192	16
google/gemma-2-27b 8bit	80.75	98	85.3	70	69.7	70	8192	8
meta-llama/Meta-Llama-3.1-70B-Instruct	80.38	98.1	76.4	68.9	78.1	70	8192	4
google/medgemma-27b-text-it	78.6	98.1	71.3	69.3	75.7	70	8192	16
meta-llama/Llama-3.3-70B-Instruct 4bit	77.7	97.5	73	69.3	71	70	8192	4
google/gemma-3-27b-it	75.55	97.9	73.5	72.4	58.4	70	8192	16
google/gemma-3-12b-it	74.38	96.5	70.7	59.1	71.2	70	8192	16
meta-llama/Llama-3.3-70B-Instruct 8bit	73.85	97.5	70.7	67.3	59.9	70	8192	8
wanglab/ClinicalCamel-70B	73.12	96.6	72.4	55.3	68.2	70	4096	4
meta-llama/Meta-Llama-3-70B-Instruct	72.47	97.8	48.3	69.3	74.5	70	8192	4
google/medgemma-4b-it	69.82	93.1	88.1	49.8	48.3	70	8192	16
google/gemma-3-4b-it	69.4	92.7	69	53.3	62.6	70	8192	16
TheBloke/Llama2-70B-OASST-SFT-v10-GPTQ	67.75	96.9	61.4	48.2	64.5	70	4096	4
aaditya/Llama3-OpenBioLLM-70B	66.93	96.3	52.5	47.9	71	70	8192	4
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	66.1	96.2	50.2	49.8	68.2	70	8192	4
TheBloke/WizardLM-70B-V1.0-GPTQ	65.1	96.4	37.3	52.9	73.8	70	4096	4
TheBloke/meditron-70B-GPTQ	58.92	96.7	19.4	44.7	74.9	70	4096	4
TheBloke/Llama-2-70B-Chat-GPTQ	58.8	95.1	48.1	44	48	70	4096	4
google/gemma-3-1b-it	21.88	37.8	14.7	9.7	25.3	70	8192	16

MIMIC Clinical Decision Making

How it works

Reproducibility