Gradio

Model	Average ⬆️	Appendicits	Cholecystitis	Diverticulitis	Pancreatitis	#Params (B)	Max Sequence Length	Bits
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	68.65	85.9	70.2	58.8	59.7	70	8192	8

Model	Average ⬆️	Appendicits	Cholecystitis	Diverticulitis	Pancreatitis	#Params (B)	Max Sequence Length	Bits
google/gemma-2-27b	68.65	85.9	70.2	58.8	59.7	70	8192	8
meta-llama/Meta-Llama-3-70B-Instruct	63.32	84.6	59	54.5	55.2	70	8192	4
meta-llama/Llama-3.3-70B-Instruct	62.42	84.2	56.8	58	50.7	70	8192	4
meta-llama/Meta-Llama-3.1-70B-Instruct	60.45	84.7	51.7	55.6	49.8	70	8192	4
TheBloke/Llama2-70B-OASST-SFT-v10-GPTQ	54.9	82	48	45.5	44.1	70	4096	4
TheBloke/WizardLM-70B-V1.0-GPTQ	53.92	78.4	47.4	44.4	45.5	70	4096	4
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	51.6	78.8	41.2	46.3	40.1	70	8192	4
TheBloke/Llama-2-70B-Chat-GPTQ	45.45	74.6	33.8	34.2	39.2	70	4096	4
aaditya/OpenBioLLM-Llama3-70B	35.52	60.7	29.5	17.1	34.8	70	8192	4

Model	Average ⬆️	Appendicits	Cholecystitis	Diverticulitis	Pancreatitis	#Params (B)	Max Sequence Length	Bits
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	83.85	97.9	82.6	71.2	82.9	70	8192	16

Model	Average ⬆️	Appendicits	Cholecystitis	Diverticulitis	Pancreatitis	#Params (B)	Max Sequence Length	Bits
meta-llama/Llama-3.3-70B-Instruct 16bit	83.85	97.9	82.6	72	82.9	70	8192	16
google/gemma-2-27b 4bit	81.32	98.1	86.1	71.2	69.9	70	8192	4
google/gemma-2-27b 16bit	80.97	98.1	86.7	69.6	69.5	70	8192	16
google/gemma-2-27b 8bit	80.75	98	85.3	70	69.7	70	8192	8
meta-llama/Meta-Llama-3.1-70B-Instruct	80.38	98.1	76.4	68.9	78.1	70	8192	4
meta-llama/Llama-3.3-70B-Instruct 4bit	77.7	97.5	73	69.3	71	70	8192	4
meta-llama/Llama-3.3-70B-Instruct 8bit	73.85	97.5	70.7	67.3	59.9	70	8192	8
wanglab/ClinicalCamel-70B	73.12	96.6	72.4	55.3	68.2	70	4096	4
meta-llama/Meta-Llama-3-70B-Instruct	72.47	97.8	48.3	69.3	74.5	70	8192	4
TheBloke/Llama2-70B-OASST-SFT-v10-GPTQ	67.75	96.9	61.4	48.2	64.5	70	4096	4
aaditya/Llama3-OpenBioLLM-70B	66.93	96.3	52.5	47.9	71	70	8192	4
deepseek-ai/DeepSeek-R1-Distill-Llama-70B	66.1	96.2	50.2	49.8	68.2	70	8192	4
TheBloke/WizardLM-70B-V1.0-GPTQ	65.1	96.4	37.3	52.9	73.8	70	4096	4
TheBloke/meditron-70B-GPTQ	58.92	96.7	19.4	44.7	74.9	70	4096	4
TheBloke/Llama-2-70B-Chat-GPTQ	58.8	95.1	48.1	44	48	70	4096	4

How it works

Reproducibility

To reproduce our results, here is the commands you can run:

For MIMIC-CDM, navigate to the MIMIC-Clinical-Decision-Making-Framework repository and execute:

python run.py pathology=appendicitis model=<YOUR_MODEL_NAME>
python run.py pathology=cholecystitis model=<YOUR_MODEL_NAME>
python run.py pathology=pancreatitis model=<YOUR_MODEL_NAME>
python run.py pathology=diverticulitis model=<YOUR_MODEL_NAME>

For MIMIC-CDM-FI, navigate to the MIMIC-Clinical-Decision-Making-Framework repository and execute:

python run_full_info.py pathology=appendicitis model=<YOUR_MODEL_NAME>
python run_full_info.py pathology=cholecystitis model=<YOUR_MODEL_NAME>
python run_full_info.py pathology=pancreatitis model=<YOUR_MODEL_NAME>
python run_full_info.py pathology=diverticulitis model=<YOUR_MODEL_NAME>

MIMIC Clinical Decision Making

How it works

Reproducibility