oobabooga benchmark

Score	Model	Size	Loader	Additional info
34/48	platypus-yi-34b.Q8_0	34B	llamacpp_HF
34/48	Meta-Llama-3-70B-Instruct-Q4_K_S	70B	llamacpp_HF	[link]
34/48	LoneStriker_OpenBioLLM-Llama3-70B-6.0bpw-h6-exl2	70B	ExLlamav2_HF
33/48	turboderp_Llama-3-70B-Instruct-exl2_6.0bpw	70B	ExLlamav2_HF
33/48	turboderp_Llama-3-70B-Instruct-exl2_5.0bpw	70B	ExLlamav2_HF
33/48	turboderp_Cat-Llama-3-70B-instruct-exl2_5.0bpw	70B	ExLlamav2_HF
33/48	Undi95_Meta-Llama-3-70B-Instruct-hf	70B	Transformers	--load-in-4bit
33/48	Meta-Llama-3-70B-Instruct.Q8_0	70B	llamacpp_HF
33/48	Meta-Llama-3-70B-Instruct.Q4_K_M	70B	llamacpp_HF
33/48	Meta-Llama-3-70B-Instruct-Q3_K_S	70B	llamacpp_HF	[link]
33/48	Meta-Llama-3-70B-Instruct-IQ3_XXS	70B	llamacpp_HF	[link]
33/48	Meta-Llama-3-70B-Instruct-IQ3_XS	70B	llamacpp_HF	[link]
33/48	LoneStriker_dolphin-2.9-llama3-70b-6.0bpw-h6-exl2	70B	ExLlamav2_HF
33/48	Llama3-TenyxChat-70B.i1-Q4_K_S	70B	llamacpp_HF
33/48	ISTA-DASLab_Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16	70B	Transformers
32/48	Meta-Llama-3-70B-Instruct-Q3_K_M	70B	llamacpp_HF	[link]
32/48	Meta-Llama-3-70B-Instruct-Q3_K_L	70B	llamacpp_HF	[link]
32/48	Meta-Llama-3-70B-Instruct-IQ4_XS	70B	llamacpp_HF	[link]
32/48	Meta-Llama-3-70B-Instruct-IQ4_NL	70B	llamacpp_HF	[link]
32/48	Meta-Llama-3-70B-Instruct-IQ3_S	70B	llamacpp_HF	[link]
32/48	Meta-Llama-3-70B-Instruct-IQ2_S	70B	llamacpp_HF	[link]
32/48	Meta-Llama-3-70B-Instruct-IQ2_M	70B	llamacpp_HF	[link]
32/48	Llama-3-Giraffe-70B-Instruct.i1-Q4_K_S	70B	llamacpp_HF	[link]
31/48	oobabooga_miqu-1-70b-sf-EXL2-6.000b	70B	ExLlamav2_HF
31/48	miqu-1-70b.q5_K_M	70B	llamacpp_HF
31/48	miqu-1-70b.q4_k_m	70B	llamacpp_HF
31/48	cloudyu_Phoenix_DPO_60B	60B	Transformers	--load-in-8bit
31/48	Meta-Llama-3-70B-Instruct-Q2_K	70B	llamacpp_HF	[link]
31/48	Meta-Llama-3-70B-Instruct-IQ3_M	70B	llamacpp_HF	[link]
31/48	Meta-Llama-3-70B-Instruct-IQ2_XS	70B	llamacpp_HF	[link]
31/48	Meta-Llama-3-70B-Instruct-IQ2_XS	70B	llamacpp_HF	[link]
31/48	Meta-Llama-3-120B-Instruct.Q4_K_M	120B	llamacpp_HF	[link]
30/48	turboderp_Mixtral-8x22B-Instruct-v0.1-exl2_4.0bpw	8x22B	ExLlamav2_HF
30/48	qwen1_5-72b-chat-q4_k_m	72B	llamacpp_HF
30/48	miquliz-120b-v2.0.Q4_K_M	120B	llamacpp_HF
30/48	falcon-180b-chat.Q4_K_M	180B	llamacpp_HF
30/48	Senku-70B-Full-Q4_K_M	70B	llamacpp_HF
30/48	Rhea-72b-v0.5-Q4_K_M	72B	llamacpp_HF
30/48	Dracones_WizardLM-2-8x22B_exl2_4.0bpw	8x22B	ExLlamav2_HF
29/48	turboderp_command-r-plus-103B-exl2_3.0bpw	104B	ExLlamav2_HF
29/48	turboderp_Llama-3-70B-exl2_5.0bpw	70B	ExLlamav2_HF
29/48	daybreak-miqu-1-70b-v1.0-q5_k_m	70B	llamacpp_HF
29/48	command-r-plus-104b-iq4_xs	104B	llamacpp_HF
29/48	bartowski_Qwen1.5-32B-Chat-exl2_5_0	32B	ExLlamav2_HF
29/48	Qwen1.5-110B-Chat-Q4_K_M	110B	llamacpp_HF	[link]
29/48	LoneStriker_Yi-34B-Chat-8.0bpw-h8-exl2	34B	ExLlamav2_HF
29/48	Llama3-ChatQA-1.5-70B.Q4_K_M	70B	llamacpp_HF	Alpaca template.
29/48	Dracones_Llama-3-Lumimaid-70B-v0.1_exl2_4.5bpw	70B	ExLlamav2_HF
29/48	34b-beta.Q8_0	34B	llamacpp_HF
28/48	turboderp_command-r-plus-103B-exl2_4.5bpw	104B	ExLlamav2_HF
28/48	turboderp_command-r-plus-103B-exl2_3.5bpw	104B	ExLlamav2_HF
28/48	dolphin-2.7-mixtral-8x7b.Q8_0	8x7B	llamacpp_HF
28/48	command-r-plus-Q4_K_M	104B	llamacpp_HF
28/48	LoneStriker_Smaug-72B-v0.1-6.0bpw-h6-exl2	72B	ExLlamav2_HF
27/48	mixtral-8x7b-instruct-v0.1.Q8_0	8x7B	llamacpp_HF
27/48	miqu-1-70b.q2_K	70B	llamacpp_HF
27/48	TheBloke_Helion-4x34B-GPTQ	4x34B	ExLlamav2_HF
27/48	Platypus2-70B.i1-Q4_K_M	70B	llamacpp_HF	[link]
27/48	Midnight-Miqu-70B-v1.0.Q4_K_M	70B	llamacpp_HF
27/48	Llama3-ChatQA-1.5-70B.Q4_K_M	70B	llamacpp_HF	NVIDIA-ChatQA template.
27/48	ISTA-DASLab_c4ai-command-r-plus-AQLM-2Bit-1x16	104B	Transformers
26/48	nous-hermes-2-mixtral-8x7b-dpo.Q8_0	8x7B	llamacpp_HF
26/48	lzlv_70b_fp16_hf.Q5_K_M	70B	llamacpp_HF
26/48	Qwen_Qwen1.5-14B-Chat	14B	Transformers
26/48	Mixtral-8x22B-Instruct-v0.1.Q4_K_M	8x22B	llamacpp_HF
26/48	Meta-Llama-3-70B-Instruct-IQ2_XXS	70B	llamacpp_HF
26/48	LoneStriker_dolphin-2.2-yi-34b-200k-8.0bpw-h8-exl2	34B	ExLlamav2_HF
26/48	LoneStriker_Yi-34B-200K-8.0bpw-h8-exl2	34B	ExLlamav2_HF
26/48	CausalLM-RP-34B.q8_0	34B	llamacpp_HF
25/48	turboderp_Llama-3-70B-Instruct-exl2_2.4bpw	70B	ExLlamav2_HF
25/48	goliath-120b.Q4_K_M	120B	llamacpp_HF
25/48	NousResearch_Nous-Hermes-2-SOLAR-10.7B	10.7B	Transformers
25/48	LoneStriker_Llama-3-70B-Instruct-Gradient-524k-6.0bpw-h6-exl2	70B	ExLlamav2_HF
24/48	upstage_SOLAR-10.7B-Instruct-v1.0	10.7B	Transformers
24/48	maid-yuzu-v8-alter.Q8_0	8x7B	llamacpp_HF
24/48	MultiVerse_70B.Q4_K_M	70B	llamacpp_HF
24/48	LoneStriker_Llama-3-70B-Instruct-Gradient-262k-6.0bpw-h6-exl2	70B	ExLlamav2_HF
23/48	xwin-lm-70b-v0.1.Q4_K_M	70B	llamacpp_HF
23/48	microsoft_Phi-3-mini-4k-instruct	3.8B	Transformers
23/48	bhenrym14_airoboros-3_1-yi-34b-200k	34B	Transformers	--load-in-8bit
22/48	wizardlm-70b-v1.0.Q4_K_M	70B	llamacpp_HF
22/48	turboderp_command-r-v01-35B-exl2_6.0bpw	35B	ExLlamav2_HF
22/48	turboderp_command-r-plus-103B-exl2_2.5bpw	104B	ExLlamav2_HF
22/48	tulu-2-dpo-70b.Q4_K_M	70B	llamacpp_HF
22/48	meraGPT_mera-mix-4x7B	4x7B	Transformers
22/48	liuhaotian_llava-v1.5-13b	13B	Transformers
22/48	Qwen_Qwen1.5-7B-Chat	7B	Transformers
22/48	MoMo-72B-lora-1.8.6-DPO-Q4_K_M	72B	llamacpp_HF
22/48	Meta-Llama-3-8B-Instruct-Q4_K_S	8B	llamacpp_HF
21/48	internlm_internlm2-chat-20b	20B	Transformers
21/48	falcon-180b.Q4_K_M	180B	llamacpp_HF
21/48	c4ai-command-r-v01-Q8_0	35B	llamacpp_HF
21/48	Undi95_Meta-Llama-3-8B-Instruct-hf	8B	Transformers
21/48	NurtureAI_Meta-Llama-3-8B-Instruct-64k	8B	Transformers
21/48	Meta-Llama-3-8B-Instruct-fp16	8B	llamacpp_HF
21/48	Meta-Llama-3-8B-Instruct-Q8_0	8B	llamacpp_HF
20/48	openchat_openchat_3.5	7B	Transformers
20/48	llama-2-70b-chat.Q4_K_M	70B	llamacpp_HF
20/48	Weyaxi_Einstein-v6.1-Llama3-8B	8B	Transformers
20/48	TheBloke_llava-v1.5-13B-GPTQ	13B	ExLlamav2_HF
20/48	Meta-Llama-3-8B-Instruct-Q6_K	8B	llamacpp_HF
20/48	Ein-72B-v0.1-full.Q4_K_M	72B	llamacpp_HF
20/48	BAAI_Bunny-Llama-3-8B-V	8B	Transformers
19/48	zephyr-orpo-141b-A35b-v0.1.Q4_K_M	141B	llamacpp_HF
19/48	mistral-7b-instruct-v0.2.Q4_K_S	7B	llamacpp_HF
19/48	microsoft_Phi-3-mini-128k-instruct	3.8B	Transformers
19/48	llama-2-70b.Q5_K_M	70B	llamacpp_HF
19/48	lightblue_suzume-llama-3-8B-multilingual	8B	Transformers
19/48	internlm_internlm2-chat-7b	7B	Transformers
19/48	internlm_internlm2-chat-20b-sft	20B	Transformers
19/48	ai21labs_Jamba-v0.1	52B	Transformers	--load-in-4bit
19/48	NousResearch_Hermes-2-Pro-Mistral-7B	7B	Transformers
19/48	Nexusflow_Starling-LM-7B-beta	7B	Transformers
19/48	Meta-Llama-3-8B-Instruct-Q5_K_S	8B	llamacpp_HF
19/48	Meta-Llama-3-8B-Instruct-Q5_K_M	8B	llamacpp_HF
19/48	Meta-Llama-3-8B-Instruct-Q4_K_M	8B	llamacpp_HF
19/48	Meta-Llama-3-8B-Instruct-IQ4_XS	8B	llamacpp_HF
19/48	Meta-Llama-3-8B-Instruct-IQ4_NL	8B	llamacpp_HF
19/48	Meta-Llama-3-8B-Instruct-IQ3_S	8B	llamacpp_HF	[link]
18/48	mistralai_Mistral-7B-Instruct-v0.2	7B	Transformers
18/48	microsoft_Phi-3-mini-128k-instruct	3.8B	Transformers	--load-in-8bit
18/48	jieliu_Storm-7B	7B	Transformers
18/48	failspy_kappa-3-phi-abliterated	3.8B	Transformers
18/48	TheProfessor-155b.i1-IQ3_XS	155B	llamacpp_HF	[link]
18/48	Qwen_Qwen1.5-MoE-A2.7B-Chat	14.3B	Transformers
18/48	Orenguteng_Lexi-Llama-3-8B-Uncensored	8B	Transformers
18/48	Meta-Llama-3-8B-Instruct-Q3_K_M	8B	llamacpp_HF
18/48	Meta-Llama-3-8B-Instruct-Q3_K_L	8B	llamacpp_HF
18/48	Meta-Llama-3-70B-Instruct-IQ1_M	70B	llamacpp_HF
18/48	LoneStriker_Nous-Capybara-34B-4.65bpw-h6-exl2	34B	ExLlamav2_HF
17/48	turboderp_Phi-3-mini-128k-instruct-exl2_6.0bpw	3.8B	ExLlamav2_HF
17/48	turboderp_Phi-3-mini-128k-instruct-exl2_5.0bpw	3.8B	ExLlamav2_HF
17/48	mzbac_llama-3-8B-Instruct-function-calling	8B	Transformers
17/48	microsoft_Phi-3-mini-128k-instruct	3.8B	Transformers	--load-in-4bit
17/48	internlm_internlm2-chat-7b-sft	7B	Transformers
17/48	grok-1-IQ2_XS	314B	llamacpp_HF	[link]
17/48	ggml-alpaca-dragon-72b-v1-q4_k_m	72B	llamacpp_HF
17/48	amazingvince_Not-WizardLM-2-7B	7B	Transformers
17/48	Undi95_Toppy-M-7B	7B	Transformers
16/48	mixtral-8x7b-instruct-v0.1.Q2_K	8x7B	llamacpp_HF
16/48	TheBloke_Mistral-7B-Instruct-v0.2-GPTQ	7B	ExLlamav2_HF
16/48	Phi-3-mini-128k-instruct-Q6_K	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
16/48	Phi-3-mini-128k-instruct-Q5_0	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
16/48	Phi-3-mini-128k-instruct-Q4_K_S	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
16/48	Phi-3-mini-128k-instruct-Q4_K_M	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
16/48	Phi-3-mini-128k-instruct-Q4_K	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
16/48	Meta-Llama-3-8B-Instruct-IQ3_M	8B	llamacpp_HF
15/48	xtuner_llava-llama-3-8b-v1_1	8B	Transformers
15/48	hjhj3168_Llama-3-8b-Orthogonalized-exl2	8B	ExLlamav2_HF
15/48	cognitivecomputations_dolphin-2.9-llama3-8b	8B	Transformers
15/48	Phi-3-mini-128k-instruct-Q5_0	3.8B	llamacpp_HF	Created without --imatrix.
15/48	Phi-3-mini-128k-instruct-Q4_0	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
15/48	Phi-3-mini-128k-instruct-IQ4_XS	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
15/48	Phi-3-mini-128k-instruct-IQ4_NL	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
15/48	NousResearch_Hermes-2-Pro-Llama-3-8B	8B	Transformers
15/48	Meta-Llama-3-8B-Instruct-IQ3_XS	8B	llamacpp_HF
15/48	CohereForAI_c4ai-command-r-v01-4bit	35B	Transformers
14/48	turboderp_Phi-3-mini-128k-instruct-exl2_4.0bpw	3.8B	ExLlamav2_HF
14/48	nvidia_ChatQA-1.5-8B	8B	Transformers	Alpaca template.
14/48	microsoft_Orca-2-13b	13B	Transformers
14/48	mattshumer_Llama-3-8B-16K	8B	Transformers
14/48	Undi95_ReMM-SLERP-L2-13B	13B	Transformers
14/48	Phi-3-mini-128k-instruct-Q5_K_S	3.8B	llamacpp_HF	Created without --imatrix.
14/48	Phi-3-mini-128k-instruct-Q5_K_M	3.8B	llamacpp_HF	Created without --imatrix.
14/48	Phi-3-mini-128k-instruct-Q5_K	3.8B	llamacpp_HF	Created without --imatrix.
14/48	Phi-3-mini-128k-instruct-Q5_1	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
14/48	Phi-3-mini-128k-instruct-Q4_K_M	3.8B	llamacpp_HF	Created without --imatrix.
14/48	Phi-3-mini-128k-instruct-Q4_K	3.8B	llamacpp_HF	Created without --imatrix.
14/48	Phi-3-mini-128k-instruct-Q4_1	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
14/48	Phi-3-mini-128k-instruct-F16	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
14/48	Phi-3-mini-128k-instruct-F16	3.8B	llamacpp_HF	Created without --imatrix.
14/48	Meta-Llama-3-8B-Instruct-Q3_K_S	8B	llamacpp_HF
14/48	Meta-Llama-3-8B-Instruct-IQ3_XXS	8B	llamacpp_HF
14/48	Gryphe_MythoMax-L2-13b	13B	Transformers
13/48	turboderp_dbrx-instruct-exl2_3.75bpw	132B	ExLlamav2_HF	Without the "You are DBRX..." system prompt.
13/48	nvidia_ChatQA-1.5-8B	8B	Transformers	NVIDIA-ChatQA template.
13/48	gradientai_Llama-3-8B-Instruct-Gradient-1048k	8B	Transformers
13/48	gradientai_Llama-3-8B-Instruct-262k	8B	Transformers
13/48	alpindale_gemma-7b-it	7B	Transformers
13/48	Phi-3-mini-128k-instruct-Q8_0	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
13/48	Phi-3-mini-128k-instruct-Q8_0	3.8B	llamacpp_HF	Created without --imatrix.
13/48	Phi-3-mini-128k-instruct-Q6_K	3.8B	llamacpp_HF	Created without --imatrix.
13/48	Phi-3-mini-128k-instruct-Q5_K_S	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
13/48	Phi-3-mini-128k-instruct-Q5_K_M	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
13/48	Phi-3-mini-128k-instruct-Q5_K	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
13/48	Phi-3-mini-128k-instruct-Q5_1	3.8B	llamacpp_HF	Created without --imatrix.
13/48	Phi-3-mini-128k-instruct-Q4_0	3.8B	llamacpp_HF	Created without --imatrix.
13/48	Phi-3-mini-128k-instruct-F32	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
13/48	Phi-3-mini-128k-instruct-F32	3.8B	llamacpp_HF	Created without --imatrix.
13/48	GeorgiaTechResearchInstitute_galactica-30b-evol-instruct-70k	30B	Transformers	GALACTICA template.
12/48	llama-65b.Q5_K_M	65B	llamacpp_HF
12/48	Phi-3-mini-128k-instruct-Q3_K_M	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
12/48	Phi-3-mini-128k-instruct-Q3_K_L	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
12/48	Phi-3-mini-128k-instruct-Q3_K	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
12/48	Phi-3-mini-128k-instruct-IQ3_M	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
12/48	NousResearch_Llama-2-13b-chat-hf	13B	Transformers
12/48	Meta-Llama-3-70B-Instruct-IQ1_S	70B	llamacpp_HF
12/48	ISTA-DASLab_Meta-Llama-3-8B-Instruct-AQLM-2Bit-1x16	8B	Transformers
12/48	HuggingFaceH4_zephyr-7b-beta	7B	Transformers
11/48	mlabonne_phixtral-2x2_8	2x2.8B	Transformers
11/48	Phi-3-mini-128k-instruct-Q4_K_S	3.8B	llamacpp_HF	Created without --imatrix.
11/48	Phi-3-mini-128k-instruct-Q4_1	3.8B	llamacpp_HF	Created without --imatrix.
11/48	Phi-3-mini-128k-instruct-Q3_K_M	3.8B	llamacpp_HF	Created without --imatrix.
11/48	Phi-3-mini-128k-instruct-Q3_K	3.8B	llamacpp_HF	Created without --imatrix.
11/48	Phi-3-mini-128k-instruct-IQ3_S	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
11/48	Meta-Llama-3-8B-Instruct-Q2_K	8B	llamacpp_HF
11/48	Meta-Llama-3-8B-Instruct-IQ2_M	8B	llamacpp_HF
10/48	facebook_galactica-30b	30B	Transformers
10/48	Phi-3-mini-128k-instruct-Q3_K_S	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
10/48	Phi-3-mini-128k-instruct-Q3_K_L	3.8B	llamacpp_HF	Created without --imatrix.
10/48	ISTA-DASLab_c4ai-command-r-v01-AQLM-2Bit-1x16	35B	Transformers
9/48	microsoft_phi-2	2.7B	Transformers
9/48	TheBloke_vicuna-33B-GPTQ	33B	ExLlamav2_HF
8/48	mistralai_Mistral-7B-Instruct-v0.1	7B	Transformers
8/48	gradientai_Llama-3-8B-Instruct-Gradient-1048k	8B	Transformers	Revision of 2024/05/04.
8/48	Phi-3-mini-128k-instruct-Q3_K_S	3.8B	llamacpp_HF	Created without --imatrix.
8/48	Phi-3-mini-128k-instruct-IQ3_XXS	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
8/48	Phi-3-mini-128k-instruct-IQ3_XS	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
8/48	NousResearch_Nous-Capybara-7B-V1.9	7B	Transformers
8/48	NousResearch_Llama-2-7b-chat-hf	7B	Transformers
8/48	Meta-Llama-3-8B-Instruct-IQ2_S	8B	llamacpp_HF
8/48	ISTA-DASLab_Meta-Llama-3-8B-Instruct-AQLM-2Bit-1x16	8B	Transformers
7/48	GeorgiaTechResearchInstitute_galactica-30b-evol-instruct-70k	30B	Transformers	Alpaca template.
6/48	tiiuae_falcon-40b-instruct	40B	Transformers	--load-in-8bit; falcon-180B-chat instruction template.
5/48	unsloth_llama-3-70b-bnb-4bit	70B	Transformers
5/48	internlm_internlm2-chat-1_8b-sft	1.8B	Transformers
5/48	TheBloke_Llama-2-13B-GPTQ	13B	ExLlamav2_HF
5/48	Phi-3-mini-128k-instruct-Q2_K	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
5/48	NousResearch_Llama-2-13b-hf	13B	Transformers
5/48	Meta-Llama-3-8B-Instruct-IQ2_XS	8B	llamacpp_HF
5/48	LoneStriker_deepseek-coder-33b-instruct-6.0bpw-h6-exl2	33B	ExLlamav2_HF
4/48	turboderp_Phi-3-mini-128k-instruct-exl2_3.0bpw	3.8B	ExLlamav2_HF
4/48	internlm_internlm2-chat-1_8b	1.8B	Transformers
4/48	TheBloke_deepseek-coder-33B-instruct-AWQ	33B	AutoAWQ
3/48	turboderp_dbrx-instruct-exl2_3.75bpw	132B	ExLlamav2_HF
3/48	TheBloke_Llama-2-7B-GPTQ	7B	ExLlamav2_HF
2/48	facebook_galactica-6.7b	6.7B	Transformers
2/48	Meta-Llama-3-8B-Instruct-IQ2_XXS	8B	llamacpp_HF
1/48	turboderp_Phi-3-mini-128k-instruct-exl2_2.5bpw	3.8B	ExLlamav2_HF
1/48	bartowski_CodeQwen1.5-7B-Chat-exl2_8_0	7B	ExLlamav2_HF
1/48	Qwen_CodeQwen1.5-7B-Chat	7B	Transformers
1/48	Phi-3-mini-128k-instruct-Q2_K_S	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
1/48	Phi-3-mini-128k-instruct-IQ2_M	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
1/48	NousResearch_Llama-2-7b-hf	7B	Transformers
0/48	openai-community_gpt2-xl	1.5B	Transformers
0/48	openai-community_gpt2-medium	0.355B	Transformers
0/48	openai-community_gpt2-large	0.774B	Transformers
0/48	openai-community_gpt2	0.124B	Transformers
0/48	gpt4chan_model_float16	6B	Transformers
0/48	facebook_opt-6.7b	6.7B	Transformers
0/48	facebook_opt-30b	30B	Transformers
0/48	facebook_opt-13b	13B	Transformers
0/48	facebook_galactica-125m	0.125B	Transformers
0/48	facebook_galactica-1.3b	1.3B	Transformers
0/48	TinyLlama_TinyLlama-1.1B-Chat-v1.0	1.1B	Transformers
0/48	Phi-3-mini-128k-instruct-Q2_K	3.8B	llamacpp_HF	Created without --imatrix.
0/48	Phi-3-mini-128k-instruct-IQ2_XXS	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
0/48	Phi-3-mini-128k-instruct-IQ2_XS	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
0/48	Phi-3-mini-128k-instruct-IQ2_S	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
0/48	Phi-3-mini-128k-instruct-IQ1_S	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
0/48	Phi-3-mini-128k-instruct-IQ1_M	3.8B	llamacpp_HF	Created with groups_merged.txt for calibration.
0/48	Meta-Llama-3-8B-Instruct-IQ1_S	8B	llamacpp_HF
0/48	Meta-Llama-3-8B-Instruct-IQ1_M	8B	llamacpp_HF
0/48	ISTA-DASLab_Llama-2-7b-AQLM-2Bit-1x16-hf	7B	Transformers
0/48	EleutherAI_gpt-neox-20b	20B	Transformers
0/48	EleutherAI_gpt-neo-2.7B	2.7B	Transformers
0/48	EleutherAI_gpt-neo-1.3B	1.3B	Transformers
0/48	EleutherAI_gpt-j-6b	6B	Transformers

Updates

2024/05/10

Dracones_Llama-3-Lumimaid-70B-v0.1_exl2_4.5bpw (70B) - 29/48

2024/05/07

turboderp_command-r-plus-103B-exl2_3.0bpw (104B) - 29/48
turboderp_command-r-plus-103B-exl2_2.5bpw (104B) - 22/48
Qwen_CodeQwen1.5-7B-Chat (7B) - 1/48
bartowski_CodeQwen1.5-7B-Chat-exl2_8_0 (7B) - 1/48
LoneStriker_Llama-3-70B-Instruct-Gradient-262k-6.0bpw-h6-exl2 (70B) - 24/48
LoneStriker_Llama-3-70B-Instruct-Gradient-524k-6.0bpw-h6-exl2 (70B) - 25/48
Llama3-TenyxChat-70B.i1-Q4_K_S (70B) - 33/48

2024/05/06

Llama-3-Giraffe-70B-Instruct.i1-Q4_K_S (70B) - 32/48
turboderp_Cat-Llama-3-70B-instruct-exl2_5.0bpw (70B) - 33/48
failspy_kappa-3-phi-abliterated (3.8B) - 18/48
turboderp_command-r-plus-103B-exl2_3.5bpw (104B) - 28/48
turboderp_command-r-plus-103B-exl2_4.5bpw (104B) - 28/48

2024/05/05

Platypus2-70B.i1-Q4_K_M (70B) - 27/48
LoneStriker_Yi-34B-Chat-8.0bpw-h8-exl2 (34B) - 29/48
LoneStriker_dolphin-2.2-yi-34b-200k-8.0bpw-h8-exl2 (34B) - 26/48
Phi-3-mini-128k-instruct-Q2_K (3.8B) - 0/48
Phi-3-mini-128k-instruct-Q3_K (3.8B) - 11/48
Phi-3-mini-128k-instruct-Q3_K_L (3.8B) - 10/48
Phi-3-mini-128k-instruct-Q3_K_M (3.8B) - 11/48
Phi-3-mini-128k-instruct-Q3_K_S (3.8B) - 8/48
Phi-3-mini-128k-instruct-Q4_0 (3.8B) - 13/48
Phi-3-mini-128k-instruct-Q4_1 (3.8B) - 11/48
Phi-3-mini-128k-instruct-Q4_K (3.8B) - 14/48
Phi-3-mini-128k-instruct-Q4_K_M (3.8B) - 14/48
Phi-3-mini-128k-instruct-Q4_K_S (3.8B) - 11/48
Phi-3-mini-128k-instruct-Q5_0 (3.8B) - 15/48
Phi-3-mini-128k-instruct-Q5_1 (3.8B) - 13/48
Phi-3-mini-128k-instruct-Q5_K (3.8B) - 14/48
Phi-3-mini-128k-instruct-Q5_K_M (3.8B) - 14/48
Phi-3-mini-128k-instruct-Q5_K_S (3.8B) - 14/48
Phi-3-mini-128k-instruct-Q6_K (3.8B) - 13/48
Phi-3-mini-128k-instruct-Q8_0 (3.8B) - 13/48
Phi-3-mini-128k-instruct-F16 (3.8B) - 14/48
Phi-3-mini-128k-instruct-F32 (3.8B) - 13/48
LoneStriker_Yi-34B-200K-8.0bpw-h8-exl2 (34B) - 26/48
microsoft_Phi-3-mini-128k-instruct (3.8B) - 18/48
microsoft_Phi-3-mini-128k-instruct (3.8B) - 17/48
Meta-Llama-3-120B-Instruct.Q4_K_M (120B) - 31/48

2024/05/04

jieliu_Storm-7B (7B) - 18/48
miquliz-120b-v2.0.Q4_K_M (120B) - 30/48
internlm_internlm2-chat-1_8b (1.8B) - 4/48
internlm_internlm2-chat-1_8b-sft (1.8B) - 5/48
internlm_internlm2-chat-20b (20B) - 21/48
internlm_internlm2-chat-20b-sft (20B) - 19/48
internlm_internlm2-chat-7b (7B) - 19/48
internlm_internlm2-chat-7b-sft (7B) - 17/48
TheProfessor-155b.i1-IQ3_XS (155B) - 18/48
ISTA-DASLab_c4ai-command-r-plus-AQLM-2Bit-1x16 (104B) - 27/48
ISTA-DASLab_c4ai-command-r-v01-AQLM-2Bit-1x16 (35B) - 10/48
ISTA-DASLab_Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16 (70B) - 33/48
ISTA-DASLab_Meta-Llama-3-8B-Instruct-AQLM-2Bit-1x16 (8B) - 12/48
gradientai_Llama-3-8B-Instruct-Gradient-1048k (8B) - 8/48

2024/05/03

gradientai_Llama-3-8B-Instruct-Gradient-1048k (8B) - 13/48
Phi-3-mini-128k-instruct-F16 (3.8B) - 14/48
Phi-3-mini-128k-instruct-F32 (3.8B) - 13/48
Phi-3-mini-128k-instruct-IQ1_M (3.8B) - 0/48
Phi-3-mini-128k-instruct-IQ1_S (3.8B) - 0/48
Phi-3-mini-128k-instruct-IQ2_M (3.8B) - 1/48
Phi-3-mini-128k-instruct-IQ2_S (3.8B) - 0/48
Phi-3-mini-128k-instruct-IQ2_XS (3.8B) - 0/48
Phi-3-mini-128k-instruct-IQ2_XXS (3.8B) - 0/48
Phi-3-mini-128k-instruct-IQ3_M (3.8B) - 12/48
Phi-3-mini-128k-instruct-IQ3_S (3.8B) - 11/48
Phi-3-mini-128k-instruct-IQ3_XS (3.8B) - 8/48
Phi-3-mini-128k-instruct-IQ3_XXS (3.8B) - 8/48
Phi-3-mini-128k-instruct-IQ4_NL (3.8B) - 15/48
Phi-3-mini-128k-instruct-IQ4_XS (3.8B) - 15/48
Phi-3-mini-128k-instruct-Q2_K (3.8B) - 5/48
Phi-3-mini-128k-instruct-Q2_K_S (3.8B) - 1/48
Phi-3-mini-128k-instruct-Q3_K (3.8B) - 12/48
Phi-3-mini-128k-instruct-Q3_K_L (3.8B) - 12/48
Phi-3-mini-128k-instruct-Q3_K_M (3.8B) - 12/48
Phi-3-mini-128k-instruct-Q3_K_S (3.8B) - 10/48
Phi-3-mini-128k-instruct-Q4_0 (3.8B) - 15/48
Phi-3-mini-128k-instruct-Q4_1 (3.8B) - 14/48
Phi-3-mini-128k-instruct-Q4_K (3.8B) - 16/48
Phi-3-mini-128k-instruct-Q4_K_M (3.8B) - 16/48
Phi-3-mini-128k-instruct-Q4_K_S (3.8B) - 16/48
Phi-3-mini-128k-instruct-Q5_0 (3.8B) - 16/48
Phi-3-mini-128k-instruct-Q5_1 (3.8B) - 14/48
Phi-3-mini-128k-instruct-Q5_K (3.8B) - 13/48
Phi-3-mini-128k-instruct-Q5_K_M (3.8B) - 13/48
Phi-3-mini-128k-instruct-Q5_K_S (3.8B) - 13/48
Phi-3-mini-128k-instruct-Q6_K (3.8B) - 16/48
Phi-3-mini-128k-instruct-Q8_0 (3.8B) - 13/48
turboderp_Phi-3-mini-128k-instruct-exl2_2.5bpw (3.8B) - 1/48
turboderp_Phi-3-mini-128k-instruct-exl2_3.0bpw (3.8B) - 4/48
turboderp_Phi-3-mini-128k-instruct-exl2_4.0bpw (3.8B) - 14/48
turboderp_Phi-3-mini-128k-instruct-exl2_5.0bpw (3.8B) - 17/48
turboderp_Phi-3-mini-128k-instruct-exl2_6.0bpw (3.8B) - 17/48
hjhj3168_Llama-3-8b-Orthogonalized-exl2 (8B) - 15/48
ai21labs_Jamba-v0.1 (52B) - 19/48
Llama3-ChatQA-1.5-70B.Q4_K_M (70B) - 29/48
Llama3-ChatQA-1.5-70B.Q4_K_M (70B) - 27/48
nvidia_ChatQA-1.5-8B (8B) - 13/48
nvidia_ChatQA-1.5-8B (8B) - 14/48
NousResearch_Hermes-2-Pro-Llama-3-8B (8B) - 15/48
TheBloke_deepseek-coder-33B-instruct-AWQ (33B) - 4/48
LoneStriker_deepseek-coder-33b-instruct-6.0bpw-h6-exl2 (33B) - 5/48

2024/04/28

facebook_galactica-1.3b (1.3B) - 0/48
facebook_opt-6.7b (6.7B) - 0/48
facebook_opt-13b (13B) - 0/48
facebook_opt-30b (30B) - 0/48
GeorgiaTechResearchInstitute_galactica-30b-evol-instruct-70k (30B) - 13/48
GeorgiaTechResearchInstitute_galactica-30b-evol-instruct-70k (30B) - 7/48
EleutherAI_gpt-j-6b (6B) - 0/48
EleutherAI_gpt-neo-2.7B (2.7B) - 0/48
EleutherAI_gpt-neo-1.3B (1.3B) - 0/48
openai-community_gpt2 (0.124B) - 0/48
openai-community_gpt2-medium (0.355B) - 0/48
openai-community_gpt2-large (0.774B) - 0/48
openai-community_gpt2-xl (1.5B) - 0/48
EleutherAI_gpt-neox-20b (20B) - 0/48

2024/04/27

Undi95_Meta-Llama-3-70B-Instruct-hf (70B) - 33/48
LoneStriker_dolphin-2.9-llama3-70b-6.0bpw-h6-exl2 (70B) - 33/48
LoneStriker_OpenBioLLM-Llama3-70B-6.0bpw-h6-exl2 (70B) - 34/48
BAAI_Bunny-Llama-3-8B-V (8B) - 20/48
mzbac_llama-3-8B-Instruct-function-calling (8B) - 17/48

2024/04/26

Weyaxi_Einstein-v6.1-Llama3-8B (8B) - 20/48
Qwen1.5-110B-Chat-Q4_K_M (110B) - 29/48
Meta-Llama-3-70B-Instruct-IQ2_M (70B) - 32/48
Meta-Llama-3-70B-Instruct-IQ2_S (70B) - 32/48
Meta-Llama-3-70B-Instruct-IQ2_XS (70B) - 31/48
Meta-Llama-3-70B-Instruct-IQ3_M (70B) - 31/48
Meta-Llama-3-70B-Instruct-IQ3_S (70B) - 32/48
Meta-Llama-3-70B-Instruct-IQ3_XS (70B) - 33/48
Meta-Llama-3-70B-Instruct-IQ3_XXS (70B) - 33/48
Meta-Llama-3-70B-Instruct-IQ4_NL (70B) - 32/48
Meta-Llama-3-70B-Instruct-IQ4_XS (70B) - 32/48
Meta-Llama-3-70B-Instruct-Q2_K (70B) - 31/48
Meta-Llama-3-70B-Instruct-Q3_K_L (70B) - 32/48
Meta-Llama-3-70B-Instruct-Q3_K_M (70B) - 32/48
Meta-Llama-3-70B-Instruct-Q3_K_S (70B) - 33/48
Meta-Llama-3-70B-Instruct-Q4_K_S (70B) - 34/48

2024/04/25

34b-beta.Q8_0 (34B) - 29/48
platypus-yi-34b.Q8_0 (34B) - 34/48
CausalLM-RP-34B.q8_0 (34B) - 26/48
MoMo-72B-lora-1.8.6-DPO-Q4_K_M (72B) - 22/48
ggml-alpaca-dragon-72b-v1-q4_k_m (72B) - 17/48
cloudyu_Phoenix_DPO_60B (60B) - 31/48
TheBloke_Helion-4x34B-GPTQ (4x34B) - 27/48
bhenrym14_airoboros-3_1-yi-34b-200k (34B) - 23/48
gradientai_Llama-3-8B-Instruct-262k (8B) - 13/48

2024/04/24

falcon-180b-chat.Q4_K_M (180B) - 30/48
falcon-180b.Q4_K_M (180B) - 21/48
grok-1-IQ2_XS (314B) - 17/48
lightblue_suzume-llama-3-8B-multilingual (8B) - 19/48
tiiuae_falcon-40b-instruct (40B) - 6/48
LoneStriker_Smaug-72B-v0.1-6.0bpw-h6-exl2 (72B) - 28/48
Rhea-72b-v0.5-Q4_K_M (72B) - 30/48
MultiVerse_70B.Q4_K_M (70B) - 24/48
Ein-72B-v0.1-full.Q4_K_M (72B) - 20/48
xwin-lm-70b-v0.1.Q4_K_M (70B) - 23/48
ISTA-DASLab_Meta-Llama-3-8B-Instruct-AQLM-2Bit-1x16 (8B) - 8/48
zephyr-orpo-141b-A35b-v0.1.Q4_K_M (141B) - 19/48

2024/04/23

microsoft_Phi-3-mini-128k-instruct (3.8B) - 19/48
microsoft_Phi-3-mini-4k-instruct (3.8B) - 23/48
meraGPT_mera-mix-4x7B (4x7B) - 22/48
Orenguteng_Lexi-Llama-3-8B-Uncensored (8B) - 18/48

About

This test consists of 48 manually written multiple-choice questions. It evaluates a combination of academic knowledge and logical reasoning.

Compared to MMLU, it has the advantage of not being in any training dataset, and the disadvantage of being much smaller. Compared to lmsys chatbot arena, it is harsher on small models like Starling-LM-7B-beta that write nicely formatted replies but don't have much knowledge.

The correct Jinja2 instruction template is used for each model, as autodetected by text-generation-webui from the model's metadata. For base models without a template, Alpaca is used. The questions are evaluated using the /v1/internal/logits endpoint in the project's API.

The questions are private.

Limitations

This benchmark does not evaluate code generation, non-English languages, role-playing, RAG, and long context understanding. The performance in those areas may have a weak or nonexistent correlation with what is being measured.