Strict Limit: Max 250M Parameters

Micro-SLM Multiplication Leaderboard

This benchmark is strictly dedicated to Small Language Models (≤ 250M parameters). Rather than relying on brute-force scaling laws, we evaluate pure architectural efficiency, tokenization strategies, and reasoning chains on 10,500 multiplication tasks.

Parameter Efficiency vs Performance

X-Axis: Model Parameters (Log Scale, 100K - 250M) | Y-Axis: Selected Evaluation Metric

Y-Axis Metric:

Qualified Models (≤ 250M): 1

#	Model	Params (≤ 250M) ↕	CoT Method	Rating Score ↕	Exact Match ↕	Perplexity ↕

How to Evaluate & Submit Your Model

Follow these steps to benchmark and list your model on this leaderboard.

1 Run evaluate.py & Follow the Interactive TUI Wizard

Simply execute evaluate.py located in the same directory. An interactive terminal (TUI) wizard will launch to guide you through the measurement setup step-by-step.

// Run script to start interactive wizard
python evaluate.py

Interactive TUI

2 Submit Result via Pull Request / Commit

Append your generated score output into the modelData array in index.html.

// Example entry to append in index.html (modelData array):
{
  "name": "YourOrg/Your-Multiplication-SLM-10M",
  "params_str": "10.2M",
  "params_num": 10200000, // Must be ≤ 250,000,000
  "cot": "cot_partial", // "cot_partial" | "cot_cot" | "cot_direct"
  "score": 1720.5,
  "exact": 92.4,
  "token_acc": 98.9,
  "ppl": 1.085,
  "highlight": false
}