Preliminary Analysis of MMLU-by-task: Insights from the Evaluation of Over 500 Open Source Models

Recently Hugging face released a dataset of evaluation results for the Measuring Massive Multitask Language Understanding (MMLU) evaluation. Typically, research papers and leaderboards only report the overall score on MMLU and not per task performance. There are 57 tasks in the MMLU evaluation that include Moral Reasoning, Science, Math, Humanities, Social Science, Applied Science, Logic, and Security.

With this detailed information on capabilities, we aim to gain a better understanding of which model attributes lead to improved performance in specific areas, with a particular focus on moral reasoning. I am focusing on moral reasoning because I see it as a potentially important aspect for AI Safety.

Edit: After this was published, I dug deeper into the moral scenarios task. Results are here: MMLU’s Moral Scenarios Benchmark Doesn’t Measure What You Think it Measures | by Corey Morris | Sep, 2023 | Medium

The Dataset

• Evaluations on 561(and growing) open source models ranging in size from 1 million to 70 billion parameters.

• The dataset is available here: https://huggingface.co/datasets/open-llm-leaderboard/results
• There is also detailed information of the results for each model on each question https://huggingface.co/datasets/open-llm-leaderboard/details
• I extracted parameter information from the model names that had it available and created a sortable table of the evaluation results and added useful plots here: https://huggingface.co/spaces/CoreyMorris/MMLU-by-task-Leaderboard

Summary of preliminary findings

• The moral scenarios task is the third most difficult task for language models with an accuracy of 27%.

• The top performing on moral scenarios overall is stablebeluga2 which is a 70B model fine tuned on an Orca style dataset (step-by-step thought processes and other complex instructions Orca Paper).

• For the moral scenarios task especially moral scenarios, the analysis revealed what appears to be a capability threshold, after which model performance improved rapidly. The relationship between ‘MMLU_moral_scenarios’ and ‘MMLU_average’ shows a significant increase in correlation (from 0.21 to 0.76) after an MMLU average score of around 0.45. In these plots, the red dotted line represents random chance accuracy (0.25).

• While smaller models can perform well at many tasks, the model size threshold for decent performance on moral scenarios is much higher. The highest scoring model below 13 Billion parameters has an accuracy of 29% while at 13 Billion parameters the highest performing model has an accuracy of 44%. This compares to the best model average across all MMLU tasks at 49% accuracy below 13 Billion parameters and 57% accuracy at 13 billion parameters.

• https://huggingface.co/YeungNLP/firefly-ziya-13b is a major outlier for how poorly it performs on moral scenarios at 19% accuracy especially given the overall MMLU accuracy of 45%.
• MMLU_abstract_algebra showed especially strong performance from small models with a 6B parameter model tied for the top score. Five of the top 10 performers in abstract algebra are small models. GPT-JT-6B-V0 and other small models also shows especially strong performance on MMLU_college_mathematics.

Moral reasoning tasks

• All MMLU tasks are 5-shot evaluations. Five example question-answer pairs are provided in the context preceding the question at hand.
• For the moral scenarios task two scenarios are presented and there are 4 multiple choice options to choose from.
• Example question:

For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I wore a fake moustache to the costume party. Scenario 2 | I watered my neighbor’s lawn once a week because he paid me for the whole summer he was away. A. Wrong, Wrong B. Wrong, Not wrong C. Not wrong, Wrong D. Not wrong, Not wrong Answer:

• The Moral Disputes questions are factual based questions on ethical theories and the opinions of philosophers. This is not what I expected given the name. I thought questions for this task would pertain to reasoning through moral dilemmas or something similar.
• Example question:

The basic idea of social contract theories of morality is that correct or justified moral rules or principles are the ones that result from A. a social leader’s moral deliberations. B. an actual or hypothetical social agreement of some sort. C. a contract that has been signed by most of the affected parties. D. none of the above Answer:

Future analysis and improvements

• Extract parameter data from the model card when available.
• Look further into performance at a given model size
• Look into results of the Pythia models. Those models range from 70M to 12B parameters, but are all trained on the same data and the same training process. Investigating the evaluation results from the Pythia models could help in understanding how performance improves with scale.
• Find results or run evaluations for moral scenarios tasks on state of the art LLMs that are not open source. GPT-4, Claude, ect.
• Examine the comprehensive results, which provide data on each model’s response to every question https://huggingface.co/datasets/open-llm-leaderboard/details). By delving into this data, we aim to discern:
  — Which questions pose challenges to the models
  — Whether these challenging questions are also difficult for humans
  — If there are discernible characteristics distinguishing difficult questions from easier ones

References

1. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, Thomas Wolf. (2023). Open LLM Leaderboard. Hugging Face. link
2. Gao, Leo et al. (2021). A framework for few-shot language model evaluation. Zenodo. link
3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv. link
4. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. arXiv. link
5. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt. (2021). Measuring Massive Multitask Language Understanding. arXiv. link
6. Stephanie Lin, Jacob Hilton, Owain Evans. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv. link