SWE-bench Lite

Overview

SWE-bench Lite provides a smaller, carefully selected subset of 300 tasks from the full benchmark, designed to:

Reduce evaluation costs while maintaining benchmark quality
Enable faster iteration cycles for model development
Provide a more accessible entry point for research groups

The 300 tasks were selected to preserve the distribution and difficulty spectrum of the original benchmark while focusing on more self-contained, functional bug fixes.

While the full SWE-bench test split comprises 2,294 issue-commit pairs across 12 Python repositories, SWE-bench Lite covers 11 of the original 12 repositories with a similar diversity and distribution. We also provide 23 development instances that can be useful for active development on the SWE-bench task.

We recommend future systems evaluating on SWE-bench to report numbers on SWE-bench Lite in lieu of the full SWE-bench set when compute efficiency is a concern.

Selection Criteria

SWE-bench Lite instances were selected using the following criteria:

Removed instances with images, external hyperlinks, references to specific commit SHAs and references to other pull requests or issues
Removed instances with fewer than 40 words in the problem statement
Removed instances that edit more than 1 file
Removed instances where the gold patch has more than 3 edit hunks
Removed instances that create or remove files
Removed instances that contain tests with error message checks
Finally, sampled 300 test instances and 23 development instances from the remaining candidates

The source code for how SWE-bench Lite was created is available in SWE-bench/swebench/collect/make_lite.

Repository Distribution

SWE-bench Lite distribution across repositories. Compare to the full SWE-bench in Figure 3 of the SWE-bench paper.

Baseline Performance

SWE-bench Lite performance for our baselines. Compare to the full SWE-bench baseline performance in Table 5 of the SWE-bench paper.

Resources

SWE-bench Lite datasets:

🤗 SWE-bench Lite 🤗 "Oracle" Retrieval Lite 🤗 BM25 Retrieval 13K Lite 🤗 BM25 Retrieval 27K Lite

Citation

If you use SWE-bench in your research, please cite our paper:

@inproceedings{
    jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}

                    Jimenez, C. E., Wettig, A., Yao, S., Yang, J., Pei, K., Jain, S., Press, O., & Narasimhan, K. (2024).
                    SWE-bench: Can Language Models Resolve Real-world Github Issues? arXiv preprint arXiv:2310.06770.

Jimenez, Carlos E., et al. "SWE-bench: Can Language Models Resolve Real-world Github Issues?" arXiv preprint arXiv:2310.06770 (2023).