SWE-smith Multilingual: Scaling Multilingual Data for Software Engineering Agents

This is the first step of the SWE-smith Multilingual effort - expanding SWE-Smith to support non-Python programming languages, starting with JavaScript.

Key Results:

What is SWE-Smith?

SWE-Smith is a tool for synthesizing realistic bugs in software repositories. By applying procedural modifications to working code (like flipping operators or swapping function arguments), it creates bugs that break tests - providing training data for code repair models and evaluation benchmarks.

Until now, SWE-Smith only supported Python. This expansion brings the same capabilities to JavaScript.

Results by Repository

The most productive repository was josdejong/mathjs, which features lots of arithmetic operations and comprehensive unit tests - an ideal combination for procedural bug generation. It produced 845 validated patches with a 62.8% pass rate.

JavaScript repository distribution showing validated patches across repositories

Modifier Effectiveness

We implemented 14 procedural modifiers for JavaScript. The func_pm_op_flip modifier emerged as the most productive, generating 1,117 validated patches - nearly double the next best modifier. This makes sense: flipping binary operators (e.g., === to !==, < to >=, && to ||) applies to a wide range of expressions found across all repositories.

JavaScript modifier distribution showing validated patches by modifier type

Scaling with Modal

Previously, running bug generation at scale meant days of local compute. We now run the entire pipeline on Modal, reducing this to hours.

The pipeline:

  1. Generate patches by applying procedural modifiers to repository code
  2. Validate patches by running each repository's test suite in isolated containers
  3. Store results in the HuggingFace Dataset

What's Next?

Here's what to expect in the coming months:

  1. Get automatic repository installation + procedural modifications working for new languages. On our roadmap: TypeScript, Java, Rust, Golang, C++
  2. Run LM based bug generation methods for all added repositories.
  3. Run experiments to better understand the impact of programming languages and repositories on model training dynamics and downstream performance on coding tasks.

Links

Team

Brought to you by: Priyank Shethia, Arpandeep Khatua, Rishi Desai, Zhehao Li, Andrew Crump, John Yang