SWE-bench Multilingual
Rusta + Devscriptor
300 real GitHub issues from 41 repositories in 9 languages. Rusta, with Devscriptor’s code intelligence, resolved 75.3% to 81.3% of them with models from DeepSeek, Z.AI and Moonshot AI — one attempt per task, scored by the official SWE-bench harness.
Three models, one stack
Every model ran every task once, at reasoning effort max, with the same prompt, the same tools and the same limits. A task counts as resolved when the harness finds that the tests written for its fix now pass, and that the tests that passed before still do.
DeepSeek V4.1-Flash
deepseek-flash- 95% interval
- 76.5% – 85.3%
- Time on a task
- 4.7 min median, 5.7 mean
- Model requests
- 44 median
- Tokens in
- 967 M, 97.8% from cache
- Tokens out
- 8.9 M
- Cost
- $13.98 at DeepSeek’s pay-per-token prices
- Used Devscriptor
- in 240 of 300 runs
GLM-5.3
glm-5.3- 95% interval
- 73.7% – 82.9%
- Time on a task
- 9.9 min median, 13.4 mean
- Model requests
- 32 median
- Tokens in
- 559 M, 96.6% from cache
- Tokens out
- 7.0 M
- Cost
- GLM Coding Plan Max subscription
- Used Devscriptor
- in 276 of 300 runs
Kimi K2.8 Preview
kimi-for-coding- 95% interval
- 70.2% – 79.9%
- Time on a task
- 10.0 min median, 13.0 mean
- Model requests
- 32 median
- Tokens in
- 1294 M, 97.6% from cache
- Tokens out
- 15.7 M
- Cost
- Kimi Code subscription
- Used Devscriptor
- in 272 of 300 runs
The interval is where the true rate lies with 95% confidence, given 300 tasks. A run that changed no file counts as unresolved: one for GLM and one for Kimi, on the same task. Tokens are summed over all 300 tasks.
Where each model does well
Go is the hardest for all three. DeepSeek has the best result in six of the eight rows, two of them shared with GLM; GLM leads in PHP and shares C++ with Kimi. The best result in each row is highlighted.
| Language | Tasks | DeepSeek V4.1-Flash | GLM-5.3 | Kimi K2.8 Preview |
|---|---|---|---|---|
| C | 30 | 87%26/30 | 87%26/30 | 83%25/30 |
| C++ | 12 | 67%8/12 | 75%9/12 | 75%9/12 |
| Go | 42 | 64%27/42 | 64%27/42 | 60%25/42 |
| Java | 43 | 93%40/43 | 88%38/43 | 88%38/43 |
| JavaScript / TypeScript | 43 | 86%37/43 | 74%32/43 | 81%35/43 |
| PHP | 43 | 81%35/43 | 84%36/43 | 70%30/43 |
| Ruby | 44 | 75%33/44 | 70%31/44 | 68%30/44 |
| Rust | 43 | 88%38/43 | 86%37/43 | 79%34/43 |
JavaScript and TypeScript share a row: several of their repositories hold both. With 12 to 44 tasks a language, a difference of a few tasks is within chance.
Next to the official leaderboard
SWE-bench’s own leaderboard runs every model in the same minimal agent, mini-SWE-agent 2.0, so it compares models. This page measures one agent with one set of tools. The scale is the same; the setups are not.
Rusta + Devscriptor
9–11 November 2026 · one attempt per task
Why the two lists do not rank against each other
- Newer models. None of the three models here is on the leaderboard. The entries nearest to them are their predecessors: DeepSeek V3.2, GLM 5 and Kimi K2.5.
- A different agent. A full coding agent with code intelligence is expected to do better than a minimal one with only a shell. How much of the gap is Rusta, Devscriptor or the newer models, these runs cannot tell apart.
- A generous budget. Reasoning effort max, up to 250 model requests and 4 hours of work per task, in an environment prepared so that the project builds and its tests run offline.
- Not an official entry. SWE-bench accepts Multilingual submissions only from academic teams and research institutions with a publication. These results are published here instead, with every file needed to check them.
Official leaderboard
mini-SWE-agent 2.0 · February 2026 · one attempt per task
From swebench.com, Multilingual tab.
How the runs were made
Rusta ran inside each task’s own container, as a developer would work on the project, cut off from anything that could hold the answer.
The agent
Rusta 1.0.0, headless, with every tool allowed except the web tools, which were denied. Each run could make up to 250 model requests and work for 4 hours; no run reached either limit.
Devscriptor 1.0.4 was connected over MCP, with the task’s repository indexed before the run began. The prompt asked the model to use it.
The environment
Each run started from the task’s own SWE-bench image, at the commit the issue was reported against, with 2 CPUs and 8 GB of memory. Its only way out was an allowlisting proxy that reached the model provider’s API and nothing else.
Before a run, the task’s build and test steps ran once on the unfixed code, so the project built and its tests ran without the network. The tests written for the fix were never in the environment.
No way to the fix
The repository’s git history was cut at the task’s base commit: every later commit, branch and remote was removed and checked gone, so the upstream fix could not be read from history. The model had no web access, and its prompt gave the issue and general instructions: no hints, no test names.
The patch and its score
The patch is every change the agent made to the repository. Scoring used the official harness, swebench 5.0.2, in the task’s original image, and every patch was scored.
Files that the build step created and the agent then changed, such as a test runner’s cache, were left out of the patch, since the scoring image lacks them. This touched 3 PHP tasks for each model.
One attempt per task
Each model ran each task once, and every result counts. A restart of the runner interrupted 4 runs before they had made a patch. They started again from scratch; none had been scored, so no result was picked from two.
An earlier smoke test ran 8 of the tasks with a prompt that did not mention Devscriptor. Those tasks were run again with the final prompt, and only the new results count. The earlier runs are compared below, and are in the archive.
The warm-up’s leftovers
The build-and-test step left its reports, caches and logs in the environment, such as Maven’s surefire reports, PHPUnit’s result cache and, in the Go tasks, the tests’ output in Go’s build cache. They name the existing tests that the scoring runs, which hints at where the fix belongs.
Some runs named them in a tool call: DeepSeek 35 (26 resolved), GLM 13 (11) and Kimi 13 (9), over 47 tasks. Even if every one of those resolutions came from the hint, the rates would fall by at most 8.7, 3.7 and 3.0 percentage points.
The prompt every run got
The task’s issue goes in place of @PROBLEM_STATEMENT@.
The repository at /testbed is checked out at the commit an issue was reported against.
<issue>
@PROBLEM_STATEMENT@
</issue>
Change the non-test files under /testbed so that the issue is resolved. The tests
that check the fix are already written and will be added afterwards, so you do not
need to write or change tests. You may run the project's existing tests and your own
scripts. The environment has no internet access; the project's dependencies are
already installed.
Devscriptor's code-intelligence tools are connected and /testbed is already
indexed: load them with select_tools and use them (semantic search, definitions,
references) to find the relevant code before reading files.
Make the smallest change that resolves the issue.How the models used Devscriptor
Most runs called Devscriptor, about twice a run. Across the three models, semantic search and the function search were the tools called most.
| Model | Runs using it | Calls | Most-called tools |
|---|---|---|---|
| DeepSeek V4.1-Flash | 240 of 300 | 5511.8 a run | semantic_search 215 · search_context 117 · search_functions 92 |
| GLM-5.3 | 276 of 300 | 6292.1 a run | semantic_search 224 · search_functions 136 · search_context 100 |
| Kimi K2.8 Preview | 272 of 300 | 6632.2 a run | search_functions 244 · semantic_search 231 · add_context_note 91 |
Runs that called Devscriptor resolved 190 of 240, 217 of 276, 203 of 272; the rest resolved 54 of 60, 19 of 24, 23 of 28 (DeepSeek, GLM, Kimi). The two groups are not alike: the runs without a Devscriptor call needed fewer model requests, which suggests easier tasks. What Devscriptor adds is measured by running the same tasks without it.
Without the prompt’s sentence
The smoke test’s 8 tasks, one in each of eight languages, run by the three models with Devscriptor 1.0.4: 24 runs a row.
| Rusta | Prompt | Runs using it | Calls | DeepSeek · GLM · Kimi |
|---|---|---|---|---|
| 1.0.0 | Without the sentence | 5 of 24 | 15 | 3 · 2 · 0 |
| 1.0.0 | With the sentence (the full run) | 23 of 24 | 50 | 7 · 8 · 8 |
| 1.0.1 | Without the sentence | 19 of 24 | 49 | 6 · 6 · 7 |
Rusta 1.0.0 loads every Devscriptor tool on demand, and without the sentence the models mostly searched with grep and read files instead, which is why the full run’s prompt has it. Rusta 1.0.1 takes its place: Devscriptor’s twelve most-used tools are in the tool table from the first request, and a reminder to start with them comes with the first prompt. The models then made about as many Devscriptor calls as with the sentence, in somewhat fewer runs. Eight tasks are too few to compare resolution rates; the archive holds these runs in full.
What these results show, and what they do not
A benchmark score is only as useful as the conditions behind it. These are the conditions.
They show
- Rusta with Devscriptor resolves three quarters or more of SWE-bench Multilingual in a single attempt, with models from three vendors.
- The same agent and configuration resolve more than half of the tasks in every language, for every model.
- DeepSeek V4.1-Flash scored highest, for $13.98 for all 300 tasks: under 5 cents a task, with 97.8% of its input read from the prompt cache.
They do not show
- How much Devscriptor contributes. That takes the same tasks run without it, which these runs do not include.
- A ranking against other agents, which would need the same models, budget and environment on both sides.
- Results free of what the models learned in training. The issues and their fixes are public; at least one trajectory shows a model trying to recall the upstream fix. With the history cut and the web closed, it could not look it up.
- Rates finer than their intervals: each is uncertain by about 4 to 5 points.
Every file, to check for yourself
The predictions, the harness’s reports and the full trajectories of all 900 runs, in one archive.
- For each model: the predictions as the harness reads them, each task’s outcome, and the harness’s files for each task it scored, including the full test output.
- Each run’s trajectory: every message in order, with the model’s reasoning, its tool calls and their results; and each run’s record of steps, tools, tokens and time.
- The prompt and the Rusta configuration every run used.
- The smoke test’s runs without the prompt’s sentence, with Rusta 1.0.0 and 1.0.1, laid out the same way, with the prompt each task got.
- In a few runs the agent printed its environment, which held the provider’s API key. Each such key is replaced with [REDACTED]; nothing else is redacted.
Score the predictions again
base=https://github.com/vladimirpesic/devscriptor-web/releases/download/swe-bench-multilingual
curl -LO $base/rusta-devscriptor-swe-bench-multilingual.tar.gz
curl -LO $base/rusta-devscriptor-swe-bench-multilingual.tar.gz.sha256
sha256sum -c rusta-devscriptor-swe-bench-multilingual.tar.gz.sha256
tar -xzf rusta-devscriptor-swe-bench-multilingual.tar.gz
cd rusta-devscriptor-swe-bench-multilingual
# Docker must be running: each task is scored in its own image.
pip install swebench==5.0.2
python -m swebench.harness.run_evaluation \
-d SWE-bench/SWE-bench_Multilingual \
-p deepseek/all_preds.jsonl \
--max_workers 4 -id rusta-deepseekTasks from SWE-bench Multilingual (SWE-bench/SWE-bench_Multilingual). Run 9–11 November 2026.