Automated Software Engineering

Automated Program Repair

---

Automated program repair

In automated program repair (or automatic bug fixing), software systems seek to automatically fix errors or vulnerabilities in code. Automatic program repair systems take in the source code of a program and some evidence that the program has a bug (typically, a failing automated test). The system then produces a modified version of the program that hopefully fixes the bug, verified by running the test suite again.

The overall goal of automated program repair is to fix bugs, without breaking other parts of the program.

Classic automated program repair systems are rule-based and rely on a set of rules to fix bugs. More recent systems use machine learning techniques to learn how to fix bugs from a dataset of buggy and fixed programs. The state-of-the-art systems typically rely on large language models.

The core idea that automated program repair systems use, regardless of the underlying technology, is to generate a pool of one more more candidate solutions that would fix the bug, and then run the test suite on the candidate solutions to identify solutions that fix the bug and do not produce additional bugs. From the solutions that fix the bug, the selection of the best candidate is based on some criteria, such as the closeness of the candidate solution to the original buggy program.

For additional information on distance measures for program repair, see e.g. the article “Evaluating Distance Measures for Program Repair”.

Loading Exercise...

The task in the SWE-bench could also be viewed as an automated program repair task, depending on the category of the issue that should be fixed. In the SWE-bench, the input is a GitHub issue and the associated codebase, and the task is to produce a solution that addresses the GitHub issue. The solution is created as a patch that is applied to the codebase. The solution is then verified against a set of automated tests.

Patch example

A patch is a textual format that describes how a file (or multiple files) should be changed. The authors of the SWE-bench article outline an example prompt template that can be used to prompt a model to generate a patch for a given issue. The template is as follows:

You will be provided with a partial code base and an issue
statement explaining a problem to resolve.

<issue>
{ISSUE TEXT}
</issue>

<code>
[start of README.md]
{README.md text}
[end of README.md]
[start of file_1.py]
{file_1.py text}
[end of file_1.py]
...
</code>

Here is an example of a patch file. It consists of changes to
the code base. It specifies the file names, the line numbers
of each change, and the removed and added lines. A single
patch file can contain changes to multiple files.

<patch>
(patch file example)
</patch>

I need you to solve the provided issue by generating a single
patch file that I can apply directly to this repository using
git apply. Please respond with a single patch file in the
format shown above.

Respond below:

Applying the template, we can prompt a model to generate a patch for a given issue. In the following example, we provide the model an input that highlights an issue in the output of a program, the concrete code of the program, and an example patch file. The model is then asked to generate a patch that fixes the issue.

Concretely, a patch would be applied using a command such as git apply patchfile.patch, where patchfile.patch is the file containing the patch. The patch would then modify the codebase to fix the issue.

Concretely, if hello.py is as follows:

print("Hello wordl!")

And my.patch is as follows:

--- a/hello.py
+++ b/hello.py
@@ -1 +1 @@
-print("Hello wordl!")
+print("Hello world!")

Then applying the patch using a git command git apply my.patch would modify hello.py to be:

print("Hello world!")

Note that both the hello.py and the my.patch files are in the same directory when applying the patch, and both of them had an empty line at the end.

Loading Exercise...