Automated Software Engineering

Building Larger Applications

---

At the moment of the latest update to these materials, in fall 2024, building larger applications with large language models is still challenging and the process cannot be fully, and often even partially automated. The key issue is that keeping track of all the relevant pieces of an application (or a codebase) is not easy, both for the developers and for large language models. New framework version releases, changing APIs, and changing requirements are a common theme in the context of building applications.

When working on a task, software developers rely on the context of the task they are currently working on when seeking to comprehend the code and to create functionality. They do not — and cannot — have a thorough understanding of very large codebases in their mind at all times.

For a review of research into programmers’ knowledge and understanding of programs, see “Synthesizing research on programmers’ mental models of programs, tasks and concepts — A systematic literature review”.

Similarly, when working on a codebase with large language models, the large language model is typically limited to the nearby context in the prompt, generating source code or completing code. If the code that the large language model is tasked to create would already exist elsewhere in the codebase, outside the context of the large language model, the large language model is likely to generate new code even if the code would already exist.

Loading Exercise...

GitClear highlighted this phenomenon in their report “Coding on Copilot: 2023 Data Suggests Downward Pressure on Code Quality”. Their report highlights that the proportion of copy-pasted code and added code has recently increased when contrasted to other types of code changes (deletion, moving, and updating).

The key to building larger applications with large language models is learning to decompose the application into sufficiently small pieces, and prompting the large language model for solutions to those pieces with the right context.

Tools such as GitHub Copilot and OpenAI Codex can be used to complete code and to generate code for individual functions or classes, but they are not yet capable of generating entire (large) applications.

Thus, to get started with an application, we may prompt a large language model for an overall structure of the application, and then refine the outputs ourselves. Although some of the solutions could be suboptimal, the large language model can provide a good starting point for the development of the application. This is where large language models shine — they can often be used to provide a fast and rough draft or a proof of concept solution, which can then be refined by a human developer.

Loading Exercise...

Recent advances in large language models and tooling have made it easier to work with large language models in the context of building applications. As an example, GitHub Copilot is increasingly receiving features that make it easier to work with larger codebases. Similarly, e.g. Cursor already provides functionality for making changes to multiple files at the same time.