Software Security and Large Language Models

Threats and Risks

---

There exists a wide range of computer security threats. These threats can be broadly considered from two perspectives: intentional and unintentional. Intentional threats are caused by individuals and groups who try to cause harm, while unintentional threats are often byproducts of actions (or inactions) without the intent of causing harm. At the core of both is typically the possibility of causing harm, which should be mitigated.

Loading Exercise...

A key approach of identifying the possibilities of causing harm is threat modeling, which is an often neglected part of the software development process. Threat modeling is used to identify potential threats to a system and to determine the risks associated with those threats.

Threat modeling is a structured approach to identifying and evaluating the security of an application. It is a process that can help identify potential threats and vulnerabilities in a system.

As a starting point for threat modeling, the four-question framework is a suitable choice. The four questions are as follows:

1. What are we working on?
2. What can go wrong?
3. What are we going to do about it?
4. Did we do a good job?

In the context of building software, the first question forms the basis for the modeling effort. The second question is about identifying potential threats through, for example, brainstorming. The third question focuses on making informed decisions on what to do with each identified threat. Finally, the fourth question focuses on evaluating whether the threat modeling effort was sufficient.

For a brief overview of the four-question framework, listen to The Four Question Framework with Adam Shostack podcast.

Large language models can be used to identify threats in a range of products of the software development life cycle. For example, large language models can be used to analyze specification to identify potential threats that may arise during the development process. In a similar way, large language models can be used to identify threats related to an application idea.

Loading Exercise...

While brainstorming is a useful technique, there are also more structured approaches avaiable for identifying threats. One of these is the STRIDE threat model. STRIDE is a mnemonic of six categories of security threats, which are as follows:

• Spoofing — an attacker pretends to be someone else.
• Tampering — an attacker modifies something (e.g. data, configurations, …).
• Repudiation — an attacker denies an action.
• Information disclosure — an attacker gains access to information that should be protected.
• Denial of service — an attacker prevents legitimate users from accessing a service.
• Elevation of privilege — an attacker gains more access than they should have.

Using STRIDE, developers can seek to identify threats in each of the category, which may yield a more complete list of potential threats.

Loading Exercise...

When considering the severity of the identified threats, they can be assessed using the DREAD risk assessment model. DREAD is a mnemonic of five categories that can be used to assess the severity of a threat, which are as follows:

• Damage — how much damage could the threat cause?
• Reproducibility — how easy is it to reproduce the threat?
• Exploitability — how easy is it to exploit the threat?
• Affected users — how many users are affected by the threat?
• Discoverability — how easy is it to discover the threat?

While DREAD was proposed as a way to rank threats, it has been criticized for being somewhat subjective and not providing a clear way to compare threats. Regardless, it can still be a useful tool for assessing and discussing the severity of threats, also in the context of large language models.

See also David LeBlanc’s blog post entry DREADful that discusses criticisms related to STRIDE and DREAD.

Loading Exercise...