Benefits of Using RiskLens to Prioritize Bug Remediation

Here’s a scenario in any given software development shop that probably seems all too familiar: You open up your production bug log in the morning and see what seems like an impossibly long list of bugs from the previous day. These bugs are likely assigned to different developers to fix, probably based on some criteria like bug difficulty/obscurity, code-owner, or a qualitative estimate (high, medium, low).

There are a number of concerns with this approach, but these concerns can mostly be summarized as follows:

This method is ad hoc and doesn’t effectively prioritize bugs based on what matters most . The reality is not all bugs will be or can be fixed, simply because of resource constraints. This is not a problem so long as the right bugs are being fixed - namely the ones that have the greatest potential impact on customer satisfaction, the bottom line, and/or software security.

The question then is, short of a crystal ball, how can you effectively prioritize the bugs that have the greatest impact on customer satisfaction, the bottom line, and/or software security?

The RiskLens platform, purpose-built for the Factor Analysis of Information Risk (FAIR™) model, the only standard quantitative model for cybersecurity and technology risk, is an ideal solution for prioritizing bug remediation in a way that matters most to a business and its customers.

The benefits of using RiskLens to prioritize both tackling overgrown bug logs and daily management of bugs are:

• Ability to track and manage entire bug remediation lifecycle from identification to remediation
• Use of the RiskLens Rapid Risk Assessment capability to quickly assess the risk associated with each bug (Note: We define “risk” as the probable frequency and probable magnitude of future loss events, i.e., how often a bad thing will happen as a result of the bug and how much it will cost us every time it does!)
• Access to advanced quantitative risk analytics, best-practice risk assessment and reporting workflows and industry specific loss data

Quantifying the financial risk associated with your bug remediation backlog using RiskLens and FAIR will enable the organization to prioritize high risk bugs and thus maintain more secure applications, reduce inefficiency by focusing on what matters most, and ultimately have a more mature Software Development Lifecycle (SDLC).

Below we will look at a simple 3-step process for bug prioritization with RiskLens.

Important Terms in Relation to FAIR Model and Bug Remediation

Before walking through the steps, it’s worth pointing out a couple of important terms in relation to bug prioritization. If you are not yet familiar with FAIR, have no fear. Check out these resources to learn the basics before applying them to bug remediation.

Threat event frequency (TEF). In relation to web applications and software development, TEF is different depending on how the bug effects the app:

• The cases where a bug makes the application’s security weaker, TEF is best modeled as the frequency that external actors target the web application (looking for bugs or weak controls)
• In the case where a bug could directly lead to accidental disclosure or cause service disruption, TEF is best modeled as the frequency that insiders accidentally release bugs into production

Vulnerability. In web applications and software development, Vulnerability is the percentage of time that a threat event leads to a loss event. For example, if an external actor performs a web app attack, what percentage of the time will they be successful? Or, if a potential service interrupting bug (availability) is released into production, what percentage of the time will it cause application disruption?

Confidentiality. These types of scenarios are related to bugs that lead to accidental disclosure or weaken security in a way that an external actor may be able to take advantage of for the purpose of breaching sensitive information (e.g PII).

Availability. In applications, availability concerns are simply service disruption. If a bug breaks the application in a way that renders it unusable (either partially or completely), then that is an availability scenario.

3 Steps to Bug Prioritization with RiskLens

Scope Out Different Types of Bugs

When using RiskLens to prioritize bug remediation, it is far more efficient to organize bugs by their type and, where possible, perform an analysis on the risk of different types of bugs rather than each bug.

While some bugs may be truly unique and require an individual analysis, most bugs should be able to be grouped together based on similar type and expected risk. There are many resources online for types of bugs - for example here is the Wikipedia page on software bug types.

Regardless of how detailed you are in your organization of bugs, you should always scope them out to define, for each bug:

• Who the threat actor is relevant to the bug, usually either an external actor or non-malicious insider (who accidentally pushes a bug to production)
• What the effect or effects of the bug could be (confidentiality, availability, integrity)
• Which asset is relevant to the bug (i.e., database information, the web app itself, etc.)

The RiskLens platform will guide you through the scoping of each risk analysis you choose to quantify, to ensure you have defined the asset, threat, and effect before moving on. Additionally, you’ll have the option to save those context-specific component definitions defined in the “important terms” section above directly to your analysis to ensure alignment.

Rapidly Assess Different Types of Bugs on the RiskLens Platform

Now that you’ve scoped the bugs you are looking to quantify in the RiskLens platform, you can begin the quantification and prioritization aspect. We all know that time is a precious and limited resource, so before spending hours or days diving deep into a specific bug, you can use the Rapid Risk Assessment capability of the RiskLens platform to prioritize your log and determine what requires additional attention and what can take a back-burner for the time being. This process takes approximately 15 - 30 minutes per analysis.

Some bugs only pose minor usability problems (i.e. minor availability issue), whereas other bugs can break the application entirely or create significant security holes that attackers can exploit. Within a given type of bug, there may even be a wide range of potential impact - to account for this variation, we will use calibrated estimation to estimate ranges that are accurate while accounting for our uncertainty. To rapidly assess risk in software development you need to broadly identify frequency and impact for each type of bug, such as:

• LEF: How often, in a given year, would/has this bug resulted in loss?
  • For availability scenarios, how often would/has this bug actually led to service disruption?
  • For confidentiality scenarios, how often would/has this bug led to accidental unauthorized disclosure or been exploited by an external actor?

With Rapid Risk Assessments, estimating this range will be easier than ever. By answering intuitive questions about known historic events, and expected environmental changes, the RiskLens platform will guide the analyst to derive an estimation.

• Primary Magnitude: How much would it cost to internally respond to and remediate the loss if it occurred?

It’s likely your organization has an incident response plan in place that dictates the incident response efforts dependent on type and/or severity of event. Using the Data Helpers within the RiskLens platform, you can gather this information once and save it for future use; gaining efficiency and consistency in future analyses.

• Secondary Loss: What percentage of the time would this loss result in secondary stakeholder fallout?
  • How much would secondary stakeholder fallout impact us financially, in terms of:
    • Response, such as responding to customer calls/complaints or litigation
    • Reputation damage, such as customer churn
    • Fines or legal judgements

If you have no idea where to start in gathering or utilizing this data - no need to fret. During your onboarding to the RiskLens platform, you’ll get 1:1 time with the experienced RiskLens Services team, during which, in addition to platform configuration, customized data libraries leveraging industry specific loss data as well as data from your security ecosystem will be built. That means that at this stage in the process, selecting the appropriate estimate will be easier than ever.

After selecting the appropriate estimates, hit the run button and go hit up the Keurig for your afternoon caffeine boost. While your java is brewing, the RiskLens platform will be running 50,000 Monte Carlo simulations based on the inputs provided. In each simulation, the platform is determining based on your inputs whether or not the specific event is expected to occur, and if so, how often and how much it will cost each time. The result? Summarized reporting at both an annualized and per event level. These results will provide an understanding of exposure from each bug, which can then be used to prioritize your remediation efforts.

NOTE: with your highest risk bugs it can be beneficial from a wider business perspective to do an in-depth analysis and understand with greater precision the frequency and magnitude of loss. However, rapidly assessing the risk of different bugs accomplishes the goal of software development bug remediation. If you want to do a more detailed analysis following the Rapid Risk Assessment, you can use the “Guided” functionality of the platform to estimate at TEF and VULN rather than LEF to gain gain some additional precision. Always remember to keep in mind the diminishing value of returns, though - is the added precision worth the additional time spent in gaining it? Are the results as they are good enough for the purposes of the decision being made?

How to Prioritize Bugs with Quantitative Risk Analysis

Understanding the exposure faced from a given bug is the foundation of prioritizing bug remediation. There is no right way to prioritize - the important point is that you use your quantitative risk analysis to make an informed decision about how to prioritize bugs in a way that is consistent with the goals of your organization.

For reference, often scenarios are prioritized based on:

• Largest impact
• Highest frequency
• Greatest cost-benefit - e.g., bugs that are easiest to fix and reduce the most amount of risk
  • This requires having a general understanding of an average length of time for a type of bug to be remediated

There can even be greater nuance to prioritization than this. For example, depending on your organization's security stance you might prioritize high frequency availability scenarios over confidentiality scenarios that have significant financial impact but rarely occur.

Long Term Value of FAIR Analysis in Software Development

Using RiskLens for bug remediation has long term value for software development efficiency and organizational risk reduction. Some good long term use cases to build around such an analysis are:

• Implement RiskLens quantitative risk assessment into regular process of bug remediation
• Perform analysis on unique bugs that don’t fit in with existing analysis
• Perform risk analysis during design phase of proposed new software features/improvement
• Plan AGILE production pipeline using RiskLens quantitative risk analysis