Here’s a scenario in any given software development shop that probably seems all too familiar: You open up your production bug log in the morning and see what seems like an impossibly long list of bugs from the previous day. These bugs are likely assigned to different developers to fix, probably based on some criteria like bug difficulty/obscurity, code-owner, or a qualitative estimate (high, medium, low).
There are a number of concerns with this approach, but these concerns can mostly be summarized as follows:
This method is ad hoc and doesn’t effectively prioritize bugs based on what matters most. The reality is not all bugs will be or can be fixed, simply because of resource constraints. This is not a problem so long as the right bugs are being fixed – namely the ones that have the greatest potential impact on customer satisfaction, the bottom line, and/or software security.
The question then is, short of a crystal ball, how can you effectively prioritize the bugs that have the greatest impact on customer satisfaction, the bottom line, and/or software security?
Tyler Britton and Taylor Maze are Risk Consultants on the RiskLens Services team.
The RiskLens platform, purpose-built for the Factor Analysis of Information Risk (FAIR™) model, the only standard quantitative model for cybersecurity and technology risk, is an ideal solution for prioritizing bug remediation in a way that matters most to a business and its customers.
The benefits of using RiskLens to prioritize both tackling overgrown bug logs and daily management of bugs are:
Quantifying the financial risk associated with your bug remediation backlog using RiskLens and FAIR will enable the organization to prioritize high risk bugs and thus maintain more secure applications, reduce inefficiency by focusing on what matters most, and ultimately have a more mature Software Development Lifecycle (SDLC).
Below we will look at a simple 3-step process for bug prioritization with RiskLens.
Important Terms in Relation to FAIR Model and Bug Remediation
Before walking through the steps, it’s worth pointing out a couple of important terms in relation to bug prioritization. If you are not yet familiar with FAIR, have no fear. Check out these resources to learn the basics before applying them to bug remediation.
Threat event frequency (TEF). In relation to web applications and software development, TEF is different depending on how the bug effects the app:
Vulnerability. In web applications and software development, Vulnerability is the percentage of time that a threat event leads to a loss event. For example, if an external actor performs a web app attack, what percentage of the time will they be successful? Or, if a potential service interrupting bug (availability) is released into production, what percentage of the time will it cause application disruption?
Confidentiality. These types of scenarios are related to bugs that lead to accidental disclosure or weaken security in a way that an external actor may be able to take advantage of for the purpose of breaching sensitive information (e.g PII).
Availability. In applications, availability concerns are simply service disruption. If a bug breaks the application in a way that renders it unusable (either partially or completely), then that is an availability scenario.
3 Steps to Bug Prioritization with RiskLens
Scope Out Different Types of Bugs
When using RiskLens to prioritize bug remediation, it is far more efficient to organize bugs by their type and, where possible, perform an analysis on the risk of different types of bugs rather than each bug.
While some bugs may be truly unique and require an individual analysis, most bugs should be able to be grouped together based on similar type and expected risk. There are many resources online for types of bugs – for example here is the Wikipedia page on software bug types.
Regardless of how detailed you are in your organization of bugs, you should always scope them out to define, for each bug:
The RiskLens platform will guide you through the scoping of each risk analysis you choose to quantify, to ensure you have defined the asset, threat, and effect before moving on. Additionally, you’ll have the option to save those context-specific component definitions defined in the “important terms” section above directly to your analysis to ensure alignment.
Rapidly Assess Different Types of Bugs on the RiskLens Platform
Now that you’ve scoped the bugs you are looking to quantify in the RiskLens platform, you can begin the quantification and prioritization aspect. We all know that time is a precious and limited resource, so before spending hours or days diving deep into a specific bug, you can use the Rapid Risk Assessment capability of the RiskLens platform to prioritize your log and determine what requires additional attention and what can take a back-burner for the time being. This process takes approximately 15 – 30 minutes per analysis.
Some bugs only pose minor usability problems (i.e. minor availability issue), whereas other bugs can break the application entirely or create significant security holes that attackers can exploit. Within a given type of bug, there may even be a wide range of potential impact – to account for this variation, we will use calibrated estimation to estimate ranges that are accurate while accounting for our uncertainty. To rapidly assess risk in software development you need to broadly identify frequency and impact for each type of bug, such as:
With Rapid Risk Assessments, estimating this range will be easier than ever. By answering intuitive questions about known historic events, and expected environmental changes, the RiskLens platform will guide the analyst to derive an estimation.
It’s likely your organization has an incident response plan in place that dictates the incident response efforts dependent on type and/or severity of event. Using the Data Helpers within the RiskLens platform, you can gather this information once and save it for future use; gaining efficiency and consistency in future analyses.
If you have no idea where to start in gathering or utilizing this data – no need to fret. During your onboarding to the RiskLens platform, you’ll get 1:1 time with the experienced RiskLens Services team, during which, in addition to platform configuration, customized data libraries leveraging industry specific loss data as well as data from your security ecosystem will be built. That means that at this stage in the process, selecting the appropriate estimate will be easier than ever.
After selecting the appropriate estimates, hit the run button and go hit up the Keurig for your afternoon caffeine boost. While your java is brewing, the RiskLens platform will be running 50,000 Monte Carlo simulations based on the inputs provided. In each simulation, the platform is determining based on your inputs whether or not the specific event is expected to occur, and if so, how often and how much it will cost each time. The result? Summarized reporting at both an annualized and per event level. These results will provide an understanding of exposure from each bug, which can then be used to prioritize your remediation efforts.
NOTE: with your highest risk bugs it can be beneficial from a wider business perspective to do an in-depth analysis and understand with greater precision the frequency and magnitude of loss. However, rapidly assessing the risk of different bugs accomplishes the goal of software development bug remediation. If you want to do a more detailed analysis following the Rapid Risk Assessment, you can use the “Guided” functionality of the platform to estimate at TEF and VULN rather than LEF to gain gain some additional precision. Always remember to keep in mind the diminishing value of returns, though – is the added precision worth the additional time spent in gaining it? Are the results as they are good enough for the purposes of the decision being made?
How to Prioritize Bugs with Quantitative Risk Analysis
Understanding the exposure faced from a given bug is the foundation of prioritizing bug remediation. There is no right way to prioritize – the important point is that you use your quantitative risk analysis to make an informed decision about how to prioritize bugs in a way that is consistent with the goals of your organization.
For reference, often scenarios are prioritized based on:
There can even be greater nuance to prioritization than this. For example, depending on your organization’s security stance you might prioritize high frequency availability scenarios over confidentiality scenarios that have significant financial impact but rarely occur.
Long Term Value of FAIR Analysis in Software Development
Using RiskLens for bug remediation has long term value for software development efficiency and organizational risk reduction. Some good long term use cases to build around such an analysis are:
RiskLens is leading a revolution in the way cyber risk is assessed, measured and managed by bringing to market a Software as a Service solution that makes cyber risk quantification a reality.We help organizations translate cyber risk from the technical into the economic language of business.Schedule a Demo