Software Quality Trends

Franco et al state that there are the 4 R's with regards to trends in Software Quality Assuranceⁱ;

• Risk
• ROI
• Regulations and Compliance
• Rich Customer Experience

ROI is defined as “Return on Investment” this term is borrowed directly from the financial industry and refers to how much of a return is gained from a given investment.ⁱⁱ

ROI is calculated using the following formula:

ROI = (Investment Gain - Investment Cost)/ Investment Cost)

The ROI of Software Quality Assurance can be calculated by utilizing COCOMO based estimationsⁱⁱⁱ.

Basic COCOMO utilized thousands of lines of code as a base metric of calculation; it also has only three classes of Software Project team size:

1. Organic Small teams with extensive experience and non-rigid requirements
2. Semi-Detached Medium sized mixed teams with mixed experience and rigid requirements
3. Embedded Tight Constraints such as hardware or operational environments

Basic Equations are as follows:

Effort Applied in Man-Months:

where KLOC is 1000 lines of code;

Development Time in Months:

People Required:

Equation Co-efficients are as follows:

Software Project
Organic	2.4	1.05	2.5	0.38
Semi-Detached	3	1.12	2.5	0.35
Embedded	3.6	1.2	2.5	0.32

COCOMO is a standard cost model developed by Bohem; we can integrate COCOMO into ROI calculations to develop cost projections for any given project.

The cost model also requires that we estimate the size and potential earnings of a given software product. Sales forecasting is a common method used by industry to estimate earnings for a given quarter based on sales from previous quarters.

A standard method to forecast sales is to use a moving average model, the following formula is a simplified method used to forecast moving averages^iv:

First we must calculate the current average using the previous quarter's earnings data:

Then we calculate the new average using the projected quarters data:

This is a simplified non-weighted formula, and is used for illustrative purposes; weighted moving averages are far more accurate but also far more complicated and may require smoothing and various other considerations given a specific dataset.

Another needed variable is the loaded labor rate for the given people used or the project member loaded labor rate, the loaded labor rate is the cost per person per hour including all incidental costs of said persons employ.

 Where w = number of hours in a working day; this value varies per employer and locale; Cb equals the cost of benefits and Cc equals the cost of office space for the individual in terms of Lease per square feet of space and Cd equals any other incidental costs. All other costs associated with the employ of this individual must be added to this formula for each employee on the project costs such as parking or transit subsidies, health club memberships should also be considered to further improve the loaded labor rate calculations.

The total SQA cost may be estimated by using COCOMO and Loaded Labor rates using the following formula:

The Cost of Software Quality equals the daily loaded labor rate times the number of days required multiplied by the the number of people involved.

Given the sales projection and the loaded labor rate cost for the project members the ROI of the SQA portion of a software product may be calculated using the standard ROI formula as follows:

Thus the return on investment includes the Sales projection minus the cost of software quality over the cost of software quality. Should the projection be very large; say in the millions of dollars then the return is positive. If the return is negative then the organization may need to re-examine the risk models for this project.

The applications of the above formula for software quality allow an organization to determine weather or not a given investment is sound and offers great value for a given project and it's software quality. Practical applications include the costs and benefits of testing a given operating system before sale and the potential gains; however this model is limited in it's projections and the availability of previous financial data. Finical estimates are just that; educated guesses about how well a product may do; thus the ROI is variable and dependent upon estimations. 

References

iFrancino, Yevette (Tech Target, March 8^th 2010) Software Quality Insights [Online] World Wide Web, Available From: http://itknowledgeexchange.techtarget.com/software-quality/sharma-opines-software-qa-trends/ (Accessed on March 25^th 2011)

iiN.A. (Princeton, WordNet 3.0) Return On Investment definition [Online] World Wide Web, Available From: http://wordnetweb.princeton.edu/perl/webwn?s=roi (Accessed on March 25^th 2011)

iiiBohem, Barry (Prentice Hall, 1981) Software Engineering Economics ISBN: 0-13-822122-7

ivN.A. (SAP, n.d.) Forecast Formulas [Online] World Wide Web, available from: http://help.sap.com/saphelp_nw70/helpdata/en/96/d26256e39011d3b78e0000e82debc6/content.htm (Accessed on March 25^th 2011)

CPM and Earned Value

Objective Achievement Comparison

We may assume that Xrider has formal methods for project management and reporting in place to manage each of these departments including formal methods of Gantt Charting[i] and project cost tracking; these charts would include costs and milestone tracking and would be managed by the assigned project managers for each of the departments. To develop any formal method of achievement analysis and comparison we would first require data and methods to be documented and available within a database. With respect to Xrider we would be using the total cost of development and the project burn rates for comparison. The daily project burn rate for a given software project will be defined as the total project development cost per day. This may be calculated with the following formula:

1/CPI

Where CPI equals the Cost performance Index.

Documenting Software Projects

Projects may be documented by Mile-stones and by using requirements to function maps. These maps would be rolled into the project management suite of software such as MS Project or Rational. These tools have built in functions to associate costs to timelines which are critical when using objective or formal methods of project cost analysis.  EVM is an ANSI Standard EIA-748A and is used widely by the DoD in the united states and was popularized in the 70’s.[ii] Once costs and timelines exist in a database they may be subject to formal analysis.  

Project Management Methods

Earned Value Management

Earned value management is a formal technique for measuring performance and progress within a given project in an objective manner[iii]. EVM is an ANSI standard available directly from them[iv]. EVM consists of the following three components:

1.      A project plan that identifies the work to be done.

2.      Valuation of planned work (called PV), or budgeted cost of work scheduled (BCWS).

3.      Predefined earning rules or metrics used to quantify the amount of completed work (Earned Value or Budgeted Cost of Work Preformed).

The following formula issued to track a project according to earned value:

Figure 1

This equation allows us to determine whether or not a project is over or under-budget[v] this equation may also directly determine the financial risk of a given project.

Figure 2

This chart is for illustrative purposes used to demonstrate how the above function may determine project budget status the value on the X-axis is the number of weeks, the value of the Y-axis is the amount of money spent in $1000’s.  As seen here the actual cost is running at less than the projected value of work, this is an example of a project delivering under-budget.

Earned Value Management in Depth:

Budget at Completion (BAC)

This is the total planned budget for the project, how much will it all cost?

Cost Variance (CV)

Figure 3

If the cost variance is greater than 0 it’s considered good. If it’s less than 0 the project is over budget.

Cost Performance Index (CPI)

Figure 4

If the cost performance index is greater than 1; the project is under budget; less than 1 the cost of competition is too high and greater than 1 the cost may be too low and quality may suffer.

Estiamte of Completion (EAC)

Figure 5

The Estimated Cost of Completion (EAC) is the projected total cost to complete the project.

Estimate To Complete

Figure 6

The estimate to complete the project is the amount required from the present to the completion of the project; ie; how much is left?

To Complete Performance Index (TCPI)

The To-complete performance index is designed to provide a projection of the anticipated performance required to achieve either the Estimated At Completion or Budget At Competition.

TCPI (BAC)

Figure 7

TCPI(EAC)

Figure 8

Independent Estimate At Completion

The IEAC is a metric used to determine the performance of a project to date to be compared to the EAC.

Figure 9

All earned value formula are provided within the standard by ANSI and are cited here for discussion.

Critical Path Methods

Other formal methods apart from Earned Value that may be used include the Critical Path Method.[vi] CPM was used by the United States government to manage the Manhattan Project.

The critical path method is designed to use PERT Charts to continuously monitor project milestones and to alert the project manager if any critical component risks project delay[vii].

The basic functionality of CPM is as follows:

1.      List all activities required to complete the project

2.      List the time taken for each activity

3.      Link each activity by it’s dependency.

Here is an example of a PERT Chart[viii]:

Figure 10

Pert charts are a good way to offer an illustrative estimate of project completion dates and can be used with CPM estimates to develop project costs.

Limitations

Critical path and Earned Value management are both formal fiscal and time methods and measure only the quantitative data available for a given project; they do not include Culture of the organization or any qualitative methods such as Errors or Defects in the software projects as released and therefore offer no considerations to project quality. These are good methods to evaluate the return on investment for a given project do not make any guarantees on project quality at delivery. Thus by only considering financial and time domain data when utilized effectively they may still result in a software product that has insufficient quality.

PERT is also subject to a large judgment bias of the project manager, and assumes that all datasets are of equal value thus it may not work with unusual development cases.

EAV is not compatible with AGILE driven software development methodologies as it requires a project plan; Both CPM and EVM require that project accounting and project network schedule management are in place at Xrider and without either neither would be useable.

References

---

[i] H.L. Gantt, (The Engineering Magazine, New York, 1910; Easton, Republished by Hive Publishing Company, 1974) Work, Wages and Profit, ISBN 0879600489

[ii] N.A. (NDIA, PMSC, 2005) ANSI/EIA-748-A Standard for Earned Value Management Systems [Online] PDF Document, Available from: http://www.srs.gov/general/EFCOG/02GovtReferences/03NDIAANSI/NDIAIntentGuide.pdf (Accessed on March 18th 2011)

[iii] N.A. (NDIA, PMSC, 2005) ANSI/EIA-748-A Standard for Earned Value Management Systems [Online] PDF Document, Available from: http://www.srs.gov/general/EFCOG/02GovtReferences/03NDIAANSI/NDIAIntentGuide.pdf (Accessed on March 18th 2011)

[iv] N.A. (ANSI, EIA, 2007 ) Earned Value Management Systems  ANSI/EIA 748B [Online] PDF Document, Avaialble from: http://webstore.ansi.org/RecordDetail.aspx?sku=ANSI%2FEIA-748-B (Accessed on March 18^th 2011)

[v]N.A. (Defense Systems Management College, 1997). Earned Value Management Textbook, Chapter 2. Defense Systems Management College, EVM Dept., 9820 Belvoir Road, Fort Belvoir, VA 22060-5565.

[vi] Kelley, James; Walker, Morgan. (Eastern Joint Computer Confrence, 1959) Critical-Path Planning and Scheduling.  [Online] PDF Document, Available from: http://portal.acm.org/citation.cfm?id=1460318 (Accessed on March 18^th 2011)

[vii] Kelley, James; Walker, Morgan.(Institute of Radio Engineers, Professional Group on Electronic Computers, 1959)  Critical-Path Planning and Scheduling. [Online] PDF Document, Available from: http://portal.acm.org/citation.cfm?id=1460318  (Accessed on March 18th 2011)

[viii] N.A. (NetMBA, PERT, ND) PERT CHARTS [Online] World Wide Web, Available form: http://www.netmba.com/operations/project/pert/  (Accessed on March 18^th 2011)

The grain and it's beach

A software unit is defined by the IEEE in 1008-1987 asⁱ:

“The smallest unit of a software program.”

Depending on the type of program being tested the software unit may represent a single function, sub-routine, or given algorithm. The languages may be functional, procedural, low level or object-oriented, however each language and program will have a defined function and desired operation.

Since all software applications are composed as the “sum of their parts” however simple or complex the program may be; each of the parts plays a role in the operations of said program. There are numerous philosophies around what makes a high quality reliable software; the Unix paradigm states that a program should be as small as possible and have only one functionⁱⁱ. This Bottom-Up approach has created Linux and the entire open-source movement. Top down approaches exist and most formal testing and development methods are top-down in nature, Agile and XP and SCRUM are examples of methodologies that are top-downⁱⁱⁱ. Each of these approaches has their respective merit; however we are simply mentioning them here to illustrate what a software unit may be defined as.

The importance of isolation of a given software unit for test is that this unit must be verified according to formal methods to ensure that for all desired operational states the unit functions as required, since malfunctions of the software unit will ultimately lead to errors, faults and failures.

Formal methods for software verification include; Black Box, White Box, Correctness and all other validation and verification methods as defined by Galin^iv.

An oft used metaphor is that of an internal combustion engine; it's purpose is to generate motive power from some combustible fuel, all engines have the same purpose and within modern engineering, one stated piece of common knowledge is that if a given engine has too many moving parts it will be less reliable than a similar one with fewer moving parts.^v Fewer parts means less to potentially go wrong.

Within a software application, the “moving parts” are composed of mathematical abstractions defined as software units. The verification of which requires that both the logic of the math behind the algorithms being tested be sound.

The methods to isolate the software unit as defined by Galin^vi include “Black-Box” and “White-Box” testing; where the only major difference between the two is access to the source code. If one has access to a programs source then each “Sub-routine” may be defined as a software unit for testing purposes. Without access to the source code, black box testing focuses on application functions versus stated business requirements.

Another method of testing quite frequently used to locate software defects is disassembly and reverse engineering; disassembly of obfuscated binaries allows anyone with physical access to computing resources to view and interpret a software programs intent and function.^vii All malware development is based on defects found utilizing static disassembly techniques. Though disassembly is not a formal method it does offer insight into the function of a given software program without access to the relevant source code.

The modes of testing may be automated; or standardized and manual. Each has their respective benefits, most automated tests using software will offer only as much insight as they are programmed to offer. Standardized manual tests for senarios that are seen as eccentric or unique will further increase time requirements but they will also increase the reliability of a given program or system.

The success or failure of a given software unit may be defined by the test data from the unit testing; this includes conformance to all formally stated requirements for function; conformance to all expected input via output and functional tests and general sanity regarding the function as defined mathematically.

Since software is the most complex machine ever developed by man, each unit equating to a given mechanical cog; should a single cog rotate ineffectively; or worse rotate backwards, then the machine will fail. Each grain of sand may not be visible on the beach; by removing the grain and inspecting it we can discern it's origin, composition and age. This is the goal of software unit testing; it's designed to ensure that the software preformes as it should by testing as many parts as is warranted by the project at hand.

iN.A. (IEEE, ANSI, 1987) 1008-1987 IEEE Standard for Software Unit Testing [Online] PDF Document, Available from: http://standards.ieee.org/findstds/standard/1008-1987.html (Accessed on March 1^st 2011)

iiMcIIlroy, M. D.; Pinson D.E.; Tague, B.A. (Bell Labratories, 1978) The Bell System Technical Journal, “Unix Time Sharing System forward” 1978, 57 (6 Part 2) P. 1992 [Online] World Wide Web, Available from: http://www.faqs.org/docs/artu/ch01s06.html (Accessed on March 1^st 2011)

iiiStallings, Brian (Agile Journal, March 2007) Agile Top-Down: Striking a Balance [Online] World Wide Web, Available from: http://www.agilejournal.com/articles/columns/column-articles/273-agile-top-down-striking-a-balance (Accessed on March 1^st 2011)

ivGalin, Daniel (Pearson Addison Welsly, 2008) Software Quality Assurance from theory to implementation P.187 ISBN:0-201-70945-7

vBrommley, Richard; Newton, David; O'Connor, Patrick T; ( John Wiley and Sons, 2002) Practical Reliability Engineering P. 215-221 ISBN: 0-470-84463-9

viGalin, Daniel (Pearson Addison Welsly, 2008) Software Quality Assurance from theory to implementation P.189-192 ISBN:0-201-70945-7

viiKrugel, Christopher; Robertson, William; Valeur, Fredrick; Vigna, Giovanni; (University of Santa Barbara California, 2004) Static Dissassembly of Obfuscated Binaries [Online] World Wide Web, Available from; https://www.usenix.org/events/sec04/tech/full_papers/kruegel/kruegel_html/disassemble.html (Accessed on March 1^st 2011)