Six Aspects of Measuring the Quality of Software

This is a post I wrote a long time ago when I first read “Espresso Coffee, The Science of Quality.”  I am finally getting around to posting it here because I just now noticed I never posted it here, it was just a thing I passed around on the Internet.

Lately, I have found myself curious about what quality means, and how I can better control it in what I do. As I researched quality, there grew in me an even greater confusion about quality, and what it means, how it is interpreted and understood,  and the implementation of it. That is, whether or not I can always  hold to the same standard of promised quality as the highest uncommon factor.

This pertains to working with clients, with other departments, with myself, and the expectations people have had regarding the deliverable, and the effective quality of the interaction they have with my output.

Value is in the eye of the beholder, quality is perceived. There is an interesting story I once read about an appraiser who was asked to appraise two antique secretaries which appeared to be identical. He was told that one of them was a forgery. He appraised them both at the same value, stating that he could not detect the forgery. When the maker revealed which one was the fake, the appraiser inquired as to how he had made it so perfectly. The cabinet maker said that he found several 200 year old pieces that were damaged and used the wood from them. He used  adhesives which he made himself from raw materials accurate to the period and heated them to accelerate the aging.  He built his own circular saw to cut the planks to size using an antique saw blade, and he discovered a process whereby he could age the cuts perfectly (he wrapped in layers of burlap and buried them in a swamp for 20 years). It took him 30 years to build the secretary, and was his life’s work.  Upon hearing this, the appraiser certified the piece as an original. He said that, for all intents and purposes, the piece was in fact 200 years old.

Software has three perceptible areas of quality…(three that I am going to talk about)

  1. The quality of the product.
  2. The quality of our responses to our clients which includes traits that are  perceptive, substantive and literal.
  3. The quality of service the product delivers.

A software company, as a whole, can be said to be in the business of producing quality.  For a software firm involved in the litigation industry, for example, total quality delivered is the offering of products and services.  This cascades outward and downward from their internal total quality approach in their service vertical, through to their customers, out to the industries they serve (their corporate and private clients), to the courts, and ultimately to the consumers who buy the products and services offered by the litigants. Essentially, a software firm in the legal industry provides a core utility that services the national infrastructure.

As such, the requirement for quality is not only essential, it is fundamental to national interests. When we neglect even one small part of the quality of our processes, it is not only detrimental to our business, but it has a much wider, and deeper impact than perhaps many realize (or care to think about).

From the point of view of a customer, then, it becomes necessary that we really think about what quality means, and how it can be measured.  We need to think about the  different states of quality, and how to develop them progressively.  Customer satisfaction, after all, is a dividend of perceived quality and expected quality. If the expected quality is high, and the perceived quality is low, the satisfaction of the customer suffers. If the perception of quality is high, and the expected quality is low, then we have exceeded expectations. Exceeding expectations is not a quality. If, in the end, the expectation is that quality will be very high, then it can’t be very well higher than itself.  In the end, “quality” is the measurement of the product against the expectation of our clients, ourselves, and each other. It measures how well we meet the expectation. Such is the expectation that we have, that it is one that cannot be exceeded.

The expectation is this – that we will do everything within our means to achieve  as near to real time excellence as is humanly possible without risking life, limb, personal society, damaging our health, or creating permanent insanity. What this means is that we care for our families, we care for our health, and we care for our clients with every available ounce of our being. Our measurement of our ability to do this then becomes not a measurement of gradient expectation, but rather one of quality, which in our work means time to resolution. Our mean time to resolution, across the board, must be faster than everyone else’s. To be the leader, and to stay out in front (or to get there), you must deliver this level of speed on a continuous basis.  The word “quality,” then, is not an aspect in and of itself.  It is a unit of measurement, not unlike an inch on a ruler stick.  To say something is “quality” is not unlike saying something is “inches” when asked “how long is that stick?”

How we use quality as a tool of measurement, then, both internally and externally, become a necessary area of focus.

Firstly, we look at the different states of quality. For this exercise, I’ve picked six different qualitative attributes.

1. Promised quality. Briefly stated, this is the expectation that a person has of a particular product delivered by a particular entity.  Externally, customers have this about the kCura product, and internally this manifests as an internal/external unified attribute, with cause and effect vectors that must be appreciated.

Externally, the promise of quality is high, both in the product and in the service. Since the two are so closely intertwined, that is, the service delivered is part and parcel of the software, I am referring to both collectively as “the service.”  There are some areas of the product where the quality of it is dissonant with the promise, but these are likely caused more by technological and human limitations than by willful oversight.

At times, this causes customers to pigeonhole the product as one that delivers a few, very strong, core competencies, but falls short on other offerings.  This requires kCura clients to do things like hire DBA’s, outsource certain aspects of IT, and purchase other software bundles which they then integrate with the Product.

Internally, at many organizations, there are few promises of quality (but the expectation and implied promise is there, it just isn’t well defined, cross departmentally). When you don’t promise anything, you don’t have to worry about not being able to execute on it.  Where the promise of quality does exist, it does not always mesh well with other, external facing promises.

At times, this causes a disconnect when those who face the customers must interact with those who typically don’t. It may even cause a blatant contradiction that results in conflict.

2. Industry (expected) quality.
These are the standards that govern the very minimum expectation that must be met. Consumers that use discovery products are accustomed to certain features working in a certain way, and a certain guarantee of accuracy. Companies that create food products, likewise, must ensure certain standards are met.  Across the board, in areas of comparability, a quality focused company typically meets or exceeds expectations, with a corrective “roadmap” in place to address shortcomings. Ultimately, there are many, many standards in the industry. Some of them are driven by legal (forensic) requirements, others are aesthetic or purely usability driven.

3. Effective Quality
These are the actual objective and subjective metrics of an organization. What are the promises made by the product, and how does it hold up against them?  The items that comprise your “effective quality” are real, tangible metrics that can be measured, discussed and improved.  Examples of these things include usability, scalability, processing speed, error handling, recovery, precision and accuracy, traceability, and most importantly, reliability.

4. Relative experiential quality
This quality refers to the differential experience created by the actual implementation of the product by the purchaser.  A more experienced team that delivers the product, on better equipment, will deliver a user experience that can have substantially varying degrees of experiential quality.
relative experiential quality = expected quality / experienced quality

5. Perceived quality
Satisfaction = (perceived quality / expected quality) * relative quality

6.  Potential quality

The degree to which a product or service can be measurably improved. This is measured and communicated by an organization’s roadmap, and should be readily available to it’s consumers.


All six of these are qualities that are assessed by consumers of a product one by one.  To be considered a quality, that is, a quantifiable entity that can be measured, each quality must map to a tangible, perceivable action that can be consumed, viewed, experienced, or understood to have intensity, duration, and frequency.  To whatever extent possible, then, quality is something that is measured against the whole.

Without the whole, without an accounting of the sum of all parts of a thing, it follows then that a measurement of quality is purely a subjective, out of context measurement that is rendered meaningless against the overall impact and perception of the qualities of a product.  Those that actually interface with the product or service in its entirety are the best at explaining its level of quality.   To measure quality requires, then, a broad range of measurements that must be painstakingly derived and executed against the subject as a whole. This is no small task, to be certain, but one that will reward the quality analyst with a rich array of tools that can be used to improve the product and services offered.