Go To Search
Click to Home
See Things Differently
By Gary Blackmer

So much of our audit training and practice seeks answers with a yes or no, a pass or fail, an appropriate or not. We are trained to think night or day, with neither a dusk nor a dawn. The options are clean, simple, and unambiguous.
shutterstock_257011963_350.jpg 


Bisecting the world this way reduces the vagaries of personal judgment in decisions and actions, and promotes precision, consistency and equity. If we find something in a grey area, then our impulse is to recommend more robust procedures because certainty is our north star. As an aside, how many times have you written a finding that says, “Not all…”? You conclude that anything less than 100 percent is non-compliant. In dire matters we should expect complete control. But for many of the others, would it suffice to report, “We found a small number of cases that did not…"? Just asking.

Innovation and creativity come when we see the world in its rich variety, even if a robust understanding makes our work more difficult. Let’s talk about a complex scenario with a big impact. You examine the productivity of 100 restaurant inspectors and find that some inspectors only examine 150 restaurants over a year’s time, while others may inspect as many as 350, with most falling around the range of 250 inspections. Of course, travel time between restaurants could be the major determinant, but this is just a thought experiment so let’s pretend it plays no part. Before you even think about audit objectives and methodology, let’s consider the graph below.

Blackmer1.png

You could audit the low productivity of the inspectors who fell below the median. With this approach your recommendations might address closer supervision or work improvement plans for poor performers. If you can cut that left tail you can increase productivity, resulting in about 2 or 3 percent more total inspections.

But consider the other end of the tail: what are these inspectors doing to accomplish upwards of 300 or 350 annual inspections? If your recommendations incorporate their successful methods and strategies, you have the potential to produce a much greater impact. (There is also the risk that their inspections are cursory, which should be one consideration before reaching your conclusion.) The graph below shows a shift of the whole bell-shaped curve to the right, to a higher level of performance. This shift would increase total inspections by 6 or 7 percent, more than doubling your audit impact. In this scenario, that increase is the equivalent of adding 10 more inspectors. Just looking at those poor performers might represent only 3 or 4 more inspectors’ work.

Blackmer2.png

Bell-shaped curves are the most common distribution of population characteristics in our world. Ask a statistician, or yourself if you took a statistics class. The challenge is to think in bell-shaped terms. Always remember that an average has a distribution on either side. You can plan for the average but expect the variance. And always be aware that an encounter with a situation, person, or statement could be an outlier on either tail.

Similarly, a large portion of an organization’s efforts will not produce good results because of variances in the world, or in people. Despite the best treatment efforts for substance abuse or mental health, many clients will fail, even while others succeed.

With this larger look at the diffuse nature of our world, let’s tackle another distribution. Think about an organization of 200 people who work in public health clinics. The staff possess a diverse group of health care skills, supported by a smaller support staff providing human resources, financial, procurement, and logistics.

Consider these questions: How many actions do they undertake in a day? How important/costly are these individual acts? What could be audited, beyond what you usually think about auditing?

In a day, those 200 people probably perform thousands of acts or decisions, most very trivial like filing a record, while some others may be life-altering for a client. Each of these acts still requires the assemblage of information, weighing of options, and orchestration of efforts. Thankfully, a huge number of these acts are simpler and are carried out under the guidance of practice or procedure. And thankfully, a single failure in a less important decision does not jeopardize the mission.

Blackmer3.png


This skewed distribution of important topics is another way to think about what to audit. On the left end we are panning for gold dust. We can more easily audit these many thousands of decisions and activities because they are simple enough to be proceduralized and are probably recorded in computer-based record systems.

As we move farther to the right, up the hierarchy of importance, we get closer to service delivery and quality, and the mission of the organization. Audits in these areas require us to develop more complex audit objectives and methodologies. We may question scheduling decisions or staff deployment. Yet this is the area of gold nuggets, the mother lode that can deliver a much bigger benefit for the public.

I caution that some smaller decisions may impede the mission. Serious impacts can result if supplies are not ordered in time, or a clinic’s heating system fails, or records are lost.

Replacing an enterprise data system is another example of a skewed distribution of programming decisions. Thousands of small interdependent components can produce a dysfunctional system if they aren’t properly orchestrated to accomplish the larger objectives and mission. On the far right of the graph, the decision to replace the system is the most significant, and then there are lesser decisions around planning, procurement, and quality reviewing.

These distributions can naturally shift over time as well. Think about new software in the office. Some people are “early adopters” and they comprise one tail, while most of the others become proficient with a little more time, and then a few are “laggards” who move much slower. Check in six months later and the curve is likely to tighten and shift toward more universally capable users of the software.

When we audit these “fuzzy” activities and results, we must be cautious and take more care in writing our findings and recommendations. Our descriptions of the results should acknowledge the data variances and add qualifications to the limits on precision and accuracy.

Our audit standards emphasize data validation for our findings and I don’t disagree with this important principle. But this too is not an absolute and our judgment is an important element in audit planning. For example, we are trained that observation is the most reliable evidence we can obtain, and that is true for the pass/fail testing. If we observe even one failure we can be more confident than any examination of records. But a few observations in a bell-shaped curve won’t give us conclusions unless our observations follow strict sampling methods, which can be costly and time-consuming.

I would argue that we can still write conclusions from even flawed data if we are trying to shift a bell-shaped curve. The standards address the need to write the narrative so that it can be supported by the evidence and we have the words to do that. If data processing chops off a tail, or data entry errors misrepresent 10 percent of the cases, or even 10 percent of the data is missing, we can still write our report in general terms. We can validate some data points to illustrate the pattern and use words like “about” and phrases like “at best.”

Also, keep in mind that the data is a flawed depiction of a bell-shaped curve that is occurring in reality. The bell is real even if our data is incomplete. Our goal is to improve performance when we have enough evidence, even if there is some missing data. When pieces are missing in a jigsaw puzzle we can usually identify the scene. We must not confuse the data for the reality, because we will have other evidence showing weak management practices for the services to the public. If the data says that 20 percent of services are unnecessarily delayed, should we suspend our work because we don’t know whether the exact number is actually 15 or 25 percent of the cases delayed? If we found strategies to improve timeliness, we should recommend them.

There is real gold to be found in a carefully weighing of the collective evidence. It is in the public’s interest for us to recommend better management of services and better data gathering, sooner than much later when management starts gathering data better. You can return for a follow-up of the audit to see what management has done to improve data gathering, and in your analysis of the data you may even see the bell curve shift toward better services.

You usually find what you’re looking for. See things differently and you’ll find more.


ABOUT THE AUTHOR

Gary Blackmer has been conducting audits for 30 years and recently retired from his position as Director of the Oregon Audits Division. The Division conducts performance, financial, and information technology audits, monitors financial audits of local governments, and responds to hotline allegations. Previously, Blackmer served 10 years as the elected Portland City Auditor, eight years as elected Multnomah County Auditor, a management auditor, and analyst for a variety of state and local agencies. Blackmer is a past-Chair of the Pacific Northwest Intergovernmental Audit Forum, and past-President of the Association of Local Government Auditors. He received the ALGA Lifetime Achievement Award in 2015.