New Technology, Same Governance Challenges
By Simone Rede and Brian Evans
A quick review of ALGA’s audit abstracts or news articles yields no shortage of examples of how implementing new technologies can go wrong. Projects that miss deadlines, cost more than expected, or deliver less than promised seem to be more the rule than the exception. Why are there so many examples of similar problems? What can auditors do to disrupt these patterns?
From our perspective there is a common causal factor – people. That may sound like an odd place to start an article about technology, but we think it’s important to keep people front and center when thinking about auditing technology. While new software (virtual tools) and hardware (tangible tools) can increasingly do the bulk of the work for us, their utility may be reduced unless the people who make decisions have clarity about what it’s supposed to do (user perspective) and how it works (technical perspective). If there’s a disconnect between these two perspectives, there’s a greater chance the new technology won’t make things better, and could make things worse.
Government auditors don’t audit people, specifically. But we can audit the structures used to inform decision making. This is governance. Effective governance is critically important to managing and getting value from new technology. Each of the technology-related audits we’ve completed over more than 10 years at Metro has included one or more recommendations to improve governance. We hope that helps performance auditors see that audits of technology don’t have to be overly technical in nature to be effective.
When deploying technology, it’s vital to have the right people at the table with clear lines of authority and responsibility. For information technology (IT), effective governance typically requires knowledge not only of the technology, but how it will be used. Having knowledgeable people from both the IT department and the department where the technology will be used is vital for successful projects. However, it also means that knowledgeable people at the top of the organization need to be involved in conversations about how the project fits with all the other IT systems that are already in place or anticipated in the future.
As such, IT governance must be applied not only to projects, but also across projects. Ideally, there will be a project-specific governance framework and a governance framework for the entire portfolio of technologies used across the agency. In practice, many organizations struggle with project governance, which makes agency-wide strategic governance seem like an impossible dream. The need for multi-level governance is why maturity models are fairly common in IT auditing. Maturity models help organizations gradually increase their sophistication by setting incremental milestones to move from successful project management to successful portfolio management.
Maturity Models Show a Path to Gradual Improvement
Two Steps Forward, One Step Back
We’ve used maturity models as criteria for our IT (and non-IT) audit methodologies. We’ve also used maturity models in reports as a communication tool to help readers of all levels understand the key aspects required to manage IT investments effectively. Maturity models seem to resonate with auditees because they show a path of gradual improvement. Each level of maturity involves specific actions and capabilities that can provide a template to implement audit recommendations.
Unfortunately, even though maturity models have been well received by management, our audits have shown that even well-intentioned governance structures can breakdown quickly when employees, senior leadership, or priorities change. We’ve summarized some of our technology-related audits below to show you how things can improve, and why things may stay the same.
In 2009, we completed an audit of IT software controls. The audit was designed to address concerns that there were redundant systems in place across the agency that led to higher costs and reduced ability to share data. We reviewed three systems in depth: one that was purchased from a vendor, one that was developed by Metro’s IT department, and one that was developed by the department where it was used. These examples showed variation in how IT projects were managed, but more importantly, they illuminated a common problem across all three – a disconnect between IT developers and IT users.
The system purchased from a vendor worked well, but no one from Metro could make changes to it as operational needs changed. The system developed by the IT department got far enough to replicate what the legacy system was used for, but didn’t deliver any of the additional functionality that was expected. The system developed in-house, but outside of the IT department, had limited security and was built using software that could not be scaled up as more and more data was collected. Each of these examples pointed to different risks that needed to be managed.
We made recommendations to address each of these system-specific risk areas, but the larger governance recommendations were written to address the root cause of these problems. Management agreed with the audit recommendations and we saw good progress in implementing them when we completed a follow-up audit several years later.
Recent Review Showed Stronger Governance Was Still Needed
In 2019, we conducted an audit of information security and technology to address cybersecurity risks. The results of that audit again demonstrated the importance of effective governance. Technology had changed, but the underlying weaknesses were almost identical.
Our audit of information security and technology found some governance best practices were in place, but they were not designed or carried out effectively to manage IT resources. Metro created a governance committee for IT projects in response to our 2009 audit, but it was disbanded in 2015. The IT department documented its five-year mission critical efforts, but they were not prioritized. Responsibility for managing IT was shared among departments, which gave the IT department less authority. We recommended Metro improve IT governance by developing a strategic plan and establishing a governance structure to oversee its implementation.
To reach these conclusions, we used general criteria across three categories to assess Metro’s governance: authority, processes and planning, and oversight. We derived those categories from a Canadian Audit & Accountability Foundation discussion paper on root cause analysis in performance auditing. That paper proposed a list of categories, possible root causes, and considerations tailored to the government environment. The list includes a mix of governance-related and operations-related matters, and we have used it to scope subsequent audits.
Helping Auditees See How Governance Reduces Risks
In our 2019 audit, we used the Fair Information Practice Principles (FIPPs) to determine if Metro’s governance structure was effective for managing information security risks for surveillance cameras. FIPPs are used by organizations worldwide, including the U.S. Department of Homeland Security, to develop policies and procedures governing the use of personal information. They include:
- Specifying and documenting the purpose(s) for using cameras
- Providing notice where cameras are used
- Protecting cameras against unauthorized access
- Minimizing storage of footage
We identified these principles from a 2013 Public Safety Camera System Audit from the Office of the Austin City Auditor. Our audit lists other potential criteria for information security and technology best practices. Criteria from the Payment Card Industry Security Standards Council and the Government Accountability Office were especially useful to our reviews of two other technologies included in the audit (payment card data protections and cloud computing). We summarized the results of those reviews, and our conclusions about Metro’s management of IT investments, in the following table.
Governance, Governance, Governance
We found that Metro followed some aspects of best practices for governing security camera usage. However, more strategic direction and oversight was needed. There were no committees to make decisions about the cameras, or performance measures to monitor their effectiveness. Policies for retention of footage were developed, but policies and procedures were not established to maintain security, privacy, and fairness. Those terms generally refer to an organization’s safeguards, limits, and openness around its collection and use of personal data. They were not fully addressed by Metro’s policies and procedures, which meant that camera use was not as transparent as it could be, and access to footage was inadequately protected. Stronger governance would help Metro better protect the cameras and the information they collected from employees and the public.
We found that Metro’s purposes for using cameras were not specified and that notice was not always provided that cameras were in use. Who had access to footage, and how it was granted, was not documented. And, footage was stored longer than Metro’s retention schedule allowed. These gaps could lead to abuse and put employees or members of the public at a disadvantage, which may cause Metro to lose customers.
To improve governance for surveillance camera usage, we recommended Metro develop policies and procedures for surveillance cameras that:
- Specify the purpose(s) for which cameras should and should not be used
- Ensure notice is provided where cameras are used
- Establish processes for granting access and sharing footage
- Retain footage consistent with Metro's Records Retention Schedule
One Year Later, Governance Improvements Still Needed
This year was the first time management was asked to self-assess the status of the recommendations we made in the 2019 audit. According to information reported in early 2020, all but one of those recommendations was in process. However, management’s comments indicated few improvements had been made to IT governance and governance for surveillance camera usage. Metro had not developed a strategic plan or established a governance structure to oversee its implementation. The information provided by management suggested some governance-related investments had been made. They reported that Metro had selected a contractor to create a strategic plan.
Policies and procedures for surveillance cameras had not been developed to address the gaps we identified in specifying and notifying their usage, protecting them from unauthorized access, and minimizing storage of footage. Management also reported that it had set up a governance group for video recording systems that would develop an agency-wide policy, as well as department-specific procedures. These investments underscored the need for a stronger governance structure. Without one, it will be more difficult for Metro to determine its strategic direction for IT, sustain actions to protect surveillance cameras and footage, and overcome other weaknesses in IT management our audits have identified.
Make Governance Recommendations, and Then Make Them Again
The audits referenced in this article show how technology can underperform when the right people aren’t at the table to work through technical and user needs. When the user perspective is overly represented in decision making, new technology may end up looking similar to existing processes, which could reduce the value of the new investment. On the other hand, when the technical side is overly represented, employees and clients may not be able to navigate the new system, which could reduce efficiency.
The technologies and specific examples we’ve audited varied over more than 10 years of IT auditing our office completed, but the underlying causes were the same – insufficient governance. We hope the examples give you ideas for IT-related audit methodologies and criteria. We also hope the article demystifies IT auditing. It can often be very similar to any performance audit. If there are specific technical risks that need to be assessed, an outside specialist may be worth the investment.
About the Authors
Brian Evans became the Metro Auditor in 2015. Brian holds a Master's degree in Public Affairs and is a Certified Internal Auditor and Certified Government Auditing Professional. He serves on the Executive Committee of the Pacific Northwest Intergovernmental Audit Forum (PNIAF) and on the Education Committee of the Association of Local Government Auditors (ALGA).
Simone Rede joined the Metro Auditor's Office in 2015 and is currently a Senior Management Auditor. She has a Master of Science in Public Policy and Management, and is a Certified Government Auditing Professional. Simone serves on the Online Resources Committee of the Association of Local Government Auditors.