Auditing Technology Deployments in Silicon Valley: Finding the Constants in an Evolving Technological Landscape
By Marisa Lin and Brittney Harvey
It was an IT nightmare: intractable software bugs. Endless testing. A nine-month project culminating in an expensive, four-year-long demise.
This was the unfortunate reality for the City of San José’s Finance Department. In 2014, the Department released a Request for Proposal (RFP) to procure a new software solution to replace its legacy business tax billing system. The system’s implementation, which began in March 2015, was scheduled to be complete in December of that year. Yet by 2019, the City was still without a working product, despite having spent over half a million dollars on development, plus additional consultant project management costs. Given the project’s difficulties, decision makers ultimately decided to suspend the project. What led to this failure? In a city that prides itself as “the Capitol of Silicon Valley,” what went wrong? Time to bring in the auditors.
A Rapidly Evolving Landscape
When we started our audit of technology deployments in mid-2019, we were entering a field that was rapidly evolving. Within the past few years, the city had launched several new initiatives to transform San José into a “smart city,” including the Mayor’s Smart City Vision, the Smart Cities Council Committee, the Office of Civic Innovation, and a dedicated project management group located in the Information Technology Department. With these initiatives, the city recognized that residents’ expectations of service delivery are shifting in this technological age, and specialized resources and management are needed to ensure successful deployments of new technology systems.
The question of how to approach an audit of a topic that feels like a moving target may sound daunting. It is best to understand, however, that no matter how much the technology changes or the swift pace by which new ideas and concepts arise, there are still constants that all technology deployments need:
- Established staffing and governance,
- Understanding that policies define authority,
- Internal controls to ensure accountability, and
- Having procedures to plan for success.
If any of these key elements are missing, there may be significant setbacks in rolling out technology systems, as was the case for the Finance Department’s new business tax system. We identified ways in each of these four areas that the City can dramatically improve its technology deployment process.
Governance: Who has the Final Say?
As technology deployments involve more stakeholders with differing workflows, goals, and business needs, they become more difficult to manage. This makes it increasingly crucial for project teams to have defined governance structures. Clear prescriptions of roles and authority streamline decision-making and prevent teams from becoming mired in competing (but valid) views, demands, and project visions.
One deployment we reviewed, for instance, was initially hampered by disagreement among departments and other stakeholders, particularly regarding whether the upgrade should be a simple facial lift or a more fundamental overhaul of the system. The lack of consensus contributed to the stalling of the project by nearly two years. As in any other project, an unclear governance structure with undefined decision-making authority and competing visions can easily derail a project from progressing.
It was only during a “project reset” that the team leading this deployment stepped back and established a governance structure with more specific decision-making authority for various roles.
The revised organization included designated leadership at each level with a clear reporting structure. A project sponsor—in this project’s case, a department director—heads the executive team and possesses the final say on project vision and other elements that greatly impact the project. Below the executive team is the transformation team, led by the product owner. This team is responsible for pushing the project towards completion while balancing the distinct needs of different stakeholders in a deployment.
Project leads, being the expert in their respective roles, assume specific roles in a deployment. These roles include an individual to manage the project, another to advocate for the business need, and still another with the technical expertise to understand the technical specifications and how the system operates within the larger technical environment. In smaller projects, individuals may be able to handle multiple roles, though that will likely not be the case in more complex, interdepartmental projects.
The creation of this formal governance structure helped move this project forward towards launch, something we noted that other projects could benefit from as well.
Policy and Authority: To Centralize or Not to Centralize? Striking a Balance
Mapping out a clear governance structure for a project not only clarifies who has the final authority on decisions, but it also helps to articulate the roles of team members, from subject matter experts to technical gurus. In particular, the role of the Information Technology Department (ITD) varied across the deployments we examined, from performing cursory preliminary reviews of RFP requirements to providing ongoing technical and project management expertise in project implementation.
Upon first diving into this audit, it became clear that existing policies did not clearly define ITD’s role and authority over technology systems in the city. According to city policy, a technology board is supposed to approve major projects, but the board having been inactive for over a decade, the Director of ITD now served as this approval authority. Understanding how this current arrangement emerged helped us determine the ideal role of ITD in facilitating technology deployments.
One reason for the varying levels of involvement of ITD across projects is the city’s history of decentralizing IT support in the organization. Multiple departments, including the Planning Department, Public Works, Department of Transportation, and the Police Department, have their own IT staff, which allows them to readily access IT support and tailor it to their needs. As an example, the deployment of a parking access and revenue control system, which manages gate-keeping and revenue collection for the city’s downtown parking garages, receives support from IT staff in the Department of Transportation who can administer 24/7 assistance that would be challenging for ITD staff to provide.
Decentralization, however, has its drawbacks. Working broadly across the organization, ITD staff have familiarity with the city-wide technology environment and associated risks (such as cybersecurity risks) that department-specific IT staff do not have. Moreover, since not all departments have their own IT staff, some lack adequate technical expertise to successfully deploy a project on time or may end up relying too much on an outside vendor to design and implement technical specifications (these were both situations we observed in our audit). Departments without IT staff did not have standard protocols or expectations in place regarding the kind of support ITD should provide for their deployments. Much of ITD’s involvement was on an ad-hoc basis, with departments consulting ITD as needed, typically through submitting help-desk tickets. We observed that some projects had minimal ITD support, while others had a dedicated ITD project manager. Through our audit, we recommended that the city clarify the role of ITD in the context of this decentralized IT environment.
Our audit identified a need to rethink the optimal balance between centralization and decentralization of IT support across the city and to standardize it through policy. This is especially pertinent in a time when cybersecurity attacks on governments and other entities are common, heightening the need for coordinated security efforts to ensure that a compromised system does not affect other systems in the city, and that adequate risk measures are in place for all departments.
Internal Controls: Accuracy, Objectivity, and Watermelons
In recent years, the city has been preparing and presenting an innovation roadmap to the City Council’s Smart Cities Committee, targeting projects that are core to the city, important to the community, and achievable at scale. This reporting has been the primary mechanism for departments to regularly update councilmembers on a deployment’s progress.
The Smart Cities Roadmap uses a color system that indicates if a project is on track (green), experiencing minor issues (yellow), or at risk of failing with need for corrective action (red). While this system is simple and straightforward, we identified some ways by which it can be improved.
One of the flaws in this reporting system was that it can oversimplify the actual situation. What determines a project’s color is not always clear. In one project we examined, an outside consultant performing a “health check” reported that the project was a “watermelon”! By this term, the consultant meant that the project was being reported as if everything was going according to plan (green), but below the surface, the project was at significant risk of failure (red). This was partly due to project progress being reported according to a revised timeline, rather than the initial one reviewed by policymakers.
Without a solid assessment of the underlying assumptions behind a project’s reported status, we may be left without a true understanding of what it takes to drive a project to completion. It also leaves decision makers and policymakers without the appropriate information to re-evaluate the resources necessary for project success.
As roadblocks—such as issues with budget, timeline, scope, or business need—arise, project staff and leadership must have accurate information to overcome them. This is why controls ensuring regular monitoring and reporting of deployments is important. Even before anything is reported, however, a team must first lay the foundation with adequate planning.
Planning: A Constant in a Swiftly Changing Environment
While not everything can be anticipated, proper planning can mitigate some of the risks related to adjusting a project’s direction later on, especially in the midst of an ever-evolving technological environment. To audit the planning phase, we first centered in on the core elements of planning that should always be present, regardless of how new or innovative the technology is.
- Engaging with those who will be most affected by the system, including those who will use it to deliver and receive services. Collecting users’ input and obtaining their buy-in can ensure that the technology system will address a real need.
- Identifying the core features that are essential to making the desired difference. Without this step, a system can become bogged down with additional features and even become an obstacle to an organization’s workflow, wasting time and resources—something we auditors shudder at the thought of.
- Resourcing a project to ensure its success. Resourcing includes both having enough funding and staffing with the appropriate knowledge, skills, and time to spend on a deployment. Absent sufficient resourcing, a project can experience unforeseen delays.
The failure of the Finance Department’s new business tax system can be attributed to a lack of planning. In a “Lessons Learned” debrief, city staff noted how the lack of preparation resulted in unnecessary complexity, disagreement, and unresolved requirements. In the next attempt, they intend to:
- Simplify the core product,
- Agree on a consistent project methodology (e.g., Waterfall or Agile),
- Establish a governance structure that would require sign-offs on product specifications, and
- Take greater advantage of ITD resources, including technical and project management expertise.
All of this goes to show that no matter what new technologies take the world by storm, deploying them involves certain fundamental practices that remain applicable today as they were yesterday. Clear governance, planning, consistent reporting, and policies and procedures should always be present, even in—or rather, especially in—the most advanced deployments. Just as change is becoming the new constant in technology, so too are these best practices in deploying the technologies themselves.
About the Authors
Marisa Lin is a Program Performance Auditor I for the City of San José.
Brittney Harvey is a Program Performance Auditor II for the City of San José, and volunteers as a member of ALGA’s Online Resources Committee.