Print Page | Contact Us | Sign In | Become a Member
Quarterly: Spring 2020 - Katie Houston

Auditing Workforce Development Programs and Adding Value Amidst the Confusion

By Katie Houston

Workforce development. A topic that virtually everyone has some exposure to, but also quite a squishy subject matter to evaluate. In 2017, our office endeavored to figure this one out. The objective of the audit was to determine if City workforce development contracting efforts are effective at achieving outcomes and preparing employees for jobs that match the needs of the Austin job market. Excuse me…what?

(Illustration by Olga Ovcharenko)

Step 1: Panic.

Like all good audits, this started out with heart palpitations and rolling hallucinations – some where I am preaching to audit committee members (obviously wearing a cape) with each councilmember hanging on every word, tearing up over my astute observations and poetic delivery, and then standing in ceremony to begin their inevitable slow clap. Other visions are filled with panic. Cameras. Florescent lighting. I’ve forgotten everything I did in this audit and I’m not entirely sure what “workforce development” really is but it’s the title of my project so I’m certain this presentation is going to be a complete disaster.

Step 2: Dive in. I’m at this for two hours and I know everything about this topic. When’s lunch?

The mania continues. After a brief trip to the Google machine, I’ve learned everything there is to know about this mythical “workforce development” and I’m ready to draft my report. Should I run for office one day? I’m killing it.

Step 3: Reality check. I don’t know everything, but I know a little bit and that’s all I need to really get started.

I’ve got a few good things going for me on this project. Solid team? Check. A relatively established City function that has been operating for a few years? Check. Reliable, quantifiable data about the needs of Austin employers? Meh… I can work with it. After a few team meetings, it’s clear to the group that at its core, workforce development is about training people in fields, or providing skills to workers, that will eventually be useful to Austin employers. Personally, I could stop here and feel accomplished, but our professional standards call for more so I’m willing to go the distance.

Step 4: You guessed it. Panic: Round II.

There are a lot of places we could have gone at this point. The focus of the audit is workforce development contracts. We could study how the contracts were written and then evaluate whether our workforce development providers were meeting the contract terms. We could back up a step and evaluate whether the City had a clear understanding of Austin labor needs to then establish effective workforce development contracts. We could also focus on the tail end of the process and evaluate whether previously vacant jobs in Austin had been filled with any of the individuals who attended the development programs provided by the City’s contractors. 

Like every fork in the road, I agonized over this. Is it more useful to evaluate an entire process at a fairly surface level? Or should we do a deep dive into one aspect of the process to really understand a particular piece, though it’s just a small part of a bigger process? Most importantly, what would be most useful to City management, Austin employers, and the program participants? And what is most likely to make Austin a better City? Ultimately, we decided to do a not-so-surface-level evaluation of the entire process. More specifically, this audit considered whether the right individuals were getting into development programs, their eventual job placement, and if Austin employers were served by these programs.

Was this painful? Yes. Would I do it again? Probably. The path of least resistance is indeed less resistant, but it may not add as much value as evaluating an audit topic in a more comprehensive manner. And this isn’t just the coffee talking now… here I am, petting my dog, writing this article that my mom and a few other people will someday read, and would I be doing this had our team not committed to conquering this complex topic so comprehensively? Put a pin in that.

Step 5: Establish a game plan that matches the project objective.

We set out to answer a few pivotal questions:

  1. Were eligible people getting into the development programs?
  2. Were participants completing the programs?
  3. Were the programs teaching the right curriculum to help workers meet Austin employers’ needs?
  4. Were the participants getting hired following the development program? And were they hired into the intended jobs?
  5. Perhaps most importantly, were the participants better off, meaning were they making more money after they went through this development program?

Step 6: Let’s do this.

What a whirlwind this was. We determined that the City studied the Austin market but didn’t effectively use that information to define what it wanted from its workforce development contracts. As a result, the contracts we established weren’t as effective as they could have been. We also found the data on services provided was all sorts of whacky. It was inconsistent, incomplete, and often missing altogether. Making sense of exactly how many people received workforce development services and the impact of those services was very difficult, to say the least.

Ultimately, we determined that some key outcomes were not achieved. True, graduation outcomes seemed favorable, meaning about 70% of people who entered the programs eventually graduated from them. However, among sampled graduates, the records we reviewed indicated that fewer than half of the participants were employed after finishing the program. Even when participants were employed, we could not determine if the participant’s employment related to the training they received. Furthermore, only 15% of the participants reviewed appeared to improve their income after completing a workforce development program. I think it’s fair to say that many participants probably benefited greatly from these programs even if they didn’t land jobs in their intended fields or made more money following the program, but we couldn’t overlook these critical outcomes.

These findings formed the basis for several recommendations for improvement. It wasn’t an easy sell, but management eventually agreed that enhancements were needed. Additionally, councilmembers agreed to halt additional funding to these programs until these enhancements were realized. Maybe they weren’t tearing up as we briefed them on the results of our work, and no, there wasn’t a slow clap at the end of the audit committee presentation, but this audit did result in the City making real changes to improve how we serve the public. And I still got to wear the cape (just kidding - some dreams never die).

Step 7: Reflection: sometimes a not-so-good thing has to happen for a good thing to come about.

Remember when we discussed the difficulty of scoping a big, scary audit to something meaningful and value-added? As I was working my way through my inner meltdown about how to audit this function, the function itself seemed to be going through its own struggle to find the best way to meet the needs of Austin’s employers. It took me a few years to see the parallel but three years later, I get it and critical enhancements have been made! The City has established new target markets to better align contracts to Austin employers’ needs. Also, just last week City staff briefed Council on these efforts and pitched the idea of consolidating funding to one external source with the goal of creating greater accountability for outcomes.

Me? I’m back at square one with my current audit topic and equally confused. What exactly is a “financing tool” per se and am I the only person who still doesn’t fully understand a bond rating? The answer is no. I think everyone with a pulse travels off into a blockbuster montage where fast cars are peeling around The Autobahn as Russian oligarchs launch grenades at the mere mention of Bond. But that, in and of itself, is victory. I now recognize just how confused we all are when we tackle important audit subjects and that puts us—at the very least—in good company. And that’s just the thing… tackling panic-inducing, complex audit topics is how we add value and grow as professionals. Maybe the more challenging it is, the bigger the payoff—for us and the communities we serve. Keep up the good work, comrades.

About the Author

Katie Houston, CPA, CIA, CFE, works as an Assistant City Auditor for Austin's Office of the City Auditor. Before joining the City in 2013, she worked in Texas State government. She received a Bachelor of Science degree in accounting from Westminster College and a Master of Business Administration degree from St. Edward's University. Katie’s first love is audit but in her free time, she enjoys all things Austin including: hiking, camping, BBQ’ing, and gardening with her pests pets, Willie and Waylon. Her greatest aspiration is winning “Yard of the Month.”