The San Francisco Municipal Transit Agency (SFMTA) is responsible for overseeing the transit, streets and taxis and the relevant operations to those functions in the city of San Francisco. The Exams Unit of SFMTA's Human Resources Division (HR) conducts exams for city positions, including the SFMTA's Transit Operators - the folks that drive the City's many buses and rail cars. In 2018, the two SFMTA units that oversee Transit Operator hiring - the Exams Unit and Employee Services - engaged with the City Performance Lean Team (PPI's Ryan Hunter and Jenessa Rozier's previous team within the City Controller's office) to partner in an initiative to improve the administration process for the Transit Operator exams.
The Transit Operator exam process was extremely labor intensive, time-consuming, and potentially error-prone, which isn't a great place to be when the City is reliant on this process to hire new drivers quickly and efficiently to meet the demand. After performing a current state analysis to better understand the current processes, the team found that both SFMTA units were conducting duplicative work when screening the same candidates at different stages of the hiring process. Furthermore, the Exams Unit would have to conduct significant extra work following up with candidates who make mistakes in submitting required documents. A potential new Transit Operator must also go through Transit Operator training classes before starting work, and the delays in exam administration would often result in longer hiring times, which can contribute to unfilled, already scheduled, Transit Operator training classes. When training classes go unfilled, the new Transit Operators are not able to start their routes, meaning the SFMTA cannot meet their planned service hour targets which results in transit delays for residents.
Improvements
From August 2018 to January 2019, the SFMTA's Exam Unit implemented improvements to the exam administration process to mitigate these issues. The two solutions they implemented were:
Eliminating duplicative driving record reviews, and
Revamping candidate communication materials
Solution #1: Eliminate Duplicative Driving Record Review
According to policy, candidates must have a clean driving record with no moving violations for the past three years, as indicated on the candidate's DMV "H6" driving record. Prior to the improvement work, candidates were required to go to the DMV, obtain their H6 document, and bring it to their exam. The Exams Unit would then review each candidate's H6 record at the exam site while candidates were taking their exams, spending significant time following up with candidates who made mistakes on their H6 submissions. Once the eligible list, the list that is created once applicants pass the exam are scored and ranked by eligibility, the Employee Services Unit would conduct a second screening of the H6. If three months had lapsed between when the candidate got their initial H6 and when they were going to be hired, the SFMTA would ask candidates to go to the DMV and obtain another copy of their H6 for screening.
Both the Exams Unit and Employee Services Unit noticed the duplication in their screening practices, and identified the review of candidates' driving record to be a major source of extra work. In December 2018, the two divisions agreed to test a process to eliminate the Exam Unit's H6 review. Though this change may have caused the Employee Services Unit to review H6 driving records for a small number of ineligible candidates (including those with a moving violation within the last 3 years), the teams still predicted that the elimination of the H6 review during the exam would free up at least a week of time from the overall hiring process.
Solution #2: Revamped Candidate Communication Materials
The team also discovered that the exam notice did not clearly inform candidates of the steps required in getting a complete H6. As a result, the team created a new exam notice for Transit Operators (pictured right). The Exams Unit used this new notice for the November 2018 exam, and it appeared to reduce the number of mistakes made by candidates: the team saw a 10% decrease in the number of instances a candidate brought an incomplete H6 driving record to the exam in November 2018.
Other Improvement Ideas:
In addition to improving the examination notice and eliminating the driving record review from the Exam Unit's task list, the team brainstormed several other large-scale improvement ideas:
Eliminate the SFMTA driving exam: In addition to a video-based driving exam conducted by the SFMTA, candidates must also pass an extensive driving exam administered by the DMV called a Class-B permit test. The Exams Unit identified the Class-B permit test to be more exhaustive than the video exam, and wanted to align requirements more closely. Eliminating the SFMTA driving exam would create a more efficient process for candidates, and could eliminate hours of work to administer and score the internal driving exam.
Continuous testing for Transit Operator candidates: The City is always in need of new Transit Operators, so implementing continuous testing could help SFMTA maintain a robust eligibility list.
Results
From August 2018 to January 2019, the SFMTA's Exam Unit implemented the above improvements to the exam administration process to mitigate some of their issues. As part of their analysis, the team chose to measure two key performance indicators to determine if their improvement efforts worked:
Time to Hire, and
Number of Complete H6 Records
The team measured time to hire from the first day of the exam administration to the date on which the Exams Unit adopted the eligibility list, which provides an overall view of the time it takes the Exams Unit to conduct the exam administration. The second metric, number of complete H6 records submitted, speaks to the team's desire to eliminate mistakes in the process. During the March 2018 exam, approximately 125 out of 500 candidates (25%) submitted an H6 driving record that either included mistakes or was not complete. The below image shows the improvement in their metrics over the time that they initially monitored them, an indication that their improvement efforts made an impact on their process!
