Target Setting

Component 2

Overview

To provide assistance transportation agencies with target setting, this chapter begins by defining target setting within the TPM framework, describing the inter-relationship between target setting and the other nine TPM components, presenting definitions for commonly used target setting terminology, and providing the legislative context for target setting. The bulk of the chapter however, is dedicated to outlining key target setting implementation steps with example applications. The chapter concludes with “keys to success,” a select list of resources and a customizable action plan.




Transportation target-setting is a data-driven, collaborative process that defines what an agency wants to achieve within a specific time frame.

The target setting process takes the goals, objectives, and performance measures of the Strategic Framework (Component 01) and defines success. Targets are used to assess progress toward achieving strategic goals, guide planning efforts, inform programmatic decisions and adjustments, and communicate with stakeholders. Targets make the link between investment decisions and performance expectations transparent across all stakeholders. In short, the process of setting targets completes the foundation (along with the Strategic Framework – Component 01) from which strategic decision-making is launched.

The target setting process is vital to the implementation of Transportation Performance Management (TPM) and offers unique and powerful benefits to an agency by:

  • Creating a crucible for testing what you have in place, particularly data quality and measurement definitions;
  • Driving a conversation about where you are today and how to get to where you want to be in the future;
  • Focusing on the connection between actions and results;
  • Guiding the prioritization and allocation of resources;
  • Enabling assessment of strategy effectiveness;
  • Forming a powerful argument for additional or changes in investments; and
  • Managing expectations by clarifying what outcomes are desired.

Inside an agency, the target setting process is intertwined with the tenets of TPM: connecting employee actions to results, motivating and focusing staff, increasing accountability, guiding the allocation of resources, and tracking the efficacy of various strategies.

Target Setting

The use of baseline data, information on possible strategies, funding constraints and forecasting tools to collaboratively set targets.

The definition for target setting is: The use of baseline data, information on possible strategies, funding constraints and forecasting tools to collaboratively set targets.

Target setting is broken down into two complimentary subcomponents:

  • Technical Methodology: Implementation of a data-drive approach for calculating a baseline and performance trend.
  • Business Process: Establishment of an intra-agency process including internal coordination and collaboration to set and modify performance targets.

The technical methodology relates to the compilation and analysis of historical, current and future performance data to guide target setting, while the business process builds internal collaboration, defines roles, and specifies the steps necessary to ensure a strong internal target setting approach is in place. The technical methodology is where “the number” is calculated upon which the target will be set. In order to do this, a baseline must be developed and trend lines examined. Alongside the solid technical approach, agencies must establish and sustain an internal business process to properly establish and refine a performance target. This includes defining the agency’s key players and their roles and responsibilities, planning for coordination across performance areas, and leveraging external collaboration. The technical and business subcomponents must be balanced in order to establish a robust target that will guide future decision-making.

Target setting should not focus on a single target value for a performance measure, but on achieving improved performance over time. The value of performance management is found in better decision-making, not target achievement.

AASHTO SCOPM Task Force Findings on MAP-21 Performance Measure Target-Setting March 2013

Because target setting relies on a carefully investigated baseline and the development of future scenarios, a transparent target setting process creates an open dialogue about specific outcomes the agency wants to achieve and articulates the connection between actions and results. Actions required to achieve established targets are clarified; later, the effects of past actions is revisited. Targets also provide the justification necessary to argue for additional resources and more or better quality data. As a transparent target setting percolates through an organization as a key part of TPM, the relationship between each employee’s day-to-day activities and the desired results becomes more real.


Inside an agency, the target setting process is intertwined with the tenets of TPM: connecting employee actions to results, motivating and focusing staff, increasing accountability, guiding the allocation of resources, and tracking the efficacy of various strategies.

Target Setting: Baseline Forecasting: Data, Analysis, Tools; Business Processes, Roles, Purpose, Collaboration

Figure 1. Target Setting Subcomponents

The ten TPM components are interconnected with each other and in many cases dependent on each other.

The linkages between target setting and each of the TPM components is depicted in Table 5.1. On the left, a summary definition is a reminder of what each component entails. On the right, see how the components are interrelated in many ways; this serves to explain their inter-dependencies and complementary functions. The process of setting targets is deeply woven throughout the overall TPM Framework.

Table 5-1 Target Setting Relationship to TPM Components

Component Summary Definition Relationship to Target Setting
Strategic Framework A well-defined and aligned strategic framework for performance management. Targets turn the goals, objectives and measures identified in the strategic framework into statements of success.
Performance Based Planning The use of a strategic framework to drive the development and documentation of agency strategies and priorities in the long-range transportation plan (LRTP) and mid-range plans. Targets define the results the strategies and priorities in these plans are striving to achieve.
Performance Based Programming Allocation of resources to projects to achieve strategic goals, objectives and performance targets. Clear linkages established between investments made and their expected performance outputs and outcomes. Targets guide the prioritization of the projects included in the S/TIP and agency budgets. Targets are needed to track progress towards expected performance outcomes.
Monitoring and Adjustment Processes to monitor and assess actions taken and outcomes achieved. Establishes a feedback loop to adjust programming, planning, and benchmarking/target setting decisions. Provides key insight into the efficacy of investments. Targets provide the “stake in the ground” around which to evaluate the effectiveness of implemented strategies, progress towards goals, identification of unforeseen issues and potential programmatic adjustments.
Reporting and Communication Products, techniques and processes to communicate performance information to different audiences for maximum impact. Targets are integral to an agency’s performance language and are used to illustrate progress made, challenges incurred and next steps related to the strategic goals.
Organization and Culture Institutionalization of a performance management culture within the organization, as evidenced by leadership support, employee buy-in, and embedded organizational structures and processes that support TPM. Targets clearly communicate to all employees what the agency is trying to achieve and where they should focus.
External Collaboration Established processes to engage and collaborate with agency partners and stakeholders on planning/ visioning, target setting, programming, data sharing, and reporting. The target-setting process provide an ideal opportunity to collaborate with outside partners in order to gain their support for agency’s efforts to make progress and find common areas of interest.
Data Usability and Analysis Existence of useful and valuable data sets and analysis capabilities, provided in usable, convenient forms to support TPM. The usability of data to develop baseline trends and prepare forecasts plays a significant role in how the target setting process evolves.
Data Management Established processes to ensure data quality and accessibility, and to maximize efficiency of data acquisition and integration for performance management. The availability of data and the quality of data are the foundation of target setting.

Common terms used in performance management can often be interchanged with each other resulting in some confusion.

To clarify the usage of TPM terminology, Table 5-2 presents the definitions for the target setting terms used in this Guidebook.

Table 5-2 Target Setting: Defining Common TPM Terminology

Common terms What do they really mean? Example
Goal A broad statement of a desired end condition or outcome; a short piece of the agency’s vision. A safe transportation system.
Objective A quantifiable statement that provides additional specificity for a goal. Reduce the number of motor vehicle fatalities.
Measure A quantifiable indicator used to track the objective. Fatality rate per 100 million vehicle miles traveled.
Baseline A trend distilled from historical data and past performance; from which implementation begins, improvement is judged, or comparison is made. The graph depicting annual fatality rate and five-year average fatality rate from 2000 to 2014.
Baseline Measure The current level of performance. 2014 fatality rate = 0.83 per 100 million miles of travel.
Target Level of performance that is desired to be achieved within a specific time frame. 10% reduction in fatality rate by 2020.
Benchmark A comparison of two numbers, often historical data with current numbers or one agency’s results against peer’s. Assessing an agency’s fatality rate by comparing it to that of a peer agency; or to historic fatality rates.

Note: A full list of common TPM terminology and definitions is included in the Appendix X: Glossary


In addition to being a cornerstone component in TPM, MAP-21 includes the following requirements regarding target setting.

State Targets

Within one year of the DOT final rule on performance measures, MAP-21 requires states to set performance targets in support of those measures. States may set different performance targets for urbanized and rural areas. 1   To ensure consistency each State must, to the maximum extent practicable:

  • Coordinate with an MPO when setting performance targets for the area represented by that MPO; and
  • Coordinate with public transportation providers when setting performance targets in an urbanized area not represented by an MPO. 2  

MPO targets

MAP-21 requires that within 180 days of States or providers of public transportation setting performance targets, MPOs must set performance targets in relation to the performance measures (where applicable). To ensure consistency, each MPO must, to the maximum extent practicable, coordinate with the relevant State and public transportation providers when setting performance targets. 3  

Plans requiring targets

MAP-21 requires the following plans to include state targets (and/or MPO targets, as appropriate):

  • Metropolitan transportation plans 4
  • Metropolitan Transportation Improvement Program (TIP) 5
  • Statewide Transportation Improvement Program (STIP) 6
  • State asset management plans under the National Highway Performance Program (NHPP) 7
  • State performance plans under the Congestion Mitigation and Air Quality Improvement program 8
  • Additionally, State and MPO targets should be included in Statewide transportation plans 9

Public Transportation: Asset Management

MAP-21 requires transit agencies receiving Federal financial assistance under Title 49, Chapter 53 to “not later than 3 months after the date on which the Secretary issues a final rule under paragraph (1), and each fiscal year thereafter, each recipient of Federal financial assistance under this chapter shall establish performance targets in relation to the performance measures established by the Secretary.” 10

Public Transportation: Safety

MMAP-21 requires transit agencies receiving Federal financial assistance under Title 49, Chapter 53 to within one year of the final transit safety rule to establish a comprehensive agency safety plan that includes, at a minimum, performance targets based on the safety performance criteria and state of good repair standards established by the Secretary. 11



Implementation Steps

The following section outlines steps agencies can follow to implement a sustainable technical methodology to target setting. The implementation steps for a technical methodology to target setting that will be further explored in this chapter are presented below.

  1. Establish a baseline
  2. Analyze historical trends
  3. Identify influencing factors (internal and external)
  4. Define target parameters
  5. Forecast future performance
  6. Document technical methodology


Establish a Baseline

Description

The most important first step in the target setting process is to assemble data to develop a baseline for the performance measures established under the Strategic Framework (Component 1). The baseline is a trend that illustrates past performance, and serves as a jumping off point from which implementation and progress, begins. Developing a baseline means digging into what data the agency has available and how it is organized. For new measures, agencies may be limited to a baseline value or one data point, but this can be the starting point from which a trend line can be assembled over time. Items to keep in mind as a baseline is developed:

Characteristics of quality data

  • Assemble existing data
  • Determine data source and ownership
  • Assess data quality
  • Identify data gaps
  • Outline ways to close data gaps
  • Use agreed upon data to establish baseline

Examples

Below are examples of different levels of data available to support assembling a baseline.

Federal Systems


  • Highway Performance Monitoring System (HPMS)
  • Fatality Analysis Reporting System (FARS)
  • National Bridge Inventory (NBI)
  • National Performance Management Research Data Set (NPMRDS)
  • Census data
  • etc.

Management Systems


  • Infrastructure (bridges, pavements, and other assets)
  • Safety (crashes, injuries, seat belts, non-motorized, etc.)
  • Mobility (traffic, real-time, congestion, transit, etc.)
  • Economic (freight movement, tourism, jobs, etc.)
  • Environment (emissions, storm water, etc.)

Individual


  • Access databases
  • Spreadsheets
  • Shapefiles
  • Other

Commercial Sources


  • Travel information
  • Freight information
  • Weather
  • Geospatial data

Linkages to Other TPM Components



Target Setting: Baseline Forecasting: Data, Analysis, Tools; Business Processes, Roles, Purpose, Collaboration

Figure 2. Characteristics of Quality Data

Description

The underlining current of this step is “look back to where you have been, for a clue to where you are going.” In other words, understanding past results can provide guidance on setting future targets. The main focus of this implementation step is answering the question: why did we get the performance results we did? Items to keep in mind as trend analysis is being conducted:

  • Confirm measure calculations are consistent over time
  • Identify any repeating patterns (e.g., economic upturn and higher congestion)
  • Investigate atypical variations in trend (e.g., severe winter weather)
  • Locate “shifts” in trend line due to policy changes (e.g., lifting of motorcycle helmet law)

Examples

The importance of confirming that performance measure calculations are consistent over time was demonstrated at the Washington Metropolitan Area Transportation Authority (WMATA). In December 2013, WMATA reported to their Board of Directors that bus on-time performance had improved to over 80%, a historically un-chronicled level. However, WMATA has to retract this statement when staff discovered that the favorable performance was due to new fleet technology failing to capture all the buses arriving early, thus compromising Bus On-Time Performance results for the latter portion of CY2013.


Example image: Square

Note: Measurement calculation error, NOT a performance improvement

Linkages to Other TPM Components

Identify influencing factors (internal and external)

Description

In order to understand past and future performance it is essential to identify both internal and external influencing factors. Internal factors include issues within an organization that can potentially be addressed over time, since they are under the control of the agency, and in some cases may even be leveraged to ensure that the target is met. External factors, by contract, are outside agency control, but still effect performance outcomes. The identification of internal and external factors is linked to the “analyze historical trends” step where the following question was asked, “why did performance change?” The answer to this question will undoubtedly uncover several factors that will continue to affect performance results. For example, the economy has a major influence on transportation performance. When the economy is strong, the roads are more congested, the pavements deteriorate due to greater truck traffic on highways, and freight flows increase. When population in a region increases, so do transportation demands. The recent oil boom in several states has resulted in changing traffic quantity and patterns in a manner that has been difficult to predict. On the safety front, could we have imagined how the improvements in vehicle characteristics over the past decade would impact safety-related fatalities and serious injuries?

Both internal and external factors must be acknowledged in order to fully understand the context of the target, to try to anticipate any impacts on performance, and to begin to address these issues as part of the TPM business approach.

In short, understanding factors helps understand limitations. With a better understanding the performance situation, targets can be more accurate and in turn more helpful. Table 5-3 contains a list of internal and external factors to consider in target setting.

Table 5-3 Target Setting: Influencing Factors

Internal External
Funding Economy
Staffing Constraints Weather
Data availability & quality Politics/Legislative requirements
Leadership Population growth
Capital project commitments Demographic shifts
Planned operational activities Vehicle characteristics
Cultural barriers Zones of disadvantages populations
Agency priorities Vehicle characteristics
Agency jurisdiction Modal shares
Senior management directives Gas prices
Policy directives (e.g., zero fatalities) Land use characteristics
Cross performance area tradeoffs Driver behavior
Collaboration across agency Traffic

In addition to building a comprehensive list of all potential factors that might influence the performance results and subsequently the target, agencies should also make an assessment of which factors will likely have the largest effect on performance results and which factors they can have the largest amount of control over. However, the quantification (to the extent possible and practical) of the effect these factors have on results could be postponed until the “forecasting future performance” implementation step. As agencies complete additional cycles of target setting, the ability to identify and understand the effect of various internal and external factors will increase. In essence this step is about agencies managing risk. Items to keep in mind as an agency identified influencing factors include:

  • Identify all potential internal and external influencing factors
  • Categorize the factors by the extent of control an agency has over the results (e.g., “extensive,” “moderate,” or “limited” control)
  • Assess potential factors by the degree of influence on performance results
  • Determine which factors should be included in the current target setting effort
  • Re-visit the list of influencing factors to drop or add factors, change assessment of agency control and/or degree of influence


Examples

Degree of Influence over Factors: Virginia Performs
(www.vaperforms.virginia.gov)
The degree of influence and agency has over factors that affect performance outcomes will vary by factor. “Virginia Performs,” an on-line publically available tracking of the state’s progress toward seven strategic goals, includes an assessment if the state has “significant” or “limited” control of the results. For example, Virginia has determined that it has “limited” control over land use related factors including population density and land development patterns.


Example image: Square

Internal Factors: Escalator Availability
A key internal factor to consider when setting targets is planned capital projects. Take for example, escalator availability at a transit agency. From the customer perspective, an escalator out of service is an inconvenience and negatively effects the travel time and the overall transit experience. Customers typically do not distinguish between an escalator being unavailable due to scheduled maintenance or due to an unforeseen issue (e.g., damage to handrail). Therefore, some transit agencies elect to include all escalator downtime in measuring the percentage of time units are available. In this case, when setting an availability target, it is vital to include the hours necessary to conduct required inspections, preventive maintenance and modernization activities that must be conducted during operating hours. Table 5-5 demonstrates that given required maintenance activities, the BEST availability is 95%. Looking at past performance trends, some unscheduled maintenance will likely occur and should be reflected in the final availability target. Granted, efforts can be made to improve the speed and quality of both scheduled and unscheduled maintenance activities, but laying out the effect of planned capital activities in an easy to digest manner can greatly assist the target setting discussion.

Table 5-5 Effect of Scheduled and Unscheduled Maintenance on Escalator Availability

Example image: Vertical

External Factors: Public Policy
In 2012, the Michigan Legislature joined 30 other states by enacting Public Act 98 to repeal the law requiring helmet usage with headlines, “Bill to let motorcyclist decide helmet use.” Michigan’s helmet-use law was originally put into place in 1967 to comply with a since lifted requirement to receive federal funds. One study by the University of Michigan Transportation Research Institute concluded that reduced helmet use accounted for approximately 24 more fatalities and 71 more serious injuries a year in Michigan. The study looked at crash data from 2009 through 2013, and calculated that the risk of fatality is 2.8 times higher for riders not wearing a helmet, while the risk of serious injury is 1.4 times higher, largely echoing studies in other states. The study also demonstrated that the fatality rate rose after the repeal of the helmet law to the highest level in 5 years. The effect of this public policy change should be considered when setting safety related targets. Michigan DOT’s Office of Highway Safety Planning outlined numerous strategies to improve the safety of the driving public including motorcyclist in its annual Highway Safety Plan.


Linkages to Other TPM Components


Define target parameters

Description

Targets can take on many different formats (e.g., percentage, directional, average value), different time horizon (e.g., daily to 25 year targets), different scope (e.g., regional, statewide, urban/rural). A key step in the technical methodology to target setting is determining these target parameters. For example, in many cases, it may take a number of years for a significant investment to produce noticeable improvements in performance; annual targets may create a sense that progress is not occurring, when it is occurring on a different time frame. In addition, the target parameters should reflect the purpose of the target (see Business Process Subcomponent 2.2). For example, agencies may elect to establish a set of targets with different time horizons to fill different needs (e.g., long range to clarify desired policy objectives, mid-range to support management documents like TAMPs and short range to guide annual budgetary decisions,). Items to keep in mind as target parameters are being evaluated:

  • What will resonate with the target audience?
  • What parameters will reflect the behavior of the performance measure?
  • Consider the cost of data collection and presentation (e.g., more frequent may equate to more costly)
  • Reflect existing reporting requirements (e.g., NHS bridges)

Examples

Target Format: How the change in performance is portrayed. Below is a list of the ways to communicate a target:
  • Number: The target is aiming for a specific number
    Example: Achieve 300 or fewer accidents on state roadways this year.
  • Directional: The target is aiming for an increase or a decrease within a measure.
    Example: Reduce the number of accidents on state roadways annually.
  • % or Rate: The target aims for a certain percent decrease or to impact a certain number of users.
    Example: Achieve a 20% decrease in number of accidents over the next 5 years.
    Or: Achieve a rate of 1 in 5 roadway users employing electronic tolling.
  • Absolute: Typically aspirational, the target can be “none” or “all”
    Example: Implement safety measures on all roadways.
  • Tiered Targets: Targets incorporating by their definition a range of outcomes, allowing flexibility in accommodating various expected influencing factors
Geography / scope:
Boundaries and filters applied to the performance area to set the extent of the target. Consider whether the scope is urban, rural, regional, corridor, one or several modes, NHS/non-NHS, etc. Keep in mind that this part of the scope is directly tied to the external factors discussed above, and that a wider geographic scope likely means more external factors to consider. The scope must take the literal realm of responsibility into account and resist overextending. Some considerations include:
  • Area of influence: Can the decisions you make impact the scope you set?
  • Federal and State requirements may determine scope
  • What is the interest of external stakeholders?
  • Does the definition of scope encourage friendly competition (e.g., across districts) or lead to incorrect conclusions?
  • Can related scopes that you set be aggregated? (e.g., interstate, non-interstate NHS, NHS, and non-NHS—this example is not a good roll up because the first two items are a subset of the NHS.)
Prior to a 2014 peer exchange, State DOTs were asked if they planned to set different performance targets for rural and urbanized areas – or for two different scopes. The majority of the respondents indicated that they “were not sure.” For the agencies that indicated they would set different targets, the reasons given included different acceptable thresholds for urban versus rural delay and different infrastructure condition needs by roadway type.

Example image: Vertical

Time horizon:
Duration of time that will be the basis for reaching the target. This parameter should be carefully considered and chosen as is appropriate to the type of target and the feasibility of attaining a specific outcome. Note: the frequency of reporting performance results is different than the target time horizon. For example, an agency may have an annual target for bus on-time performance but the performance data are assessed on a daily basis.

Table 5-6 Target Setting: Horizon Options

Type Short-Range Mid-Range Long-Range
Years Weekly, Monthly, Annual to < 5 years > 5 years; < 10 years > 10
Usage Business Plans Asset Management Plans Long Range Transportation Plans
Most Useful For: Areas where agency has more direct control (e.g., asset condition) Areas where change occurs very slowly or a long lag time exists between investments and results


Linkages to Other TPM Components


Forecast future performance

Description

Given that targets are set out in the future, it will be necessary to attempt some forecasting of future results. As discussed in the “identify influencing factors” steps, there are both internal and external factors that should be considered in any performance forecasting. Forecasting future performance needs to account for factors that will affect results (internal and external). A range of tools, models and methods exist to predict future performance. As a result of this step, an agency has a clearer picture of what is feasible to achieve. However, the purpose of this step is NOT to set targets, but to identify a range of performance targets to feed into the target setting business process (Subcomponent 2.2). Items to keep in mind as forecasts are being developed:

  • Document assumptions
  • List what factors are considered in forecasts
  • Develop future scenarios based on different funding levels


Examples

Below are examples of analytical tools and methods that agencies have used to forecast and communicate performance results to support the identification of targets.

Bridge

  • Bridge Management software (BrM), formerly Pontis
  • Deterioration models to predict future bridge condition based on past data and bridge age
  • Algorithms to process NBI and Element data to set targets
  • Forecasting tool that combines historic performance and historical funding level then predicts expected condition using expected funding target for the bridge program
  • Full life cycle (75 year) analysis of bridge condition combined with revenue projects and construction inflations used to maximize the investments impact on the bridge assets
  • A deficit report based upon current investment and condition compared with future investment

Pavement

  • Pavement Management System (PMS): model future pavement conditions on a set of criteria such as traffic levels, asset type, age of pavement, and resource constraints
  • GIS for data analysis and visualization


Example image: Square

Safety

  • Linear regression, best-fit regression analysis, non-linear regression, time-series analysis
  • Below is an example of a safety trend line that has been set, which shows the amount of fatality reduction required per year within the requested time frame in order to achieve the target: 12 per year. It is helpful to show the “tail” to show the high and low of what is probable to achieve. This shows the temporal difference necessitated by the target rather than just the arithmetic of number or percent of change.


Example image: Square

System Performance

  • Travel demand models
  • Highway Capacity Manual
  • System performance management systems
  • Model estimating the economic benefits infrastructure improvements (e.g., Highway Economic Requirement System (HERS), Transportation Economic Development Impact System (TREDIS))
  • National Emissions Inventory (NEI), Air Quality System (AQS) and Mobile 6.2

The figure below depicts the future levels of congestions the North Central Texas Council of Governments (NCTCOG), the MPO for the Dallas Fort Worth area, forecasted using its travel demand model. It is important to note that the predicted rise in congestion is based on the assumption that all the strategies outlined in their Mobility 2030: The Metropolitan Transportation Plan were implemented. Having the understanding that congestion will increase despite a plethora of projects, programs and policies is critical information to have when setting targets.



Example image: Square

Funding Scenarios: Rhode Island DOT

As one of the most often cited influencing factors on performance results, it will be important to develop forecast based on different funding scenarios. In the example below, Rhode Island Department of Transportation (RiDOT) shows bridge condition results based on different funding projections. The trend lines show that future performance results from the baseline are actually moving further away from the target performance. Portraying several different data lines on this chart shows the constraints (annual investment) as it relates to the percentage of good condition, and again drawing that out over time highlights the difference in achieving a target or not. This type of representation is very compelling for why a target cannot be met at current funding levels.



Example image: Square


Linkages to Other TPM Components


Document technical methodology

Description

The data driven methodology for setting targets has been developed and now must be documented. The completion of this step means that an agency setting a target has a good understanding of their current status regarding data availability and capabilities, influencing factors, and tools to forecast results. This will be a rulebook and record of how the target methodology was conducted and why this approach was used. Documentation may not sound like the most exciting aspect of target setting, but it will ensure the technical methodology is replicable from cycle to cycle, can withstand staff turnover and establishes an air of transparency around target setting. Material from this document can also be used to communicate with internal and external stakeholders about the agency’s target setting technical methodology.

As an agency goes through multiple cycles of target setting, update the documentation. For example, agencies may enhance their abilities to identify and understand the effect external factors have on performance results. Similarly, as various strategies are applied to improve performance, a better understanding of the linkage between actions and results can lead to improved target setting. Target setting, like many TPM components, is an iterative process so it will be critical to document steps that were taken and subsequent adjustments made to an agency’s technical methodology. Topics to address in the technical methodology documentation include:

  • Roles and responsibilities of involved staff
  • Outline of business process milestones and schedule
  • Process flow map
  • Recommended adjustments for future target setting cycles
  • Specific issues related to each implementation step (see Table 5.7)

Table 5-7 Topics to Address in Technical Methodology Documentation

Technical Methodology Step Topics to Document
Establish a baseline
  • Data source and owner
  • Data gaps
Analyze historical trends
  • Measure calculations and any changes over time
  • Explanation for “why performance changed”
Identify influencing factors
  • Define influencing factors used
  • Categorization of agency influence
  • Identify factors to include in next cycle and why
Define target parameters
  • Target format, geography/scope and time horizon
Forecast future performance
  • Assumptions
  • Tools and methods used
  • Define scenarios parameters and conclusions obtained


Examples

Documentation of Technical Methodology: PennDOT

As part of the Pennsylvania Department of Transportation’s (PennDOT’s) performance management approach, the agency developed a set of “Production User Manuals” to document the definition, data source, calculations, reporting cycle, and purpose for each key performance measure (see Figure below). These documents also describe how staff can use the Highway Administration Performance Dashboard (HAPD) to access raw data, view results, generate reports, and enter comments. Internal and external stakeholders responded positively to this transparent documentation of the data and technical methodology behind the targets posted on the HAPD scorecard.

Process Flow Mapping: Example

There are many approaches to creating a process flow map, but some rules of thumb are to highlight the distinct steps taken, the flow of the steps and any linkages between the steps.



Example image: Square



Example image: Square



Example image: Square


Linkages to Other TPM Components


















































Under MAP-21 agencies will be allowed to set their individual targets and determine whether to set separate targets for urban vs. rural areas under an agency’s jurisdiction.


















































“Agencies will need to calculate a baseline of performance and forecast expected performance based on that baseline in order to set targets. For many agencies, this will require an understanding of tools that does not currently exist.”
Source: Target Setting Peer Exchange (2014)


















































































































“The Production User Manuals pulled back the curtain to the technical methodology behind our performance scorecard providing improved clarity and transparency to previously often assumed and frequently misunderstood processes. As a result people’s trust in the data and published results improved because everyone knew where the numbers came from and how they were calculated.”
- Jim Ritzman, Deputy Secretary for Planning (PennDOT)

On the process side, staff and stakeholders need to be informed and organized so that they align positively to finalize the selection of targets. The targets options identified through the technical methodology should feed into a target setting business process. The business process answers the “who” and the “how” targets will be set. The following section outlines steps agencies can follow to implement a sustainable business process to target setting.

  1. Assign internal stakeholder roles and responsibilities
  2. Clarify purpose of the target
  3. Gather information through benchmarking
  4. Reflect external stakeholder interests
  5. Document the business process

Assign Internal Stakeholder Roles and Responsibilities

Description

A key first step in the target setting process is the identification of who will be involved in picking the final target and what role each person will fulfill. This would answer questions related to who at each agency level is responsible for setting targets as well has how to keep all affected offices engaged. Without clarity about who is accountable for what throughout the process especially, who has decision-making control, the business process will likely be ineffective. Key roles to determine include:

  • Process leadership: Who will lead, coordinate and marshal the target setting business process. If target setting is new to an agency, it will be important to identify a strong process facilitator.
  • Input providers: Practitioners invited to the table for the duration of the process, who drive the conversation and make recommendations and suggestions to decision makers. Can include external stakeholders but predominately internal agency staff.
  • Feedback contributors: Group of individuals whose opinion and signoff is helpful, but who, for sake of expediency and organization, do not need to be at the table as part of the input group.
  • Trackers: The tracking group may be made up of individuals also serving within capacities on the list above; this group is responsible for collecting and analyzing data used to establish and monitor performance targets.
  • Decision makers: These may vary at different stages of the process, but it should be clear who has the final decision on what the final target will be.

All stakeholders in these roles need to understand how the baseline and target have been calculated, how they will be used, and how they will be communicated internally. They also need to understand who is in charge of decision-making and accountability, to ensure a clear chain of command and eliminate confusion and dead-ends. Items to keep in mind as a baseline is developed:

  • Identify a champion to lead the business process
  • Input from technical methodology should serve as the business process foundation (e.g., baseline, historical analysis, forecasts)
  • Participants need to represent interest across agency silos (including staff who express resistance to target setting)
  • Link to existing processes as much as possible (e.g., LRTP, budget development)
  • Conduct an open dialogue about how targets will be used internally
  • Clearly identify how targets will be finalized

Examples

Internal Roles

Internally, roles and responsibilities of key players need to be defined and the coordination needs across performance areas must be established. This entails acknowledging that target choices in one performance area may affect performance in another area. For example, lowering pavement condition targets could have an impact on safety performance, or increasing mobility targets (increasing average speed for example) could impact safety. While the chance of success is impacted detrimentally if one or more key people are missing at the table, bear in mind that there is a tradeoff to inviting too many voices to the conversation: The more people involved, the longer the process of target setting will be and the more cumbersome your business process will likely be. A rule of thumb is the formula 2x people = time2. Rather than issuing a blanket invitation to all potential stakeholders, their individual roles must be strategized and discussed at the very beginning of the process, including the key decision of who is inviting to engage vs. who is informed after the fact.

The word cloud below is a visual representation of State DOT responses to a survey distributed prior to the AASHTO Standing Committee on Performance Management (SCOPM) MAP-21 Target-Setting Workshop held in June 2013. The specific question asked was, “who within your agency would participate in the target setting process for bridge condition, pavement condition, safety, freight, system performance and CMAQ.” Besides the observation that the range of practioners involved in the target setting process is wide, there appears to be the needs to bring together the system performance area experts and planning staff.

Once the internal stakeholders have been identified and the involve vs. inform roles clarified, it should be made clear to all how the technical methodology (Subcomponent 2.2) produced potential targets for consideration. Only if all stakeholders are using the same set of assumptions and the same set of constraints and factors can a solid process occur. Transportation performance management goes beyond even the typical amount of coordination and collaboration of transportation planning, requiring different practice areas to consider how targets in each relate to each other, what tradeoffs there may be, and how to support multiple goals simultaneously. As a result the group involved in the target setting business process should pull from many areas of the organization.

Linkages to Other TPM Components

Clarify purpose of the target

Description

Targets can take on many different roles within an agency from motivating staff to managing external expectations. A key step in the target setting business process is obtaining an agreed upon and well understood purpose for the target by answering two questions: (1) Who is the intended end user or audience for the target? and (2) what the type of target that best fits that audience?

(1) Target Audience: Who is the end user or audience of the target? Focusing a target depending on different end users can reflect different purposes, some of which can overlap.

(2) Target type: Depending on the purpose and audience of the target, an agency must strategize what target type is most relevant. The main types of targets to consider are:

  • Aspirational: Vision driven, motivational, not necessarily attainable. Designed to set a vision and a direction for people to rally around.
    Example: Zero Traffic Fatalities
  • Realistic: A bit of a stretch, but it’s attainable given baseline data and forecasts.
    Example: Reduction of fatalities and injuries by a certain % within a certain timeframe.
  • Safe: On track to attain. Does not require greater focus or dramatic change/ investment. Heavy reliance on baseline data.
    Example: Reducing fatalities to a certain number that is clearly attainable based on historical data.

Together the target audience and type define the purpose for the target. In other words, the target audience and type are linked. For example, if an agency has recently experience lower performance results and was interested in motivating staff, it would be counterproductive even demoralizing to set an unattainable target. Instead, the agency may consider a safe or realistic target to boost agency confidence. In this situation a “safe” target may also help build trust with external partners that the agency can deliver on a “promise.” The purpose of the target should also be reflected in the target parameters defined under the Technical Methodology Subcomponent 2.1. Items to keep in mind as the purpose of each target is being determined:

  • Who is target audience?
  • What legislative mandates exist?
  • How will results be communicated if target is not attained, met or exceeded?
  • What type of target should be used? (e.g., aspirational, realistic, safe)

Examples

Target Type: Aspirational, Realistic, Safe

The three examples below demonstrate target types around a similar topic: safety and decreasing traffic fatalities.

Aspirational

The Toward Zero Deaths program from USDOT is a prime example of an aspirational goal. Due to many reasons, actually achieving zero deaths may not be possible; but the use of the zero target is strong and concrete, confirming the direction of “working toward no fatalities across all modes of travel” within their Strategic Plan. The vision of Toward Zero Deaths sums up a strong approach towards safety. It is understood that, as an aspirational goal, it is a high-level vision that is customized and broken into discrete parts such as the “4 Es” (education, enforcement, engineering, and emergency services) with their own further targets. See http://safety.fhwa.dot.gov/tzd for more.

Realistic

An example from Minnesota DOT illustrates a “realistic” version of a target within the same safety target area. Instead of a strong visionary statement, they have determined “the number” – a specific target of 300 or fewer traffic fatalities by 2020. This number was determined and considered realistic by examining previous years’ data, determining a trend line, and setting the target number. However, note that Minnesota refers back to USDOT’s brand and aspirational vision of Toward Zero Deaths. See http://www.minnesotatzd.org for more.

Safe

The third example from Maryland DOT’s version of “Toward Zero Deaths” is a “safe” target. As was done in Minnesota, Maryland followed the process to examine data, set a baseline, determine at trend line, and then set a target number. However, they decided to shorten the timeframe (2015 as opposed to Minnesota’s 2020) and aim for fewer than 475 fatalities, a decrease that is well within the historic trend line of their data. See http://towardzerodeathsmd.com for more.

During a 2014 survey, the majority of State DOT respondents indicated that they typically set realistic versus aspirational targets, but 26% of respondents indicated both types of targets were used. These findings illustrated in the chart below reinforce the importance of having an open dialogue about who the audience will be for the targets and obtaining an agreed upon and well understood purpose for the target.

Linkages to Other TPM Components

Gather information through benchmarking

Description

Simply stated, benchmarking is a method to improve performance results by looking at (a) one’s own historical patterns, (b) peer agency results or (c) “best in class” practices. In the realm of target setting, all three types of benchmarking can provide valuable insights to the final selection of a target value. The “comparing against yourself” approach, or analyze historical trends, was covered under the previous section (Technical Methodology Subcomponent 2.1). Gathering target information from peer agencies can clarify what are regional and national trends in specific performance areas, create a context for a target, and help explain a proposed target value to external stakeholders. However, to properly bring external target values into an internal agency’s target setting process will require accurately identifying peer agencies (or clearly explaining the differences), confirming similar data sources were used and ensuring consistent measure definitions were applied.

Items to keep in mind as agencies gather benchmarking information for use in target setting:

  • Identify peer agencies based on similar attributes (e.g., infrastructure size, population, weather, topography, economy)
  • Beware of benchmarking information being used to incorrectly compare agencies
  • Think through how the benchmark information will be used because the results may not fit the need

Examples

Benchmarking with Peer Agencies

In 2004, several state DOT CEO requested a multi-year research effort to look into the feasibility of sharing performance measurement results across agencies. As a result, ten NCHRP projects were developed (NCHRP 20-27 (37) Reports A-L) that developed peer groupings, compiled detailed performance data, and calculated commonly defined measures. The research series demonstrated that sharing information between agencies can provide useful insights into target setting. Take for example, pavement condition. The chart below illustrates how pavement condition has similar patterns in each of the four regions suggesting some factors exists (e.g., weather conditions) that effect performance results. Therefore, when looking to gather peer information for target setting, agencies should reach out to peers in their geographic areas.

Benchmarking with “Best in Class”: Missouri DOT

The Missouri DOT (MoDOT) considers customer satisfaction one of the agency’s key performance indicators. Every year, the agency call approximately 3,500 randomly selected Missourians to gather feedback. These telephone survey results are presented in MoDOT’s annual performance report, Tracker. What is unique about MoDOT, is they also include customer satisfaction results from “best in class” private industries compiled by the American Customer Satisfaction Index (see figure below).



Example image: Square

Benchmarking: Measure Definitions Matter

It is not uncommon for elected official and oversight bodies to ask public agencies how proposed performance targets compare to peer agencies. When the WMATA Board of Directors asked agency staff to defend the proposed bus on-time-performance targets to peer transit agencies, WMATA staff found themselves in a difficult situation. At the time, no industry standard definition existed for how to measure on-time performance. WMATA staff found wide diversity in the definitions used even by bus system operating in the same area (see table below). Given the absence of a consistent definition, WMATA staff recommended that the agency benchmark against itself to document improving or deteriorating on-time performance and set targets based on baseline trends. The WMATA Board agreed to the staff recommendation.



Example image: Square

Linkages to Other TPM Components

Reflect external stakeholder interests

Description

An important input into the targets setting business process is listening to external stakeholders. Reflecting their interests and perspectives in agency targets will help foster positive working relationship and support from external stakeholders.

Example image: Square

Gathering feedback from external stakeholders is a common practice as part of agency’s planning processes. However, what is newer is the reflection of those external viewpoints in the setting of agency targets. Items to keep in mind include:

  • Collecting feedback on tolerable performance thresholds
  • Piggy back on existing meetings
  • Develop educational material

Note: collaboration with external stakeholders to set agency targets as required under MAP-21 will be covered under Component B: External Stakeholders.

Examples

Understanding Public Priorities: MnDOT

During the develoment of their 2014-2033 Minnesota State Highway Investment Plan (MnSHIP), the agency conducted nine stakeholder meetings, conducted educational webinars, and created a new online interactive toolkit (see Figure below) to better understanding of what performance levels stakeholder expected. Three scenarios were shared with the public representing a different mix of strategies, funding allocation and outcomes. In addition, the public were asked to prioritize all investment categories (e.g., pavement , safety, bridge condition). The results from these external engagement efforts was then used to revise the agency’s performance targets.



Example image: Square

Understanding tolerable thresholds: MoDOT

Missouri DOT’s annual customer report card includes an “importance-satisfaction” analysis that plots the percenteage of Missourians who indicated a service offered by MoDOT is very important (not not) against the percentage of Missourians who are satisfied (or dissatified) with that current service. The simple graphic gives MoDOT guidance on where to focus transportation resources. For example, in 2010 when the agency was facing a notable funding shortfall, the importance-satisfaction chart highlighted an opportunity to shift resources from one service to another. As the figure below illustrates, Missourians were relatively satisfied with MoDOT’s mowing/triming services, but overall this was of less importance to citizens. Subsequently, MoDOt reduced their moving practices from four to three times a year resulting in $2.5million in savings that was reallocated to other system performance areas. The next survey showed this maintenance practice change had zero effect on customer satisfaction.



Example image: Square

Linkages to Other TPM Components


“Montana DOT has adopted a formal process for developing targets. Over time, the state legislature has gained confidence in the process and related funding requests because it provides consistent, quantifiable performance information that is fiscally constrained. The process is well received by the districts because it sets the type of work needed to meet the targets, but provides flexibility in terms of actual project selection.”
Source: NCHRP 551Performance Measures and Targets for Transportation Asset Management


































“New initiatives need good leaders.”
Source: NCHRP 8-92 Implementing Transportation Data Program Self-Assessment



Internal Key Players

They could include:

  • Agency Leader (Secretary, CEO, Chief Engineer)
  • Senior management
  • Planning group
  • Program managers
  • Finance group managers / budget staff;
  • Performance measure drivers;
  • Data owners; and
  • Technical analysts

Each of these has a role in data gathering, resource allocation or funding, and/or project selection. The stakeholder group may change depending on how these tasks are assigned in your organization, and may change as the target setting process is refined.


































































































































“Comparing MoDOT’s customer satisfaction results against well-known private sector companies send the message to our customers that we want to give them a “best in class” experience on our transportation system.”
- Karen Miller, Organizational Performance Specialist MoDOT


























































“Our public engagement effort gave us valuable guidance on where MnDOT should focus our efforts and in turn how to set our performance targets.”
Deanna Belden, Director of Performance, Risk & Investment Analysis
















“Public opinion surveys can also be helpful in the target-setting process to understand the relationship between different transportation system performance levels and the level of inconvenience or discomfort perceived by users.”
NCHRP 551: Performance Measures and Targets for Transportation Asset Management (Vol II, pg. 34)

Document the business process

Description

The collaborative business process for setting targets has been developed and now must be documented. The completion of this step means that staff within an agency have a clear understanding of their role in setting targets, the purpose of the targets, and an approach to reflect the interests of external stakeholders. The documentation of the business process will serve a rulebook and record of how the process was conducted, justification for the final targets and an explanation about why this approach was used. Documentation may not sound like the most exciting aspect of target setting, but it will ensure the business process is replicable from cycle to cycle, can withstand staff turnover and establishes an air of transparency around target setting. Material from this document can also be used to communicate with internal and external stakeholders the agency’s target setting business process.

As an agencies goes through multiple cycles of target setting, update the documentation. For example, as agency staff become more comfortable with target setting, more areas of the agency may want to get involved thus enhancing collaboration and the integration of performance management practices across the agency. Target setting, like many TPM components, is an iterative process so it will be critical to document steps that were taken and adjustments made to an agency’s business process.

Topic to address in the technical methodology documentation include:

  • Roles and responsibilities of internal agency staff
  • Outline of business process milestones and schedule
  • Process flow map
  • Highlight adjustments to process to implement in next target setting cycle
  • Specific issues related to each implementation step (see Table 5.8)

Linkages to Other TPM Components


Targets should not be set in stone, but periodically re-examined and adjusted based on documented reasoning.



Examples - Business Process Descriptions

In conjunction with each statewide long range plan a Governor-appointment external oversight body works with agency staff to determine appropriate targets out five to 25 years. In this iterative process, staff first reach concurrence on recommended targets and then engage in back and forth discussion with the Commission over several months. The inclusion of input from other external stakeholders adds a month or two to the process.

System performance area staff use data to identify needs, planning program staff provide traffic growth projections, results based on different scenarios established, final draft targets reviewed and approved by Executive staff with consensus from Chief Engineer/Field Division Engineers, final draft targets submitted to external oversight body for final approval. Process is marshalled along by an internal performance working group.

Technical asset management group develop proposed targets that will be vetted through senior executive steering committee. This process is an iterative process allowing functional areas the flexibility of proposing achievable targets while enabling the senior executive steering committee to provide oversight and guidance until achieving objectives that are aligned with Department goals.

Technical experts make recommendations to the executive group – which considers policy implications and implementation actions. The Statewide Congestion Working Group discusses the technical and policy aspects of target setting.

Technical asset group at Headquarters works with regional asset managers to review data and ensure accurate reporting before projecting performance at various funding levels. Headquarters group then works with the planning and finance groups to develop forecasts for various long-term funding scenarios. For annual budget setting, these scenarios are initially presented to Senior Management including the Executive Director and Chief Engineer, and once approved to an external oversight body. Proposed targets are revisited when final budgets are established by the legislature, then signed by the Governor. The forecasted condition at the approved budget is by default the annual target. Long range targets, however, continue to fall back to the most recent Statewide long range plan.

These descriptions come from State DOT responses to a survey distributed prior to the AASHTO Standing Committee on Performance Management (SCOPM) MAP-21 Target-Setting Workshop held in June 2013.

Keys to Success and Resources

Key precedent work on target setting includes: (more will be added)

Target Setting Action Plan

This section provides food for thought via a template for identifying steps to improve practices.