close

Вход

Забыли?

вход по аккаунту

?

Performance Management 101: Learn How to Measure What Learn

код для вставки
Performance Management 101:
Learn How to Measure What
Matters
November 1, 2010
1
Presented by:
Britta Johnson and Shari Grossarth
Evaluation Support Division
Office of Strategic Environmental Management
Offi off Policy
Office
P li
U.S. Environmental Protection Agency
Presentation Goals
 Enable participants to
to:
• Use a logic model to help understand how program
resources and activities produce specific outputs and how
they connect to outcomes to achieve program/project
goals and objectives
• Understand the different types of performance measures
(resource, outputs, outcomes, efficiency, productivity, etc.)
• Use a step-by-step
step by step approach to develop
develop, assess
assess, and
choose the appropriate measures for their organization,
program, or project
2
Steps to Developing, Implementing and
Reporting Performance Measurement
Information
I. Identify Team/Develop
Performance Measurement Plan
II. Describe the Program
III. Develop Performance
Measurement Questions
IV. Develop Measures
V. Collect Information
VI. Analyze and Interpret
Information
VII. Develop the Report
3
Performance Measurement:
Session Agenda
g
 Module 1: Planning for Performance
Measurement
4
 Module 2: Identifying and Developing
Performance Measures
PERFORMANCE MANAGEMENT TOOLS
PERFORMANCE MANAGEMENT
Performance management includes activities to ensure that goals are consistently
being met in an effective and efficient manner. Performance management tools
include logic models, performance measurement and program evaluation.
Logic Model
Performance
P
f
Measurement
Program
Evaluation
Tool/framework that
helps identify the
program/project
/
resources,
activities, outputs
customers, and
outcomes.
Helps you
understand
d t d what
h t
level of
performance is
achieved by the
program/project.
/ j t
Helps you
understand and
explain why you’re
seeing the
program/project
results.
5
Drivers for Performance
Measurement
 Good Program Management
 Government Performance and Results Act ((GPRA)) of 1993
• Requires EPA to report schedules for and summaries of program
evaluations that have been or will be conducted and identify
those that influence development of the Agency’s Strategic Plan.
 Environmental Results Order 5700.7
• Requires EPA grant officers and grant recipients to identify
outputs and outcomes from grants and connect them to EPA’s
strategic plan.
 Agency/Organizational Priorities
6
Orientation Exercise
 In small groups, discuss and answer
th ffollowing
the
ll i question:
ti
• How do we use measurement in our
everyday lives?
 After discussing,
g, select a reporter.
p
7
Module 1:
Planning for Performance
Measurement
8
Steps to Developing, Implementing and
Reporting Performance Measurement
Information
I. Identify Team/Develop
Performance Measurement Plan
II. Describe the Program
III. Develop Performance
Measurement Questions
IV. Develop Measures
V. Collect Information
VI. Analyze and Interpret
Information
VII. Develop the Report
9
Part A. Identify Team Members
 Individuals responsible for the design,
collection and reporting of performance
information
 Indi
Individuals
id als with
ith intimate kno
knowledge
ledge of
the program/project
 Individuals who have a vested interest
in the conduct/impact of the program
 Individuals with some knowledge of the
measurement planning process
 Identify a Skeptic!
10
Part B. Develop the Measurement
Plan: Things
g to Consider
 Purpose
p
of the p
performance measurement
system
 Project/
j
p
program
g
mission
 Primary audience
 S
Scope ((including program description// logic
model)
 Context
C t t (organizational,
(
i ti
l management,
t political)
liti l)
 Role, expectations for program staff,
participants,
ti i
t and
d kkey stakeholders
t k h ld
11
Part B. Measurement Plan Outline
 Performance measurement questions
q
 Data collection/analysis
 Reporting (including feedback loop)
 Resources (staff and budget)
 Timeline
 Communication
 Steps to monitor implementation of the plan
12
Steps to Developing, Implementing and
Reporting Performance Measurement
Information
I. Identify Team/Develop
Performance Measurement Plan
II. Describe the Program
III. Develop Performance
Measurement Questions
IV. Develop Measures
V. Collect Information
VI. Analyze and Interpret
Information
VII. Develop the Report
13
14
What is a Logic Model?
A logic model is a diagram and text that
describes/ illustrates the logical (causal)
relationships among program elements and
the problem to be solved, thus defining
measurements of success.
15
We use
W
these
resources…
For these
activities…
To
produce
these
outputs…
So that these
customers can
change their
ways…
Which
Whi
h lleads
d
to these
outcomes…
Leading
L
di
to these
results!
Elements of the Logic Model
What
we
have.
Who we reach/
influence.
The
results
we see
see.
WHY
HOW
What we produce/
deliver.
Whatt we
Wh
do.
O tcomes
Outcomes
Resources/
Inputs:
Activities
Outputs
Customer
Short-term
Intermediate
Programmatic
investments
available to
support the
program.
p
g
Things you do–
activities you
plan to conduct
in your program.
Product or
service delivery/
implementation
targets you aim
to produce.
User of the
products/
services. Target
audience the
program is
d i
designed
d tto
reach.
Changes in
learning,
knowledge,
attitude, skills,
understanding.
Changes in
behavior,
practice or
decisions.
Change in
condition.
Behavior
C diti
Condition
PROGRAM
Attitudes
Longterm
RESULTS FROM PROGRAM
External Influences
Assumptions
Factors outside of your control (positive or negative) that may influence
the outcome and impact of your program/project.
The beliefs you have about the program, the
people involved, the context and the way you think
the program
will work.
p g
Thanksgiving Dinner 2010
C t
Customers
Resources
Activities
Develop a
Menu
Outputs
Intermediate
outcomes
Long-term
outcomes
Menu
Me
My Sister
$
Me & My
Sister
Find
Recipes
Short-term
outcomes
Realize we
can
Cook
Dinner
Tasty
Thanksgiving
Dinner
Grocery
Shop
External Factors: Too Caught Up in Football
17
Logic Model for SSO/Infrastructure Integrated Enforcement
and Assistance Initiative
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
A
Resources
Region 1 Staff:
OES (A&P2,
water technical,
legal)
OEP (SI, Permits)
OARM GIS
Funding:
OW, OPEI,
CA, OECA
Monitoring
(OEME?)
UMD software
NBC
Partners
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
B
Activities
Measurement
ID performance measures
Develop logic model
Collect baseline data
Track SSO occurrence
Internal Planning/Development
Integrate enforce/assistance
ID/integrate regional priorities
Coordinate with ORA
Develop initiative letter
Coordinate with HQ
Technical Assistance/Outreach
Conduct workshops
Develop web site and tools
Give assistance (on-site, phone)
Conduct Energy Management
System Demonstration pilots
D l EMS workbook
Develop
kb k
Develop community outreach
GIS mapping/support
RI DEM
Enforcement
Develop/issueAOs
Create RI SSO database-id targets
Conduct inspections
Review respondent docs.
Pursue penalty actions
Info gathering (incl 308 letters)
External Networking
Communication with RI DEM
Identify and communicate
with stakeholders
Notify municipal contacts
18
C
Outputs
Target Audience
Internal Coordination
Develop strategy
Briefing package
Meetings
Region 1 OEP/OES
management, staff
Assistance
Workshops
On-site
Web site
Phone calls
Use of TEAMS software
PM template
Press release
Fact sheet
EMS workbook
GIS Data Model
Voting citizens
Enforcement
Joint letter
Tiered enforcement list
AO
Checklist
APO
308 letters
External Networking:
Phone calls
Meetings with municipal
officials and associations
Coordination with
RI DEM:
Meetings, phone
calls, briefings
Outcomes
D
Collection
systems/utilities
Municipal officials
& decision makers
Short-term
F
Intermediate
Long-term
Acknowledgement that
EPA should use
enforcement and
assistance
Common recognition of
problem & potential
solutions:
Understanding of role
Willingness to deal with
issue
F1
F-1
Public supports/votes to
support public spending
for infrastructure
Implement measures to
reduce energy consumption:
Pilot test Energy Man. Sys.
Implement EMS workbook
Inc. in awareness,
skills, knowledge, &
use of assistance tools
Increased reporting of
SSOs
F-4
Trade associations
Municipal officials
Rhode Island DEM
Acknowledgement that
EPA should use
enforcement and
assistance
Increased energy
efficiency(links back
to TA/Outreach)
F-2
Assess system, develop &
p
use CMOM,, preventive
maintenance and asset
management plan
F-3
Awareness of initiative
and need to respond to
AO and issues in letter
Communities have
sustained and strategic
commitment to water
g
infrastructure ((e.g.
invest in long-term
capital improvements,
water management
plans)
F-5
Work with federal & state
officials to engage in
initiative outcomes
F-6
Well-maintained,
managed and funded
water infra.
infra system
system.
Correct problems
leading to SSOs
Reduce the # and
severity of SSOs and
backups
F-7
Improved water quality
* Reduction of bacteria
* Reduction in # of
beach and shellfish bed
closures.
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
External Factors Influencing Program and Outcomes:
-Entity specific factors, including competing municipal interests
interests,
, budgets,
budgets,
contracts,
personnel, personnel,
Contracts,
etc.
etc.
-Impact of other efforts within Rhode Island to address water compliance
com
pliance issues
issues
-General trend of facility improvement over time
-Resistance to/complications associated with using an integrated
approach
approach
E
-Weather
-Perception/expectation of the regulated community about Initiative
Initiati
ve impacts
impacts
Communication among
partners & stakeholders
(links back to Outreach
& External Networking)
19
An Important Rule to Follow
For every action identified in the Logic Model,
there must be an output that connects to an
outcome through a specific customer.
20
THINK CONNECTIONS!
Exercise 1: Logic Modeling
Developing a logic model
21
Logic Model for SSO/Infrastructure Integrated Enforcement
and Assistance Initiative
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
A
Resources
Region 1 Staff:
OES (A&P2,
water technical,
legal)
OEP (SI, Permits)
OARM GIS
Funding:
OW, OPEI,
CA, OECA
Monitoring
(OEME?)
UMD software
NBC
Partners
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
B
Activities
Measurement
ID performance measures
Develop logic model
Collect baseline data
Track SSO occurrence
Internal Planning/Development
Integrate enforce/assistance
ID/integrate regional priorities
Coordinate with ORA
Develop initiative letter
Coordinate with HQ
Technical Assistance/Outreach
Conduct workshops
Develop web site and tools
Give assistance (on-site, phone)
Conduct Energy Management
System Demonstration pilots
D l EMS workbook
Develop
kb k
Develop community outreach
GIS mapping/support
RI DEM
Enforcement
Develop/issueAOs
Create RI SSO database-id targets
Conduct inspections
Review respondent docs.
Pursue penalty actions
Info gathering (incl 308 letters)
External Networking
Communication with RI DEM
Identify and communicate
with stakeholders
Notify municipal contacts
C
Outputs
Target Audience
Internal Coordination
Develop strategy
Briefing package
Meetings
Region 1 OEP/OES
management, staff
Assistance
Workshops
On-site
Web site
Phone calls
Use of TEAMS software
PM template
Press release
Fact sheet
EMS workbook
GIS Data Model
Voting citizens
Enforcement
Joint letter
Tiered enforcement list
AO
Checklist
APO
308 letters
External Networking:
Phone calls
Meetings with municipal
officials and associations
Coordination with
RI DEM:
Meetings, phone
calls, briefings
Outcomes
D
Collection
systems/utilities
Municipal officials
& decision makers
Short-term
F
Intermediate
Long-term
Acknowledgement that
EPA should use
enforcement and
assistance
Common recognition of
problem & potential
solutions:
Understanding of role
Willingness to deal with
issue
F1
F-1
Public supports/votes to
support public spending
for infrastructure
Implement measures to
reduce energy consumption:
Pilot test Energy Man. Sys.
Implement EMS workbook
Inc. in awareness,
skills, knowledge, &
use of assistance tools
Increased reporting of
SSOs
F-4
Trade associations
Municipal officials
Rhode Island DEM
Acknowledgement that
EPA should use
enforcement and
assistance
Increased energy
efficiency(links back
to TA/Outreach)
F-2
Assess system, develop &
p
use CMOM,, preventive
maintenance and asset
management plan
F-3
Awareness of initiative
and need to respond to
AO and issues in letter
Communities have
sustained and strategic
commitment to water
g
infrastructure ((e.g.
invest in long-term
capital improvements,
water management
plans)
F-5
Work with federal & state
officials to engage in
initiative outcomes
F-6
Well-maintained,
managed and funded
water infra.
infra system
system.
Correct problems
leading to SSOs
Reduce the # and
severity of SSOs and
backups
F-7
Improved water quality
* Reduction of bacteria
* Reduction in # of
beach and shellfish bed
closures.
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
External Factors Influencing Program and Outcomes:
-Entity specific factors, including competing municipal interests
interests,
, budgets,
budgets,
contracts,
personnel, personnel,
Contracts,
etc.
etc.
-Impact of other efforts within Rhode Island to address water compliance
com
pliance issues
issues
-General trend of facility improvement over time
-Resistance to/complications associated with using an integrated
approach
approach
E
-Weather
-Perception/expectation of the regulated community about Initiative
Initiati
ve impacts
impacts
22
Communication among
partners & stakeholders
(links back to Outreach
& External Networking)
Module 2:
Identifying and Developing
Performance Measures
23
Definitions:
Performance Measurement:
The ongoing monitoring and reporting of program
progress and accomplishments, using pre-selected
performance measures
measures.
Performance Measure:
A metric used to gauge program or project
performance.
Indicator:
Performance measure defined operationally in
terms of how the measure is actually taken or how
th data
the
d t are collected.
ll t d
24
Uses of Performance Measurement
 Monitoring
g and reporting
p
g
 Strategic planning
 Budgeting and financial management
 Program
g
management
g
 Process improvement
 Contract management
 Communication with public
Source: Chapel, T., Center for Disease Control and Prevention, Power Point Presentation, Program Alignment,
Performance Measurement, and Program Improvement
25
Limitations and Pitfalls of
Performance Measures
 Provide descriptive data, not rigorously
evaluative.
 Can
C encourage undesirable
d i bl b
behavior.
h i
 Mayy require
q
too much time and effort.
 Can be ignored, not automatically used.
 Performance measurement is time-bound
and context-bound.
26
Steps to Developing, Implementing and
Reporting Performance Measurement
Information
I. Identify Team/Develop
Performance Measurement Plan
II. Describe the Program
III. Develop Performance
Measurement Questions
IV. Develop Measures
V. Collect Information
VI. Analyze and Interpret
Information
VII. Develop the Report
27
Performance Measurement
Questions
 What are they?
• Questions designed to assess progress/
accomplishments
p
of various aspects
p
of a
program/project (at any point on the
performance spectrum/logic model).
28
• Performance measurement questions ask/tell
you what your program is doing
doing.
• Theyy should reflect stakeholders needs.
Key Questions to Consider
 What is my program doing to/with whom?
 How well is my program doing it?
 Who’s better off?
 What evidence is there that my program
caused changes?
 Are there any unintended outcomes?
 What role, if any, did context play?
29
Developing Questions


Use a p
program’s
g
logic
g model to:
•
Review program purpose and objectives.
•
Identify what aspects of program you want to evaluate.
Brainstorm a variety of different questions by
asking:
•
I would really like to know _______ about this program
element.

Group questions by logic model categories
categories.

Select highest priority questions
30
Performance Questions Across the
Logic
g Model Spectrum
p
PROGRAM
ELEMENTS:
PERFORMANCE
QUESTIONS:
Resources
Activities/
(We use
these)
Outputs
Do we have
enough,
The right,
The
necessary
level,
The
consistency?
Which
resources
are most
effective for
achieving
program
objectives?
Why?
EXTERNAL
INFLUENCES:
Target
Customer
Short term
Outcome
Intermediate
Outcome
Long-Term
Outcome
(For these
people)
l )
(To change
th
them
i th
in
these
ways)
(So they can
d th
do
these
things)
(Which leads
t these
to
th
outcomes)
Are we
executing
the p
program
g
as intended?
Are we
reaching
the target
g
audience?
Are we
producing
products and
services at
the levels
anticipated?
Are we
reaching
the
anticipated
numbers?
Did the
customer’s
attitude,
knowledge,
skills or
understanding
change?
Are
customers
using
g the
change as
expected?
With what
results?
What
changes in
condition
have
occurred?
What role did
the program
have in
observed
changes?
Are
customers
served
changing in
the expected
direction and
level?
(To do these
things)
Why or why
not?
Are
customers
satisfied?
Which
program
services were
most effective
in generating
change?
Did the
program
achieve its
goals and
objectives?
If so, what
did we
(others) do
to cause the
change?
How died external factors influence my program’s success? How could the program have
mitigated for these effects?
31
Steps to Developing, Implementing and
Reporting Performance Measurement
Information
I. Identify Team/Develop
Performance Measurement Plan
II. Describe the Program
III. Develop Performance
Measurement Questions
IV. Develop Measures
V. Collect Information
VI. Analyze and Interpret
Information
VII. Develop the Report
32
Measures Across the Logic Model
Spectrum
p
Element
Definition
Example Measure
Resources/
Inputs
Measure of resources consumed by
the organization.
g
Amount of funds, # of FTE, materials,
equipment,
q p
, supplies
pp
((etc.).
)
Activities
Measure of work performed that
directly produces the core products
and services.
# of workshops offered as designed;
Hours of technical assistance training
for staff.
Outputs
Measure of products and services
provided as a direct result of program
activities.
# of technical assistance requests
responded to; # of EMS workbooks
developed/delivered.
Customer
Reached
Measure of target population receiving
outputs.
% of target population trained; # of
target population receiving technical
assistance.
Customer
Satisfaction
Measure
easu e o
of sat
satisfaction
s act o with
t outputs
outputs.
%o
of custo
customers
esd
dissatisfied
ssat s ed with
t
training; % of customers “very
satisfied” with assistance received.
Outcomes
Accomplishment of program goals
and objectives (short-term
(short term and
intermediate outcomes, long-term
outcomes--impacts).
# of municipal officials aware of the
initiative; % increase in public support
for future infrastructure spending; %
decrease in the # of beach and
shellfish bed closures
33
Work Quality Measures
Category
Definition
Examples
Efficiency
Measure that relates outputs to
costs
costs.
Cost per workshop
conducted; cost per
workbook produced.
Productivityy
Measure of the rate of production
per some specific unit of resource
(e.g., staff or employee). The focus
is on labor productivity.
Number of enforcement
cases investigated per
inspector.
Cost
Effectiveness
Measure that relates outcomes to
costs.
Cost per infrastructure
problems corrected; cost
per mile of beach cleaned.
Service
Quality
Measure of the quality of products
and services produced.
Percent of technical
assistance requests
responded to within one
week.
34
Steps for Developing Measures
 Step
p 1: Identify
y Potential Measures
 Step 2: Assess Each Measure
35
 Step 3: Choose the Best Measures
 Step 4: Identify Baseline, Target, Timeline and
Reporting Schedule
Key Steps in Identifying Potential
Measures
 Identify the information needed and the audience
 Identify measures in existing documents
 Review the logic model and select the
appropriate
pp p
logic
g model element
 Express the logic model element as a
performance measure
 Determine if the measure clearly relates to the
program/project goal or objective
36
Identify the Information Needed
and the Audience
 Review the performance measurement
questions developed earlier
 Consider what information is needed to
assess whether your program/project is
meeting its goals and objectives.
37
 Ask yourself: Who needs to know what
about the program
program, why
why, and in what
format?
Identify Measures in Existing
Documents
 Review measures specified in:
• Program/Project Mission, Goals, Objectives,
Service standards
• Legislation, Strategic plans (GPRA), Court
Orders, Regional Plans, National Program
Management Guidance,
Guidance Regional Priority
Commitments
• Previous evaluations and research reports
• Consider
C
id other
th sources
38
Review the Logic Model
•
Identify the aspects of performance that are
most important to measure (resources,
activities outputs
activities,
outputs, outcomes)
•
Identify contextual factors that could influence
the program either positively or negatively and
generate measures for them as appropriate
39
Logic Model for SSO/Infrastructure Integrated Enforcement
and Assistance Initiative
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
A
Resources
Region 1 Staff:
OES (A&P2,
water technical,
legal)
OEP (SI, Permits)
OARM GIS
Funding:
OW, OPEI,
CA, OECA
Monitoring
(OEME?)
UMD software
NBC
Partners
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
B
Activities
Measurement
ID performance measures
Develop logic model
Collect baseline data
Track SSO occurrence
Internal Planning/Development
Integrate enforce/assistance
ID/integrate regional priorities
Coordinate with ORA
Develop initiative letter
Coordinate with HQ
Technical Assistance/Outreach
Conduct workshops
Develop web site and tools
Give assistance (on-site, phone)
Conduct Energy Management
System Demonstration pilots
D l EMS workbook
Develop
kb k
Develop community outreach
GIS mapping/support
RI DEM
Enforcement
Develop/issueAOs
Create RI SSO database-id targets
Conduct inspections
Review respondent docs.
Pursue penalty actions
Info gathering (incl 308 letters)
External Networking
Communication with RI DEM
Identify and communicate
with stakeholders
Notify municipal contacts
40
C
Outputs
Target Audience
Internal Coordination
Develop strategy
Briefing package
Meetings
Region 1 OEP/OES
management, staff
Assistance
Workshops
On-site
Web site
Phone calls
Use of TEAMS software
PM template
Press release
Fact sheet
EMS workbook
GIS Data Model
Voting citizens
Enforcement
Joint letter
Tiered enforcement list
AO
Checklist
APO
308 letters
External Networking:
Phone calls
Meetings with municipal
officials and associations
Coordination with
RI DEM:
Meetings, phone
calls, briefings
Outcomes
D
Collection
systems/utilities
Municipal officials
& decision makers
Short-term
F
Intermediate
Long-term
Acknowledgement that
EPA should use
enforcement and
assistance
Common recognition of
problem & potential
solutions:
Understanding of role
Willingness to deal with
issue
F1
F-1
Public supports/votes to
support public spending
for infrastructure
Implement measures to
reduce energy consumption:
Pilot test Energy Man. Sys.
Implement EMS workbook
Inc. in awareness,
skills, knowledge, &
use of assistance tools
Increased reporting of
SSOs
F-4
Trade associations
Municipal officials
Rhode Island DEM
Acknowledgement that
EPA should use
enforcement and
assistance
Increased energy
efficiency(links back
to TA/Outreach)
F-2
Assess system, develop &
p
use CMOM,, preventive
maintenance and asset
management plan
F-3
Awareness of initiative
and need to respond to
AO and issues in letter
Communities have
sustained and strategic
commitment to water
g
infrastructure ((e.g.
invest in long-term
capital improvements,
water management
plans)
F-5
Work with federal & state
officials to engage in
initiative outcomes
F-6
Well-maintained,
managed and funded
water infra.
infra system
system.
Correct problems
leading to SSOs
Reduce the # and
severity of SSOs and
backups
F-7
Improved water quality
* Reduction of bacteria
* Reduction in # of
beach and shellfish bed
closures.
The image cannot be display ed. Your computer may not hav e enough memory to open the image, or the image may hav e been corrupted. Restart y our computer, and then open the file again. If the red x still appears, y ou may hav e to delete the image and then insert it again.
External Factors Influencing Program and Outcomes:
-Entity specific factors, including competing municipal interests
interests,
, budgets,
budgets,
contracts,
personnel, personnel,
Contracts,
etc.
etc.
-Impact of other efforts within Rhode Island to address water compliance
com
pliance issues
issues
-General trend of facility improvement over time
-Resistance to/complications associated with using an integrated
approach
approach
E
-Weather
-Perception/expectation of the regulated community about Initiative
Initiati
ve impacts
impacts
Communication among
partners & stakeholders
(links back to Outreach
& External Networking)
Logic Mod
del Elements
Express the Logic Model Element as
a Performance Measure
Resources
Activities
Outputs
Customer
reached
Short-term
Outcome
Intermediate
Outcome
Long-term
Outcomes
Region 1 Staff
$
Monitoring
(OEME)
UMD Software
NBC
Rhode Island
DEM
Develop initiative
letter, press
release
Workshops
Collection
systems/ utilities
Common
recognition of
problem and
potential solutions
Assess system,
develop & use
C O
CMOM,
preventive
maintenance &
assess
management plan
Well-maintained
& funded water
infra
infrastructure
system
Increased
I
d
reporting of SSOs
# and severity of
SSOs reduced
#/%
#/% off entities
titi
that:
%
% change
h
in
i
41
number of SSOs
since baseline
Issue AOs,,
APOs, and 308
letters
EMS Workbook
Municipal officials
& decision makers
Increased
awareness,
skills, knowledge,
& use off
assistance tools
Tiered
enforcement list
Inspection
checklist
Problems
leading to SSOs
corrected
Conduct
inspections
 Description
D
i ti off
resources/ $$
invested by EPA
Region 1
Exa
ample Measure
es
Provide
assistance
Website
Description of
resources/$$
invested by Rhode
Island DEM
#
# off news outlets
tl t
running stories on
integrated initiative
# of workshops
provided
 # of AO, APOs,
and 308 letters
issued
# of inspections
conducted
IIncluded
l d d iin
activities
measures
 #/% off collection
ll ti
systems/utilities
that received AOs
#/% of collection
systems/utilities
that received
assistance
#
# off collection
ll ti
systems/utilities
that report
increased
management
attention to SSO
issues after
launch of
initiative
Average # of
times EPA
assistance
requested
t d
- Improved
system mapping
since launch of
initiative
- Implemented
Asset
Management
practices
- Partially
y or fully
y
met SSO
reporting
requirements
% change in
number of
basement
backups since
baseline
% change in
average # of
gallons released
from SSOs
reported
Express the Logic Model element
as a p
performance measure

Consider how to express the measure in
terms of data:
•
•
•
•
•
Raw numbers (total # of SSOs reported per
year)
Averages (average # of SSOs reported per
entity)
Percentages (% reduction in number of SSOs
reported)
Ratios (Program cost per SSO reduced)
Rates (Gallons of SSO releases per 1,000
residents)
42
Determine whether the measures
clearly
y relate to the mission/goal
g
 Review the program/project mission and or goal
• What key activities, outputs or outcomes are specified in the
mission or goal?
 Review the list of potential measures developed
• Will the data collected from the measures developed
p
clearly demonstrate that the mission and or goal was
accomplished?
43
Determine whether the measures
y relate to the mission/goal
g
clearly
 Rhode Island Sanitary Sewer Overflow Integrated
S t
System
P
Purpose:
Reduce the number and severity of SSOs and improve the
management and long-term
long term sustainability of Rhode Island
collection systems.
 Performance Measures:
• % facilities that have made (or are taking steps toward) in
how they manage collection system information
p
g asset management
g
p
practices
• % entities implementing
• % entities that have developed (or are developing) a written
preventative maintenance plan
p
in their reporting
p
g of
• % facilities that have made improvements
SSOs
• % reduction in the frequency and severity of SSOs
44
Exercise 2: Performance
Measures
D
Developing
l i Y
Your O
Own M
Measures
45
Step 2: Assess Each Measure
Assess the feasibilityy of the measures for:
 data collection
 data quality
 analysis
 reporting
46
Assess Each Measure for Data
Collection
 Availability
• Existing data vs. new data
 Frequency
• One time collection, continued data collection
 Utility
• Data available for use
• Supports an acceptable baseline
 Cost
• Overall implementation cost
47
Assess Each Measure for Data
Qualityy
 Reliability
• Provides consistent readings
g
 Validity
• Measures what it is supposed to measure
 Objectivity
• Free from bias and represents reality
48
Assess Each Measure for Analysis:
 Statistically reliable
• Population data
• Sample data
 Type of analysis
• Trends over time
• Actual performance against targets or standards
• Variation across units (internal benchmarking)
• Against benchmarks (external benchmarking)
 External factors
• Direct or near direct control over the measure
• Account
Acco nt for impact of e
external
ternal factors on the meas
measure
re
49
Step 3: Choose the Best Measures
 Assess the value of the measures in relation to
the goals and objectives of the program
program.
• Required
• Important
• Interesting
 Select final list of measures – yyou won’t be able
to collect data for all measures.
 Check in with managers
g
and stakeholders.
 Identify a priority list of measures
50
Tips for Choosing the Best
Measures
For each measure ask…

Does the measure clearly relate to the project goal
and objective?

Is the measure important to management and
stakeholders?

Is it possible to collect accurate and reliable data for
th measure?
the
?

Taken together, do the measures accurately reflect
the keyy results of the p
program,
g
activity
y or service?

Is there more than one measure for each goal or
objective?

Are your measures primarily outcome, efficiency, or
quality measures?
51
Step 4: Identify a Standard
For each performance measure
develop a:
1. B
1
Baseline
li – currentt state
t t
2. Target – desired level of performance
3. Timeline – date when performance
will be achieved
52
Criteria for Useful Performance
Measures
easu es
Is each measure:
If so, then it will be:
Objective-linked
Directly related to clearly stated objectives for your program.
Responsibilitylinked
Matched to specific organizational units and people that are
responsible for AND capable of taking action to improve performance.
Organizationally
acceptable
Valued by all levels in the organization
organization, used as a management tool
tool,
and viewed as being “owned” by those accountable for performance.
Comprehensive
Inclusive of all relevant aspects of the program performance; e.g.,
measuring quality and quantity.
C dibl
Credible
Based
B
d on accurate
t and
d reliable
li bl d
data
t sources and
d methods,
th d and
d tto th
the
extent possible, not open to manipulation or distortion.
Cost-effective
Acceptable in terms of data collection, processing, and reporting.
Compatible
Integrated with existing information systems.
Comparable with
other data
Useful in making comparisons; e.g., performance can be compared
from period to period
period, with peers
peers, to other programs
programs.
Easy to interpret
and report
Presented graphically and accompanied by commentary!
[1] Adapted from Price Waterhouse – Center for Performance Measurement
53
Contacts:
Shari Grossarth
grossarth.shari@epa.gov
(202) 566-2242
566 2242
Britta Johnson
johnson.britta@epa.gov
(202) 566
566-1465
1465
www.epa.gov/evaluate
54
Документ
Категория
Без категории
Просмотров
3
Размер файла
1 422 Кб
Теги
1/--страниц
Пожаловаться на содержимое документа