Planning Analysis:
Qualitative/Quantitative Research and Measurement Concepts/Evaluation Criteria

Three Categories of Qualitative Methods

1. Study of Built Form - qualitative methods can be used to document the impact of physical structures on people. This can include attitudes on the aesthetic quality of physical development.
   
2. Study of Human Interaction - Two types of techniques can be used (1) observations; and (2) interviews
   
3. Study of Planning Process and Organization Structures - can focus on citizen involvement for goal setting, alternative development, assessment, prediction, and forecasting using a wide variety of techniques - SWOT, Delbecq, Impass, etc.

Measurement Validity

Indicator is a valid measure of a concept if it accurately measures the concept it is intended to measure. If a indicator contains little error it is valid.

Types of measurement validity:

• Face Validity - if manager or others accept it as a valid indicator
• Consensual Validity - if numerous persons in different situations accept the indicator as valid. (i.e., recidivism rate as a measure of a prisons ability to reform a criminal)
• Correlational Validity - if indicator correlates strongly with other indicators that are accepted as valid (i.e., income as a measure of individual’s ability to afford housing--correlates with housing value, employment, etc)
• Predictive Validity - if it correctly predicts a specified outcome (i.e., scores on SAT predict student performance)

Measurement Reliability

An indicator is reliable if it consistently assigns the same numbers to some phenomenon. For example, if you were to measure the effectiveness of the police force two different times and arrive at the same value, the indicator would be considered reliable.

Another way to define reliability is that if the values obtained are not affected by who performed the measurement, or by any other factors, then the indicator is considered reliable.

Indicators of Reliability

• Subjective Measures - relies on judgement of a measurer or survey respondent. Some unreliability because of judgement (i.e., how old is the building you live in?)
• Lack of Precision - surveys typically have a margin of error of +-5%
• Corroboration – will repeated measurements yield the same results?

Measurement Scales

• Nominal – data are classified into exhaustive, mutually-exclusive categories, but not ordered categories. Nominal measurement totally lacks any sense of the relative size or magnitude, it only allows to say that things are different.

• Ordinal - data are classified into exhaustive, mutually-exclusive categories, and ordered categories. With this level it is possible to say that an object has more or less of a given characteristic than another, but not possible to say how much more or less.

• Interval - the measurement is based on a unit or interval that permits them to be measured exactly, and is accepted as standard units of measurement that can be infinitely divided.

• Ratio – the measurement is based on a scale that allows them to be measured exactly using generally accepted units of measurement, and that includes a non-arbitrary zero-point.

Hierarchy of Measurement

Evaluation Criteria

a. This is a key step in application of the rational model

b. In short....what measures or standards are we going to use to evaluate alternative policies....operationalize definitions

c. Objectives, criteria and measures

d. Example: Efficient transportation system (objective)......Cost (criteria).....NPV (measure)

Effectiveness: Measures whether a policy or program has its intended effect

• Direct/Indirect/Induced Impacts
• Long term/Short term

Adequacy: measures whether a policy or program fully meets its stated objectives

Costs/Benefits: Measures whether the anticiapted costs outweigh the benefits of a policy or program

Measures:

• NPV
• Total Cost
• Total Benefit
• Spillover

Measures whether the alternative is acceptable or can be made acceptable to relevant groups

Acceptability = to actors, or are they receptive to new policies

Appropriateness - does it mesh with peoples values

Responsiveness - will the program meet the target groups needs

Legal - is it within existing law

Equity - is it fair

Authority

Institutional committment

Capability

Organizational support

Good Criteria Are...

• clear
• measurable
• general - relate across alternativess
• trick is ranking and weighting criteria....see Chapter 7-8

[Home | Syllabus | Schedule | Assignments | Project |  Links]

This page maintained by Bob Parker, ©2002
October 07, 2003