Why conduct survey research?

To gather primary data that:

• Assists in planning or policy development

• Evaluates the effectiveness of a particular program

• Gains knowledge about behaviors and attitude

Survey Procedures

• Determine objectives

• Establish accuracy level desired

• Decided in conjunction with client

• Balance of accuracy and cost

• Select the most appropriate survey method

• List resources

• Make budget estimate

• Develop timeline

Consider mail when:

• Research sample is widely distributed geographically

• You want to give your subjects time to think about their responses

• Questions are written in a close-ended style

• Research sample has a moderate to high investment in the topic

• Want to give your subjects privacy in answering

• Your questions work better in a visual rather than oral form

• You have limited people and resources to help you conduct the survey

Draw a sample

Why sampling works:

• Flip 10 coins, and count the number of heads

• Do this 100 times, and plot distribution

• If you do this 1,000 more times, or 1 million more times, will distribution be different?

Its the size of the sample, not the size of the population, that matters

What sampling frames are available?

For each sampling frame:

• What is the availability, cost, and accuracy?

Sampling pitfalls

• Out-of-date lists

• Lack of coverage among various groups

• Do you want individuals or households?

Two types of samples

• Nonrandom

• statistical validity not a concern

• convenience surveys (e.g., in supermarkets, at tourist sites)

• surveyors tend to pick someone like themselves

• Random

• Unbiased, since everyone has equal chance of being selected

• Sample can only be as good as the list from which it was drawn

Types of Sample Schemes:

• simple random - select random number to start, every nth thereafter

• stratified sampling - divide population into subpopulations, then every nth

• cluster sampling - survey all units in a stratified area

Sampling frame - the list you draw your sample from (who is missed?)

Potential error:

• Sampling error - difference in distribution of characteristics between sample and population as a whole (this is an estimate, since we can’t really measure whole population)

• Level of confidence - sampling error is only half of our concern for accuracy

• Sampling error indicates how close our sample is to the population

• But how confident are we that the true value really does lie within that range?

• Accepted confidence level is 95% - nothing sacred about this number, just most commonly used

Sampling error and confidence level work together

Other sources of error:

• Frame error

Researcher should make every effort to minimize before survey implementation

• Non-response bias

                Who are non-responders?

                Difficult to measure - can be corrected for after implementation

Survey Development

Develop an initial questionnaire

        Types of questions

• Open-ended - short (what is your age?) and long (what do you think about ..)

• try to keep these to a minimum

• Survey Development (continued)

• Closed-ended

• Y/N

• Multiple choice - will you allow multiple responses?

• Ranking - people tend to have trouble

• Rating - can be difficult to word, can be subjective, how many categories?

• Likert scales: 3, 5, 7, or more responses possible

• Unipolar (E,VG,G,F,P) or bi-polar (SA,SA,NO,SD,SD)

What about  “don’t know?” - should you have one?

Composing Questions

     Basic Attributes of Questions

• Focus

• Brevity

• Clarity

• Expressing the Questions

• Vocabulary

• Grammar

• Potential Problems

Use of unstated criteria

Wrong: How important is it for stores to carry a large variety of different brands of this product?

Right:   How important is it to you that the store you shop at carries a large variety of different brands of this product?

Inapplicable questions

Wrong: How long does it take you to find a parking place after you arrive at the plant?

Right:   If you drive to work, how long does it take you to find a parking place after you arrive at the plant?

Examples in questions

Wrong: What small appliances, such as countertop appliances, have you purchased in the past month?

Right:   Aside from major appliances, what other small appliances have you bought in the past month?

Overdemanding recall

Wrong: How many times did you go out on a date with your spouse before you were married?

Right:   How many months were you dating your spouse before you were married?

Overgeneralizations

Wrong: When you buy fast food, what percentage of the time do you order each of the following types of food?

Right:   Of the last 10 times you bought fast food, how many times did you order each of the following types of food?

Overspecificity

Wrong: When you visit museums, how many times do you read the plaques that explain what the exhibit contains?

Right:   When you visit museums, how many times do you read the plaques that explain what the exhibit contains?  Would you say always, often, sometimes, rarely, or never?

Overemphasis

Wrong: Would you favor increasing taxes to cope with the current fiscal crisis?

Right:   Would you favor increasing taxes to cope with the current fiscal situation?

Two Questions in One

Wrong: Do you take vitamins to avoid getting sick?

Right:   Do you take vitamins?  Why or why not?

Leading Questions

Wrong: Don’t you see some danger in implementing community policing?

Right:   Do you see any danger in implementing community policing?

Loaded Questions

Wrong: Should we reduce the speed limit to save lives?

Right:   Should we lower the speed limit?  Why or why not?

Checklist to avoid potential problems

• Does the question state the criteria for answering?

• Is the question applicable to all respondents?

• Does the item contain an example that’s also a possible answer?

• Does the question require the respondent to remember too much detail or recall distant events?

• Is the question as specific as it can reasonable be?

• Are some of the words ambiguous?

• Does the question include only one issue?

• Does the question lead respondents toward a particular answer?

• Is the question loaded with a reason for responding in a particular way?

Question organization and format

• Begin with simple easy to answer questions

• Begin with questions that pertain to the purpose of the survey

• Questions should follow some logical order for respondents

• Vary the type of questions

• Put demographic and sensitive questions at the end

• Use simple words

• Have clear skip patterns

• Adequate spacing for open-ended questions

• Make survey visually appealing

• Be careful with maps (see examples)

• Lots of white space

• Attractive cover (see examples)

Pre-test survey

    Conducted at two levels:

• with experts

• with sample

Pretests

Can resolve issues such as:

• How many response categories should you have?

• Do the response categories represent reasonably spaced increments?

• Do you want a “don’t know” category?

• Is a question too intrusive?

• Can people answer all the questions?

• Are you using the proper local vernacular?

Survey Development

• Develop the final questionnaire

• Incorporate changes from pretest

• Review with client for final approval

• Begin writing methodology (see example)

• Develop the cover letter

• One page maximum

• Professional letterhead

• Have a contact person

• Catchy first sentence

Cover letter

• Tell respondent why their response is important

• Explain who is being asked to participate and how you got their name and address

• Are results confidential?

• How do they get survey back to you?

Survey Implementation

• How will you track responses?

• Personalize the mailing as much as budget allows

• Stamp instead of metered letter

• Stamp instead of business reply return envelope

• Individually addressed cover letter

• Hand-signed cover letter

Increasing response rates

• Incentives - A reward now is always better than later

• Difficulties with immediate incentives

• Political costs

• Advance publicity

• Media announcements

• Postcard

• Initial mailing

• Follow up postcard

• Second mailing

Survey Analysis

• Code questionnaires

• Code open-ended questions

• Arrange for data entry

• Analyze the data

• Present the results

[Home | Syllabus | Schedule | Assignments | Project |  Links]

This page maintained by Bob Parker, ©2002
September 30, 2002