NOTE: We include hypothetical implementation examples from among many possible scenarios for surveys using different modes below. These are designed to provide concrete examples, but additional scenarios / examples / options are certainly possible in real survey data collections.
Step 1: Identify Priorities
Identify your highest priorities / primary objectives for the survey data collection. Options include remaining on budget, controlling nonresponse error, producing a sufficient number of interviews for planned analyses, on-time delivery of data, etc.
a. What are the largest sources of error?
b. What are the negative consequences of going over budget? Producing biased estimates? Being underpowered?
c. What is the highest priority?
d. What is the next highest priority?
Rationale: Optimizing a survey design is a difficult task. In the absence of highly detailed cost and error estimates, it is necessary to have a sense of what the most important aspects of quality are in order to make decisions aimed at minimizing errors.
When to complete: Ideally, this step should be completed at the earliest stages of the study design. Many of the major decisions will be made at the very earliest stages of planning, e.g. which modes to size, sample sizes, etc. Identifying the priorities at this early stage will help in planning as you move to the next step.
survey example: We want to reduce
survey example: We need more
- Mail survey
example: We want to reduce
variance in our response rates across geographic areas.
- Web survey
example: We want to raise response
Step 2: Identify Major Risks
Identify the primary risk associated with your highest priority / primary objective from Step 1.
a. Which design estimates that relate directly to the highest priority are least certain?
b. Alternatively, which design estimates, if incorrect, will have the largest impact on the prioritized outcome?
survey example: Interviewer
efficiency is lower than expected.
survey example: Nonresponse bias in
key estimates (assuming that mobile respondents differ from landline
respondents on key variables).
- Mail survey
example: Increased variance in survey
estimates (due to increased variance in the survey weights).
- Web survey example: Nonresponse bias in key estimates.
Rationale: RSD is a method for dealing with uncertainty. As in any risk management strategy, one of the first steps is to identify major areas of risk. Once these have been identified, it becomes possible to monitor and intervene when problems arise.
When to complete: Once major design decisions have been made, it is possible to identify the major areas of risk. These may be based on the priorities identified in Step 1. That is, if minimizing nonresponse error is a key objective, then issues related to nonresponse are likely to be areas of risk. However, it is also important to evaluate the uncertainty. The design parameters with the largest uncertainty may also be a major risk. For example, if the response rate estimate has a large variance, then this may be an important area of risk.
Step 3: Define Indicators
Define measurable indicators of the risk from Step 2.
a. Are there direct measures of the risk? If not, are proxy measures available?
b. Is a system in place for collecting the data and summarizing the measures?
survey example: Hours per interview
(HPI), overall and by interviewer, in addition to daily measures of yield.
survey example: Response rates among
landline and mobile phone numbers, and estimates of key survey statistics for
both groups as well.
- Mail survey
example: The coefficient of variation
in the response rates across areas.
- Web survey example: The fraction of missing information (FMI) for key statistics.
Rationale: This step operationalizes the areas of risk defined in Step 2. This is a necessary step as there may not be direct indicators for each risk. In many situations, it may be necessary to define multiple indicators for some areas of risk. For example, there are no direct measures of nonresponse bias in the vast majority of settings. Instead, indirect indicators like response rates, subgroup response rates, and other comparisons of responders and nonresponders can be useful.
When to complete: This step needs to be completed after Steps 1 and 2. The survey design needs to be defined sufficiently that the available data from the sampling frame, paradata, and survey data are known.
Step 4: Define Decision Rules
Define a concrete decision rule for when you will intervene and change the data collection protocol.
a. At what value of the indicator defined in Step 3 will you make the change specified in Step 5?
survey example: HPI has continued to
rise for a week straight with no corresponding increase in daily yield.
survey example: The response rates
differ significantly between landline and cell phone cases in the sample, and
this difference has persisted for one week.
- Mail survey
example: The coefficient of variation
in the response rates across areas exceeds 0.20.
- Web survey example: The FMI for a given key variable is equal to the nonresponse rate, suggesting that low-responding subgroups on an auxiliary correlate should be targeted with additional effort.
Rationale: In order to create a replicable process, it is important to have pre-defined decision rules. This may also be important operationally so that the infrastructure can be in place in time to implement the intervention.
When to complete: Once Steps 1-3 are in place, Steps 4 and 5 need to be developed. There may be some iterations between steps 3-5. As the nature of the interventions becomes clear, this may impact the timing of their implementation and even the indicator used to trigger the intervention. For example, in the web survey case, analyses of correlations between any available auxiliary variables and survey responses of interest may need to be completed before deciding on an optimal auxiliary variable for defining subgroups and implementing the intervention.
Step 5: Modify the Survey Design and Monitor the Outcomes
Modify the data collection protocol when the decision rule has been met, and carefully monitor the effects of the intervention on the indicators from Step 3.
a. Are the procedures and systems in place for making the desired protocol change?
b. Is a system in place for monitoring and analyzing the indicators from Step 3 after the change has been made?
- Face-to-face survey example: Select a subsample of active sampled cases,
oversampling cases more likely to respond, and consider increasing the
incentive for this subsample of cases.
survey example: Focus interviewer
effort on mobile phone cases for a period of one week via changes in the sample
- Mail survey
example: Target follow-up efforts to
geographic areas with substantially reduced response rates, potentially
including additional incentives in follow-up mailings.
- Web survey example: Target follow-up efforts (additional emails, mention of a sweepstakes prize, etc.) to sampled cases belonging to the identified subgroups.
Rationale: The theoretical underpinning for changing the design is the idea that different features of the survey design will be important for each sampled person. Varying the design, therefore, leads to the widest possible recruitment from the sample. The goal is to vary the design features in ways that are complementary. For example, if the first phase of a data collection recruits more men than women, then the second phase should be designed to recruit relatively more women.
When to complete: Steps 4 and 5 depend upon Steps 1-3 being completed. However, Steps 4 and 5 are strongly related and developing the design change or intervention and the decision rule about when to implement that change may need to be done simultaneously.
Step 6: Compute Estimates
Compute overall estimates combining cases collected before and after the protocol change.
a. Is any weighting adjustment needed for combining the two groups of cases?
survey example: Adjust survey weights
for subsampled cases to reflect the probability of being included in the second
survey example: No adjustment needed.
- Mail survey
example: No adjustment needed.
- Web survey example: No adjustment needed.
Rationale: The estimator should reflect the design. When combining data across several phases of designs, it is important to reflect this multi-phase design. That can include any weighting differences due to subsampling, but also differences in who responds and the uncertainty associated with these patterns of response.
When to complete: It is good to have an analysis plan in place as soon as the design is in place. Before data collection begins, the plan should be ready.
Step 7: Document
Document your lessons learned, and communicate them in the form of academic papers, technical reports, or internal memoranda.
- Face-to-face survey example: Wagner et al. (2012), Journal of Official Statistics (Lesson learned: Dont attempt interventions in the second phase of the design!)
- Telephone survey example: Laflamme and Karaganis (2010), Implementation of Responsive Design (RD) for CATI Surveys at Statistics Canada (available at http://www.census.gov/fedcasic/fc2010/ppt/01_laflamme.pdf)
- Mail survey example: Hansen and Hurwitz (1946), Journal of the American Statistical Association (first known example of two-phase sampling using a mail survey).
- Web survey example: Sauermann and Roach (2013), Research Policy.
See the RSD bibliography for additional details.
For Step 7, the web site (hyperlinked) for this research education program will enable you to share success stories and difficulties alike in an interactive forum. This forum is designed to facilitate future communication and generate new ideas for improving RSD practices. Please make sure to continue documenting the results of your RSD efforts at this site!
Rationale: Documenting what was done makes it possible for others to replicate RSD procedures. This is important feature of the production of scientific findings the ability to replicate these findings in repeated tests. Further, well-documented procedures make it possible to extend and improve the design for the next survey.
When to complete: Documentation should begin during the design phase. That is, while the design is being developed, the documentation should also be developed. After the survey is complete, this documentation can then be updated with results and lessons learned.