Using Cognitive Interviews to Develop Structured Surveys
Prepared by the Center for Aging in Diverse Communities (CADC)
University of California, San Francisco
Cognitive interviews are used widely in questionnaire development to detect items that are not understood by respondents as intended by the survey developers. Cognitive interviews can be used to revise or develop new items so that they are appropriate to respondents’ cultural context and lifestyle. Cognitive interviews are used usually during the pre-testing phase of a survey. Results of cognitive interviews illustrate the types of errors made by respondents, and how they interpret and answer questions. Generally, cognitive interview methods reflect a theoretical model of the survey response process that involves four stages: comprehension or interpretation, information retrieval, judgment formation, and response editing. In other words, the respondent must first understand the question, then recall information, then decide upon its relevance, and finally formulate an answer in the format provided by the interviewer. One cognitive interview technique is to ask respondents to verbalize their thoughts while answering survey questions (think aloud). In recent years, cognitive interviewing has relied more heavily upon verbal probes about the interpretation of questions and recall strategies. Such probes may be scripted or spontaneously created by the interviewer; they may be administered immediately after individual survey questions or after completion of the entire questionnaire. Below we identify several key articles, books and book chapters that illustrate the use of these methods for pre-testing surveys.
Beatty P. The dynamics of cognitive interviewing. In: Presser S, Rothgeb JM, Couper MP, et al., eds. Methods for testing and evaluating survey questionnaires. Hoboken, N.J.: John Wiley & Sons, Inc.;2004:45-66.
As a way to formulate standard methods for cognitive interviewing, the author examines specific interviewer behaviors and their impact on the results of cognitive interviews. In a series of small laboratory based cognitive tests the author determined the effects that different types of probing had on the participants’ ability to choose an answer (precision), and how interviewer behavior influenced or biased participants’ responses. Recommendations for interviewing practices are made.
Carbone ET, Campbell MK, Honess-Morreale L. Use of cognitive interview techniques in the development of nutrition surveys and interactive nutrition messages for low-income populations. Journal of the American Dietetic Association. 2002; 102(5):690-696.
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=12008995&query_hl=1&itool=pubmed_docsum
This article describes the use of cognitive interviews to assess comprehension of nutrition survey items and nutrition intervention messages in low-income families in North Carolina. Twenty-three technical community college students responded to a posted school notice about the study. The sample included 8 males and 15 females, all were over 18 years old, and half were African American. The cognitive interviews identified problems with unclear instructions, confusing terms and response options, variable interpretations of terms, and misinterpretation of dietary recommendations. The results were utilized to clarify ambiguous instructions and terms, expand response options, and more clearly specify dietary recommendations to make the survey and intervention messages more suitable for use in low-income groups.
DeMaio TJ, Rothgeb JM. Cognitive Interviewing Techniques: In the Lab and in the Field. In: Schwarz N, Sudman S, eds. Answering Questions. Methodology for Determining Cognitive and Communicative Processes in Survey Research. San Francisco: Jossey-Bass, Inc.; 1996: 177-196.
This article provides a general review of cognitive interviewing techniques used in laboratory and field settings. Using three real life examples, the authors demonstrate the use of cognitive testing in both interviewer administered and self report surveys, review the procedures used in each case, and provide a brief analysis of the results. They also describe varying methods used during respondent debriefing, a technique commonly used to conduct cognitive testing in the field.
Harris-Kojetin LD, Fowler FJ Jr., Brown JA, Schnaier JA, Sweeney SF. The use of cognitive testing to develop and evaluate CAHPS 1.0 core survey items. Consumer Assessment of Health Plans Study. Medical Care. 1999; 37 (3 Suppl):MS10-21.
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=10098555&query_hl=4&itool=pubmed_docsum
The authors performed 150 cognitive interviews to facilitate the development of the CAHPS core survey questionnaire, a standardized survey used to obtain information on the experiences of health care plan enrollees. The sample included 14 low literacy adults, 22 parents or guardians of children, 29 adults with chronic conditions, 31 Medicare beneficiaries and 54 Medicaid recipients. The authors describe the process of conducting cognitive interviews and their role in survey development. They report their findings pertaining to three methodological areas: 1) using ratings versus reports when asking respondents to assess their experience; 2) the length of the reference time specific to the questions; and 3) ensuring respondents utilize the ‘not applicable’ response when appropriate in self-report surveys.
Jobe JB, Mingay DJ. Cognitive research improves questionnaires. American Journal of Public Health. 1989;79(8):1053-5.
http://www.ajph.org/cgi/reprint/79/8/1053
The authors review several National Center for Health Statistics (NCHS) reports and demonstrate how cognitive testing techniques can be used to improve patient understanding of questions, recall of past events, and understanding of medical conditions in national health surveys such as the NHIS.
Johnson TP, Cho YI, Holbrook AL, O’Rourke DO, Warnecke RB, Chavez N. Cultural variability in the effects of question design features on respondent comprehension of health surveys. Annals of Epidemiology. 2006;16(9):661-8.
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=16473526&query_hl=7&itool=pubmed_docsum
The authors performed behavioral coding of 345 face-to-face interviews with African Americans (n=86), Mexican Americans (n=101), Puerto Ricans (n=74), and non-Latino Whites (n=84) to identify question characteristics that might account for cultural variability in responses. Behavior coding involves the systematic coding of respondent and interviewer behaviors using a pre-determined coding scheme to identify problems with survey questions. The interview included standard questions from the National Health Interview Survey (NHIS), the Behavioral Risk Factor and Surveillance Survey (BRFSS), and the former National Household Survey of Drug Abuse. All interviews were conducted in English. Race/ethnicity was found to be independently associated with question comprehension problems, with minority group members more likely to report comprehension problems.
Nápoles-Springer AM, Santoyo J, O’Brien H, Stewart AL. Using cognitive interviews to develop surveys in diverse populations. Medical Care, 2006; 44(11 Suppl 3):S21-30.
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=17060830&query_hl=9&itool=pubmed_docsum
This article illustrates the importance of cognitive testing in assessing the conceptual adequacy of new or adapted self-report measures across ethnic groups in studies of health disparities. The authors demonstrate the use of two complementary cognitive testing techniques, interaction analysis and behavior coding, to evaluate a survey administered to an ethnically diverse population. Forty-eight cognitive interviews were completed, including 14 with African Americans, 20 with Latinos and 14 with Whites. Six Latinos were interviewed in English and 14 in Spanish. The mean age was 46 years and 58% were women.Results of cognitive testing permitted the identification of both general and ethnic group specific problems with wording and concepts, revealed the nature of the problems, and enabled decisions to modify or drop items based on respondents’ comments.
Oksenberg L, Cannell C, Kalton G. New Strategies for pretesting survey questions. Journal of Official Statistics. 1991;3:349-365.
This article demonstrates the need for more rigorous pre-testing methods of surveys and explores cognitive interviewing techniques involving behavior coding and probe questions. The authors detail the process of conducting both behavior coding and cognitive probes on a sample of 164 Michigan residents through a general telephone health survey. Participants were selected through a probability sampling of telephone numbers of residents of southeastern Michigan. Once the household was contacted the participant was selected to obtain an equal number of males and females in the final sample. The authors explore the benefits and limitations of using either behavior coding or directed probes independently, and suggest that using the two techniques together provides the researcher with the most systematic and objective method for evaluating the survey questions.
Presser S, Couper MP, Lessler JT, Martin E, Martin J, Rothgeb JM, Singer E. Methods for testing and evaluating survey questions. Public Opinion Quarterly. 2004;68(1):109-130.
The authors provide a detailed review of cognitive pre-testing including its history and development, supplemental techniques, experiments to test the usefulness of revisions made to surveys based on cognitive interview results, statistical modeling and different modes of data collection. The authors argue that the methods used to conduct cognitive testing influence the results, and thus developing optimal cognitive interviewing techniques is imperative. Directions for future research are explored in depth.
Rosal MC, Carbone ET, Goins KV. Use of cognitive interviewing to adapt measurement instruments for low-literate Hispanics. The Diabetes Educator. 2003:29(6);1006-1017.
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=14692374&query_hl=16&itool=pubmed_docsum
This article describes the use of cognitive interviewing to adapt measures for use in populations for whom they were not originally developed. Cognitive interviewing was used to adapt four instruments (the Audit of Diabetes Knowledge, the Insulin Management Diabetes Self-Efficacy Scale, and the Center for Epidemiological Studies-Depression Scale) for use with low-literate, Caribbean Spanish-speaking older adults with diabetes. The 5 participants in the study were Puerto Rican (3) and Dominican (2), aged 57-80 years, and 3 were female. All of the participants had 5 years or less of schooling. Adaptations included development of interviewer administered versions of these instruments since they were originally designed for self-administration, translation of measures into Spanish where necessary, modification of negatively worded items, deletion of items that were not culturally relevant, and clarification of response options to make them more concrete. Probes addressed the suitability of specific words used in items, the clarity of response options, and suggestions for alternate wording. The results of content analysis of the interviews were used to modify instructions and specific survey items for improved comprehension.
Sudman S, Bradburn NM, Schwarz N. Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco, CA: Jossey-Bass Publishers; 1996.
This book explores the methods that people use to answer questions, the problems that they have, and the implications that it has for survey based research. The authors explore the four steps in cognitive processing and identify possible sources of difficulty or measurement error at each stage including autobiographical event recall, estimation, frequency counts, ranking, effects of attitude or context on answers, and the influence of question ordering on subsequent questions. They provide in-depth examples and evaluations of several methods used to measure response error such as think aloud, cognitive experts, and behavioral coding. The implications of the cognitive processes involved in survey response for questionnaire design are discussed.
Warnecke RB, Johnson TP, Chavez N, Sudman S, O’Rourke DP, Lacey L, Horm J. Improving question wording in surveys of culturally diverse populations. Annals of Epidemiology 1997;7(5)334-42.
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=9250628&query_hl=18&itool=pubmed_docsum
The article reviews the theory of survey response and how differences in race/ethnicity lead to systematic measurement error in survey responses to standardized questionnaires. The authors performed cognitive testing on questions from several well established national health surveys (BRFSS, etc) in an ethnically diverse sample. The purposive sample of 423 adults included equal numbers of African Americans, Puerto Ricans, Mexican Americans, and non-Hispanic whites. They specifically evaluated how race and ethnicity influenced respondents’ performance of the four cognitive tasks involved in responding to survey questions: question interpretation, information retrieval, judgment formation, and editing. Differences were found by race/ethnicity for several of the stages of question response, suggesting that more work is needed to develop surveys that are valid across racial and ethnic groups.
Willis GB. Cognitive Interviewing. Thousand Oaks, CA: Sage Publications; 2005.
This book is a complete guide to using cognitive interviews to develop and refine questionnaires. The author explores the theory behind cognitive interviewing, specific techniques to conduct interviews including developing probes, training interviewers, recruitment of participants, and documenting and analyzing results. Other helpful sections include cognitive testing of sensitive questions and across age groups.
Updated March 2007 |
