Kristen Miller, the director of the CDC’s Question Design Research Lab, will be at DePaul next week sharing her survey know-how with anyone who wants to learn more about how survey research on a grand scale operates on the ground. Check out the schedule below and join us at the SSRC for a promising display of survey and methodological insights and derring-do.
Friday, February 10, 1 pm: Faculty Seminar
“Development and Evaluation of a Sexual Identity Measure for the National Health Interview Survey (NHIS)”
Miller will describe the use of qualitative research in developing a precise sexual identity measure for a large-scale quantitative survey and the resulting complications.
Monday, February 13, daytime: Lab Visits
Faculty are invited to schedule appointments to meet with Miller to discuss their research, questionnaire design, or other research questions.
Monday, February 13, 6 – 7:30 pm: Public Lecture
“Question Evaluation at the National Center for Health Statistics”
This lecture, open to the public, will center on Miller’s work at the CDC and will consider examples of questions that inadvertently compromised data quality through a lack of rigorous evaluation.
I talked with Kristen today to learn more about what she does and why it matters.
What do you do at the Question Design Research Laboratory? What does an ordinary day look like?
KM: We’re the National Center for Health Statistics, we’re the main data collectors for the Department of Health and Human Services. That’s what our agency does, and it’s through surveys. My staff, the QDRL, is a consulting piece of NCHS.
Basically what we do is we evaluate questions and we develop methodologies to be able to evaluate questions. Specifically, when I talk about evaluating questions, I’m talking about validity, and are the questions really capturing what analysts believe that they are capturing.
What are some of the methods that you employ to evaluate any particular question?
KM: The main thing we do here, and the main thing that the Federal Statistical System uses, is called cognitive interviewing. First you ask them the survey question, they give their answer, then you interview them to figure out what they were thinking and why they answered the way they did.
From that then, you do your analysis and you can figure out specifically what the constructs are that the questions are capturing.
So it’s an interesting combination of using qualitative methods to improve a quantitative methodology.
KM: Absolutely, and that’s very much how we see it. At this point, if you go into the literature and you look at cognitive interviewing methods, you’ll find that the literature is rather [sparse]. What we’re working on is applying more of a sociological perspective, and then pulling in the qualitative literature. So we very much talk about Glaser and Strauss, doing grounded theory, and that sort of thing.
When you are looking at any particular survey, what are the 3 most frequent critiques that you at the QDRL have for survey design? What do you suggest to people most often to improve questions?
KM: I really can’t answer that question. Where we’re coming from is that no one can write a survey question. No one knows how it’s going to be taken. Pulling in our sociological background, people from different social locations can really interpret questions differently. If you understand it that way, one single person can’t sit there and write a question and know how everybody’s going to understand it. So what we believe is that the only way to realy do that is to perform these qualitative studies and really find out how people from all of these different social locations go about understanding questions. Our main issue is that you can’t just sit there and critique a question and change a question. You need to actually study how it performs. Only after you study how it performs, is when you can start tinkering with questions.
Because we do population surveys, we started thinking about all the different types of minority groups. I think that the biggest group that we have problems with are low-socioeconomic-status (SES) people. These are the people who really have difficulty understanding a question as question designers intended it. It makes sense, right? Because there are people with Ph.D.’s sitting in Washington writing these questions with specific intentions—”Do you have coronary heart disease?” Well, people who don’t have good health care, they’re not going to know what that is. We’ve done studies in rural Mississippi and people will say, “No, I don’t have that,” and you’ll look at the guy’s health record and you’ll see that not only does he have coronary heart disease but he’s got congestive heart failure. I’m really concerned that the federal statistical system is not getting equal quality data, and that’s one of my most passionate subjects, and I try to shed light on that and to improve it.
What specifically are some of the dangers that you see from a survey that doesn’t take seriously some of these design considerations, that isn’t anticipating the different kinds of groups that are going to be responding to the survey? What’s the big danger here?
KM: There’s multiple dangers. The big danger is that you’re not collecting the data that you think you’re collecting and that there’s actually bias in the data that you’re collecting. Some groups are under-reporting, some groups are over-reporting, it’s just like what I talked about. But I also think that this leads to ethical issues. We’re the federal government, we’re supposed to be democratic, we’re supposed to be making sure that not one group is privileged over another group. And I also think that it’s an obligation for survey methodologists to be able to say, or to prove, or to give evidence that they are indeed capturing what they say they are capturing. I think it should be a best practice in survey research to be able to say, “This is what this measure gets and this is how I know.” And that’s just not done.
And so we have these dual concerns of both empirical validity, but also a democratic ethical obligation.
KM: Correct. And you know, there’s all these standards for surveys to show their response rate, sampling error, and go through all of these other issues related to survey research, but it has not been addressed regarding item-response error and validity. It’s such a huge piece of the puzzle.
I know that you’ll be talking (at DePaul) about your work designing a measure for sexual identity for national surveys. Could you talk a little bit about what some of your concerns were?
KM: We were having huge problems getting good measures for that question. We looked in our own surveys and asked about it, but even if you looked internationally, there were so many people that would say “Don’t know” or there was non-response. People didn’t know these terms. People didn’t know what heterosexual meant. People didn’t know what homosexual was, so they just wouldn’t answer. And what was happening, and what does happen, is that analysis would just go in and lop off the missings and then look at the data from there. But what was happening was that the missings were not random, they were largely low-SES people and Hispanics. And so when you’re doing a cross-tab of obesity and sexual identity, what one of our surveys was finding was that lesbians were more likely to be obese than any other sexual identity group. Well, what was happening was that they really weren’t. The next year, we actually fixed the question and it turned out that lesbians were actually the least likely to be obese. Because what was happening was low-educated Hispanic heterosexual respondents, so you’re taking a particular group out of the analysis. So what I’ll be talking about are all the design problems we knew about and our understanding of the response errors that we were seeing, and how we made a question, tested the question, and this is the question that is in the field now.