Usually organizational blogs, especially blogs by civic tech organizations, focus primarily on sharing successes. However, we have found that we actually learn far more from the failures of others than from their successes.

So, we created this series to share our areas for growth, in hopes that you can learn just as much from our process as from our successes. This series was developed from ongoing conversations by a diverse group of 18F members, including both term-limited and career employees who have varying levels of experience and tenure (e.g., freshly arrived to more than 5 years). We joined together to talk about ways to improve our techniques, approach, dynamics, and strategy. Part 2 of our series is about mixed methods.

It is natural for design researchers to use qualitative methodologies in their work. It is what they are trained to do as they attempt to answer questions about needs, values, emotions, desires, problems, and complexities. It is also natural for a data scientist or statistician to use quantitative methodologies to answer questions about statistical trends, how often an event occurs, or how many people experience a phenomenon. We have learned, however, that there is power in bringing the two methodologies together for a mixed methods approach.

Mixed methods is an approach in the social sciences in which you gather both quantitative and qualitative data in an effort to make more informed and integrated interpretations based on the combined strengths of both types of data. This approach allows a researcher to triangulate information to create a rich, more complete narrative of an experience or situation. Beyond understanding a person’s experience with a service or product, mixed methods also help to identify repeated patterns and to what extent this pattern is demonstrated among the people using a service or product. Lastly, this approach provides a clear way to connect the value of a service or product to the human users (which is often qualitative and sometimes quantitative) to the value it has to an organization (often quantitative).

When surveying our projects, it’s clear that we could use a mixed methods approach more often. There are definitely projects that may not have needed it. There are also projects that could have used it. By adding quantitative research to the qualitative research that was already done, the project’s interpretation and recommended strategy may have been further validated. However, it also may have changed, colored, or tweaked the findings in such a way that the final recommendations took a slightly or largely different direction. Using mixed methods allows us to gather information at varying scales, depths, and types in order to form a holistic picture to better inform our design choices.

Three Approaches

There are three high-level general approaches to employing mixed methods, which are: a convergent approach, an explanatory approach, and an exploratory approach. The level of intention varies between the options.

In a convergent approach, we create research goals and then research questions. From the beginning, we intentionally plan both qualitative research and quantitative research based on our questions. We look for the similarities and differences between the quantitative and qualitative data, and use the insights to generate a report showing our findings and recommendations.

In an explanatory approach, we also create research goals and research questions. However, here we plan a quantitative study based on our questions in order to look for patterns among the quantitative data of relatively large numbers of people. After studying this, we may have new questions about why something is happening or about the emotions or needs of the people experiencing a particular event. To answer these questions and explain what we are seeing, we design a qualitative study to complement our quantitative work. When we report our findings, the quantitative data is enriched by the explanatory qualitative insights.

In an exploratory approach, our research questions lead us to plan a qualitative study to consider the experiences and stories of people who use or could use a service or product. Through synthesis of qualitative research, we develop hypotheses that we can explore by planning an experimental quantitative study, usually with a larger group. We then report on our qualitative findings and quantitative results.

Call for Help

It’s not necessary that every design researcher be able to conduct both qualitative and quantitative research. What is helpful is every design researcher being able to recognize the need for quantitative research or a mixed methods approach, and to know when to invite the help of a data scientist. Using a mixed methods approach and working with a data scientist on quantitative analysis can ultimately increase the research’s validity and uncover insights that provide a more complete view of what is happening. In the end, the public who use our services and products will benefit.