Measuring Societal Impact for Convergent Research

Resource: Explainer
Description
Three people seated with their backs to the camera with one woman standing facing them as they engage in a discussion
Photo Credit: SESYNC

A downloadable version of this explainer is available here: 

Explainer – Measuring Societal Impact.pdf (1.2 MB)

Convergent research is research that has societal relevance, leads to outcomes that are useful (i.e., actionable), and requires integrative interdisciplinary team research. As environmental challenges continue to gain importance in community, government, and industrial contexts, there is a growing emphasis on accurately measuring the positive social outcomes of research. A research enterprise’s value is in the eyes of the stakeholders, and measuring it is difficult because evidence of the research’s impact may take years to emerge. While there are no universal, high-quality metrics for assessing socio-environmental (S-E) research’s societal impact and research on the subject is ongoing (Dotti and Walczyk 2020), we provide some options for measuring it below.

Stakeholder Value Propositions and Design Thinking

A critical first step for ensuring societal impact is for research teams to create value propositions for each stakeholder associated with the project, as summarized in Westerhoff et al. (2021). Different stakeholders will benefit (or avoid harm) in different ways. Establishing value propositions is related to design thinking, which is an interactive team process of exploring a problem, re-thinking or re-structuring it with those who need solutions, and identifying potential research paths forward. This process begins with having empathy for the user—that is, thinking first about stakeholders’ needs, challenges, and motivations. Creating value propositions is the first phase of using a design-thinking approach in which teams move between creative ideation, emphasizing brainstorming, and more structured research frameworks, like the use of flow charts. There are many tools and activities to begin implementation of design thinking, one of which is co-development of a mental model with the stakeholders. After this stage, researchers will have a clearer view of the type of insights and solution sets that stakeholders will find most valuable and implementable.

Logic Models, Theory of Change and “Horizons of Accountability"

It is important to consider the likely timeline (“lag”) for a project’s impact and the types of external factors that can mediate impact. This planning will help in the development of an assessment process, and it will reveal potential limits of project-impact attribution—also called “horizons of accountability.” There are a variety of impact “levels,” from individual to local to global, some of which are difficult to demonstrate in the short term or difficult to demonstrate if other external factors also contribute to societal impact.

You cannot expect to take full credit for broad societal impact since it is always influenced by a variety of factors other than your contributions. A useful approach to begin identifying potential impacts is to use a logic model or theory of change (sometimes called program theory) to detail the project’s goals from the highest to lowest levels and identify indicators of change or impacts at each level. Each level should have associated strategies for accomplishing those goals. SESYNC clearly identified their horizon of accountability in Figure 1 below for societal impact (listed to the right of flow chart) of their theory of change, as well as indicators of each impact level (boxes within the chart) and strategies (in the bottom-most “Core Objectives” box) for accomplishing impacts. Note that SESYNC’s impacts were very specific to its mission.

Theory of Change
Figure 1. This schematic of SESYNC’s “Theory of Change”(TOC)  provides an overview of the core strategic objectives and impacts at multiple stages in the life of the Center. These impacts range from short-term effects and outputs that are not difficult to measure all the way to longer-term, broader societal impacts, many of which are very difficult to measure and can never be fully attributed to any one factor (i.e., above the “horizon of accountability” for SESYNC alone).  The TOC was pivotal to SESYNC planning—helping the Center to adapt programs and processes based on both internal and external (independent) evaluative and summative assessments.

Impact Metrics: Outputs vs. Societal Outcomes

Traditional Academic Output Measures

Most measures of impact are actually more correctly referred to as outputs because they represent products rather than societal impact. These outputs are still important measures, especially for funding agencies to value data on productivity and engagement of other researchers. Examples include altmetrics, which monitor the scope and impact of research on other researchers using online analytics, including citations, journal prestige, and the author’s H-index (metric for measuring an author’s output and impact). But systematic evidence to support altmetrics’ efficacy does not yet exist (Thewall 2020; Dardas et al. 2023), and there is no evidence that scholarly impact reflects societal impact (Bormann 2013). When altmetrics are not used, traditional, individual academic output metrics include: number of publications, presentations, products, patents; students and postdocs trained; number of disciplines engaged; number of “hits” in the media or website clicks; etc.

Societal Relevance and Impact Measures

As indicated above, all impact measures have limitations, but some come closer to indicating real societal impact—meaning changes in behaviors, thinking, or doing. Direct evidence of research results’ impact—such as measures of changes in behavior, health, and quality of life—are the best but hard to obtain. Examples of direct evidence would be:

  • Disseminated research results that show lower agricultural water use; reduced risk of disease or illness; more efficient energy use; or reduced pollution.
  • Written or recorded policies or practices by organizations or actors (agencies, institutions, civil society, leaders, businesses, etc.) outside the university sector that indicate that the research products were used or were influential in ways that benefit members of society.
  • Evidence of stakeholder engagement with research teams that has been beneficial for the stakeholders or that shows how insights arising from stakeholders have influenced the research teams
    • This evidence might present itself during interviews by external evaluators; from surveys; from invitations to engage in dialogue with stakeholders.
  • Capacity building, such as evidence of education, training, or development of collaborative abilities that resulted from the research activities 
    • It's usually assessed with pre or post surveys or testimonials.
    • Case studies are also often quite useful.

Direct evidence will only apply to some projects and typically takes a long time to appear.

Additional Resources

  • Bornmann, L. (2013). What is societal impact of research and how can it be assessed? Journal of the American Society for Information Science and Technology, 64(2), 217–233. https://doi.org/10.1002/asi.22803
     
  • Dardas, L.A., Sallam, M., Woodward, A. et al. (2023). Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive. Publications, 11(1), 5. https://doi.org/10.3390/publications11010005
  • Dotti, N.F., & Walczyk, J. (2022). What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors. Evaluation and Program Planning, 95, 102157. https://doi.org/10.1016/j.evalprogplan.2022.102157
     
  • Newig, J., Jahn, S., Lang, D.J. et al. (2019). Linking modes of research to their scientific and societal outcomes. Evidence from 81 sustainability-oriented research projects. Environmental Science & Policy, 101, 147-155. https://doi.org/10.1016/j.envsci.2019.08.008
     
  • Ramos-Vielba, I., D’Este, P., Woolley, R. et al. (2018). Introduction to a special section: Balancing scientific and societal impact—A challenging agenda for academic research. Science and Public Policy, 45(6), 749-751. https://doi.org/10.1093/scipol/scy033
     
  • Rau, H., Goggins, G., & Fahy, F. (2018). From invisibility to impact: Recognising the scientific and societal relevance of interdisciplinary sustainability research. Research Policy, 47(1), 266-276. https://doi.org/10.1016/j.respol.2017.11.005.
     
  • Thelwall, M. (2021). Measuring societal impacts of research with altmetrics? Common problems and mistakes. Journal of Economic Surveys, 35(5), 1302-1314. https://onlinelibrary.wiley.com/doi/full/10.1111/joes.12381
     
  • Tschimmel, K. (2012). Design Thinking as an effective Toolkit for Innovation. Proceedings of the XXIII ISPIM Conference: Action for Innovation: Innovating from Experience. https://doi.org/10.13140/2.1.2570.3361

 

Authors
Date
Share

Related Content