Broader Impacts

1. Special Issue on the Advancing Research Impact in Society (ARIS) Broader Impacts Toolkit Project. Journal of Community Engagement and Scholarship. (Vol. 17, Issue 2; articles 1-14.)
   The ARIS Toolkit is a suite of digital documents and interactive web tools designed to provide guidance and effective practices aimed at improving BI outcomes and impacts for researchers and their collaborating partners.
2. Woodson, T., and Boutilier, S. (2022). Impacts for whom? Assessing inequalities in NSF-funded broader impacts using the Inclusion-Immediacy Criterion. Science and Public Policy (Vol 49, Issue 2).
   Introduces the new Inclusion-Immediacy Criterion, to assess who benefits from research impacts as divided into three groups: (1) advantaged groups; (2) the general population; and (3) marginalized groups.
3. Renoe, S. D. (2025). An insider perspective on broader impacts. BioScience (Volume 75, Issue 3).
   Gives a brief history of the broader impacts criterion and suggestions for how to effectively address it.
4. Advancing Research Impact in Society (ARIS). Resources.
   Provides a number of helpful resources focused on NSF's broader impacts criterion. Resources include Guiding Principles 2.0, a Broader Impacts Guidesheet.
5. Advancing Research Impact in Society (ARIS) Broader Impact BI Toolkit.
6. Advancing Research Impact in Society (ARIS), The Handbook of Broader Impacts. Forthcoming.

Assessment and Evaluation

Foundational, Field-Defining Publications

1. Patton, M. Q. (2014). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). Sage Publications.
   A foundational text introducing utilization-focused evaluation, emphasizing how evaluation should be designed and conducted to support real-world decision-making. Provides rigorous qualitative methods widely used in program and research evaluation.
2. Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). Sage Publications.
   A cornerstone reference for theory-driven evaluation, offering structured guidance on program design, causal inference, outcome measurement, and impact assessment.
3. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2010). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Pearson.
   Provides a comprehensive overview of major evaluation approaches (e.g., utilization-focused, participatory, theory-based), helping evaluators select methods appropriate to context and purpose, across academic, research, and public sector settings.
4. Chen, H.-T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd ed.). Sage Publications.
   A leading resource on theory-driven evaluation, demonstrating how to link program design, implementation, and outcomes through logic models and causal pathways.
5. Scriven, M. (1991). Evaluation thesaurus (4th ed.). Sage Publications.
   A seminal reference defining core evaluation concepts, terminology, and standards, widely used to ground evaluation practice and clarify distinctions across approaches.

Practice-Oriented Publications for STEM Grant Evaluation

6. Frechtling, J. A. (2010). The 2010 user-friendly handbook of project evaluation. (NSF Series Vol. 2, Issue 57). National Science Foundation, Directorate for Education and Human Resources, Division of Research and Learning in Formal and Informal Settings.
   A practical NSF guide for planning and implementing evaluations, including formative and summative evaluations, in STEM education and research grants.
7. National Science Foundation (2024). Proposal & award policies & procedures guide (PAPPG) (NSF 24-1) [Evaluation and broader impacts guidance]. nsf.gov.
   The authoritative policy document outlining NSF expectations for evaluation, accountability, and broader impacts, including how outcomes should be defined, measured, and reported.
8. National Science Foundation. (2022). Faculty Early Career Development Program (CAREER): Program solicitation NSF 22-586 [Solicitation].
   Defines expectations for integrating research and education, including evaluation of broader impacts, student development, and long-term scholarly contributions.
9. National Science Foundation. (2023). Research experiences for undergraduates (REU): Sites and supplements (NSF 23-601).
   Outlines evaluation expectations for REU programs, including student learning gains, mentoring quality, diversity participation, and STEM pathway outcomes.
10. W.K. Kellogg Foundation. (2004). W.K. Kellogg Foundation logic model development guide.
    A widely used, accessible resource for developing logic models that connect inputs, activities, outputs, and outcomes, forming the backbone of strong evaluation plans.

Higher-Education and Research Program Evaluation Resources Relevant to Rice Faculty

11. National Academies of Sciences, Engineering, and Medicine. 2017. Undergraduate Research Experiences for STEM Students: Successes, Challenges, and Opportunities. Washington, DC: The National Academies Press.
    A landmark consensus report synthesizing evidence on REU-style programs, including what outcomes to measure, how to measure them, and how program design influences student success and persistence.
12. American Association for the Advancement of Science. (2025, January 22). Guidebook: Preparing research and evaluation plans for NSF S-STEM grant proposals. S-STEM Resource and Evaluation Center.
    Provides step-by-step guidance on designing rigorous evaluation plans, including logic models, outcome tracking, and equity-focused approaches.
13. Weston, T. J. & Laursen, S. L. The undergraduate research student self-assessment (URSSA): Validation for use in program evaluation. CBE Life Sci Educ. (2015). Fall; 14(3):ar33. doi: 10.1187/cbe.14-11-0206.
    Presents a validated survey instrument widely used in REU and undergraduate research programs to measure student gains in skills, identity, and confidence.
14. National Science Foundation. (2023). Research Experiences for Undergraduates (REU) Program Solicitation & Evaluation Expectations (NSF 23-601).
    Defines gold-standard expectations for evaluation design, including mentoring quality, participant outcomes, and broader impacts, which is essential for competitive proposals.
15. Hunter, A.-B., Laursen, S. L., & Seymour, E. (2007). Becoming a scientist: The role of undergraduate research in students’ cognitive, personal, and professional development. Science Education, 91(1), 36-74.
    A landmark mixed-methods study demonstrating how undergraduate research contributes to scientific thinking, identity development, and career trajectories, informing outcome selection in evaluation design.

Graphics

1. Scott McCloud. Understanding Comics: The Invisible Art. William Morrow, 1994.
2. Edward R. Tufte. Visual Explanations: Images and Quantities, Evidence and Narrative. 6th Ed. Graphics Press, 2003.
3. Richard Williams. The Animator's Survival Kit: Runs, Jumps, and Skips. Faber & Faber, 2021.

Team Science

1. Bisbey, T. M., Wooten, K. C., Campo, M. S., Lant, T. K., & Salas, E. (2021). Implementing an evidence-based competency model for science team training and evaluation: TeamMAPPS. Journal of clinical and translational science, 5(1), e142.
   Develops an evidence-based competency model designed specifically for science teams, called TeamMAPPS, in response to calls for identifying key knowledge, skills, and attitudes underlying effective collaboration. Through a review of team and team science literature, the authors identify three core competency domains: awareness and information exchange, psychological safety, and self-correction and adaptation.
2. Vogel, A. L., Knebel, A. R., Faupel-Badger, J. M., Portilla, L. M., & Simeonov, A. (2021). A systems approach to enable effective team science from the internal research program of the National Center for Advancing Translational Sciences. Journal of Clinical and Translational Science, 5(1), e163.
   Describes a systems-level approach used by the National Center for Advancing Translational Sciences (NCATS) to support cross-disciplinary team science within its internal research program. Outlines how organizational policies, structures, and processes (such as performance evaluation criteria, co-located workspaces, shared resources, fluid team assembly, and project management) are designed to facilitate collaboration and team functioning.
3. Lotrecchiano, G. R., DiazGranados, D., Sprecher, J., McCormack, W. T., Ranwala, D., Wooten, K., ... & Brasier, A. R. (2021). Individual and team competencies in translational teams. Journal of Clinical and Translational Science, 5(1), e72.
   Identifies core individual and team competencies required for effective translational science teams using a modified Delphi method involving expert consensus. Defines translational teams as cross-disciplinary, interprofessional groups working toward shared objectives related to improving human health, and develop a framework consisting of five competency domains.
4. Hall, K. L., Vogel, A. L., Huang, G. C., Serrano, K. J., Rice, E. L., Tsakraklides, S. P., & Fiore, S. M. (2018). The science of team science: A review of the empirical evidence and research gaps on collaboration in science. American psychologist, 73(4), 532.
   Reviews empirical research in the science of team science (SciTS), synthesizing findings from 109 studies published between 2006 and 2016. Identifies five major themes in the literature: the value of team science, team composition, team formation, team processes, and institutional influences on collaboration.
5. Begerowski, S. R., Traylor, A. M., Shuffler, M. L., & Salas, E. (2021). An integrative review and practical guide to team development interventions for translational science teams: one size does not fit all. Journal of Clinical and Translational Science, 5(1), e198.
   Provides an integrative review of team development interventions (TDIs) for translational science teams, focusing on practical approaches to improving team effectiveness. Identifies common barriers to effective collaboration across individual, team, and organizational levels, including challenges related to communication, coordination, and interdisciplinary integration.
6. Borner, K., Contractor, N., Falk-Krzesinski, H. J., Fiore, S. M., Hall, K. L., Keyton, J., ... & Uzzi, B. (2010). A multi-level systems perspective for the science of team science. Science translational medicine, 2(49), 49cm24-49cm24.
   Describes recent developments in the science of team science (SciTS), an emerging field focused on understanding how scientific teams organize, communicate, and conduct research. Highlights the increasing prevalence of team-based research and define SciTS as an area concerned with examining collaborative processes and outcomes across scientific teams.
7. Bolduc, S., Knox, J., & Ristroph, E. B. (2023). Evaluating team dynamics in interdisciplinary science teams. Higher Education Evaluation and Development, 17(2), 70-81.
   Examines how evaluation approaches can better capture the dynamics of interdisciplinary research teams through a case study of a five-year externally evaluated research program. Uses a mixed-methods evaluation design that includes surveys, interviews, and participation in team activities to assess both developmental processes and outcomes over time.
8. Schwarz, R. M., & Bennett, L. M. (2021). Team effectiveness model for science (TEMS): using a mutual learning shared mindset to design, develop, and sustain science teams. Journal of Clinical and Translational Science, 5(1), e157.
   Introduces the Team Effectiveness Model for Science (TEMS), a normative model designed to explain and guide how science teams can be developed and sustained. Shows how the model centers on a "mutual learning" (ML) shared mindset, which links team values and assumptions to behaviors, processes, structures, and outcomes.
9. Mendell, A. M., Knerich, V., Ranwala, D., Jones, C. T., Piechowski, P., Striley, C. W., ... & Cross, J. E. (2024). Team science competencies across the career life course for translational science teams. Journal of clinical and translational science, 8(1), e111.
   Examines how individual and team competencies in translational science (TS) teams vary across the career life course and across three constituency groups: trainees and faculty, clinical research professionals (CRPs), and community partners. Using a modified Delphi approach, builds on an existing competency framework to assess the relevance and level of mastery of competencies at different career stages.
10. Stvilia, B., Hinnant, C. C., Schindler, K., Worrall, A., Burnett, G., Burnett, K., ... & Marty, P. F. (2011). Composition of scientific teams and publication productivity at a national science lab. Journal of the American Society for Information Science and Technology, 62(2), 270-283.
    Investigates how the composition of scientific teams affects their publication productivity using data from 1,415 experiments conducted at the National High Magnetic Field Laboratory (NHMFL) between 2005 and 2008. Analyzes 89 teams. Concludes that highly productive teams tend to have high disciplinary diversity, lower seniority diversity, and strong cohesion. Also highlights that team composition involves trade-offs between diversity and coordination, suggesting the need for balanced team structures in scientific collaboration.
11. Specht, A., & Crowston, K. (2022). Interdisciplinary collaboration from diverse science teams can produce significant outcomes. PLoS One, 17(11), e0278043.
    Examines how diversity in scientific teams influences research outcomes using a mixed-method study of 22 working groups. Finds that team diversity, particularly disciplinary diversity, positively affects publication output and citations through increased interdisciplinarity in the research process. However, also finds that greater diversity is associated with lower team satisfaction and perceived effectiveness, reflecting challenges in collaboration. Identifies factors such as trust, leadership, and team processes as important for supporting effective collaboration in diverse teams.
12. Kelly, P. W., Chladek, J., & Rolland, B. (2023). Toward a translational team science hierarchy of needs: exploring the information management challenges of team science. Journal of Clinical and Translational Science, 7(1), e210.
    Examines how translational research teams manage information and how these practices affect team functioning and scientific progress using qualitative interviews with 10 team members. Identifies institutional barriers and a lack of cohesive strategies as key challenges, while also showing that shared approaches emphasizing transparency, accountability, and trust improve team functioning. Introduces a Translational Team Science Hierarchy of Needs, proposing that effective information management is a foundational requirement for developing strong team processes and achieving high levels of team performance.
13. O'Kane, C., Mangematin, V., Zhang, J. A., & Haar, J. (2024). How research agendas are framed: Insights for leadership, learning and spillover in science teams. Research policy, 53(7), 105029.
    Shows that these approaches differ in how research agendas are coordinated, adapted, and transformed, as well as in how leadership and team learning environments operate. Also describes how research agendas evolve through initial framing, reframing, and eventual transformation into new directions or spin-off projects.
14. Nearing, K., Rainwater, J., Neves, S., Bhatti, P., Conway, B., Hafer, N., ... & Wasko, M. (2021). I-Corps@NCATS trains clinical and translational science teams to accelerate translation of research innovations into practice. Journal of clinical and translational science, 5(1), e66.
    Investigates the implementation and evaluation of the I-Corps@NCATS training program, which aims to equip clinical and translational science teams with entrepreneurial skills to accelerate the translation of research into practice. Describes a 5-week training program delivered to 62 teams across multiple institutions, focusing on customer discovery, value proposition development, and business model design.
15. Wuchty, S., Jones, B. F., & Uzzi, B. (2007). The increasing dominance of teams in production of knowledge. Science, 316(5827), 1036-1039.
    Examines trends in scientific collaboration by analyzing 19.9 million research articles and 2.1 million patents across multiple disciplines over several decades. Finds a widespread shift from solo authorship to team-based research, with increases in both the proportion of team-authored work and average team size across sciences, social sciences, humanities, and patents.
16. Fiore, S. M. (2008). Interdisciplinarity as teamwork: How the science of teams can inform team science. Small group research, 39(3), 251-277.
    Examines interdisciplinary research through the lens of teamwork, arguing that interdisciplinary science is fundamentally a team-based activity. Defines distinctions between cross-disciplinary, multidisciplinary, and interdisciplinary research, emphasizing that true interdisciplinarity requires the integration of concepts, methods, and perspectives to create new approaches and knowledge.
17. Salas, E., Cooke, N. J., & Rosen, M. A. (2008). On teams, teamwork, and team performance: Discoveries and developments. Human factors, 50(3), 540-547.
    Provides a comprehensive review of major discoveries in the science of team performance over several decades. Synthesizes key insights, including the importance of shared cognition, advances in team training, and the role of measurement and modeling in understanding teamwork. Emphasizes that team performance is a dynamic, multilevel process shaped by interactions among cognitive, behavioral, and contextual factors.
18. Kozlowski, S. W., & Ilgen, D. R. (2006). Enhancing the effectiveness of work groups and teams. Psychological science in the public interest, 7(3), 77-124.
    Reviews over 50 years of research on work groups and teams to identify key processes that contribute to team effectiveness. Conceptualizes teams as dynamic, multilevel systems embedded in organizational and environmental contexts, and emphasize cognitive, motivational, and behavioral processes as central mechanisms. Organizes the literature around team processes and emergent states, as well as interventions that can shape these processes to improve effectiveness.
19. Mathieu, J. E., Hollenbeck, J. R., Van Knippenberg, D., & Ilgen, D. R. (2017). A century of work teams in the Journal of Applied Psychology. Journal of applied psychology, 102(3), 452.
    Reviews the historical evolution of work team research over the past century, with a focus on publications in the Journal of Applied Psychology. Traces a shift from early research centered on individual-level processes and comparisons between individuals and groups to more recent work emphasizing teams as collective, multilevel systems embedded within broader organizational contexts.
20. Salas, E., Reyes, D. L., & McDaniel, S. H. (2018). The science of teamwork: Progress, reflections, and the road ahead. American Psychologist, 73(4), 593.
    Synthesizes key insights from decades of team research and highlights major reflections emerging from the literature on teamwork. Identifies core themes, including the development of multilevel theories, the importance of context, the role of psychological safety, and the identification of transportable teamwork competencies such as coordination, communication, and adaptability.