In 2013, 11 outcome‐driven National Science Challenges were established by the New Zealand government to address the country’s biggest and most challenging issues e.g. health care, nutrition, biodiversity, childcare, housing, natural hazards, climate change and natural resource management. Funding was approved for ten years between 2014 and 2024 with a mid‐term review in 2018. The expectation of the Challenges is to generate impact through working in more inter‐ and trans-disciplinary ways. Given these expectations, the midway point for the Challenges presented an opportunity for reflection. With its focus on outcomes, the was identified as useful for assessing the potential for impact of the Challenges.
Proposed innovations to OSF
Two innovations to OSF were identified as useful for assessing the potential for research impact by the Challenges. These have been called OSF+.
Innovation 1: Separating knowledge stocks and flows
We proposed a separation of knowledge stocks and knowledge flows. Identifying the relationship between, and resources dedicated to, knowledge stocks (i.e. doing research) versus knowledge flows (i.e. making the research available to those that need or want to use it) were identified as useful indicators of potential for science impact as it would bring into view worldviews that influence decisions about where resources are to be dedicated, (i.e. not only producing knowledge but also outcomes for making it meaningful and useful).
Innovation 2: Trialling a method of identifying intended, actual and desired outcomes
Because the Challenges were all at a half-way point in 2018, we wanted to see how the programmes changed over time.
Participants were first asked to map their initial expectations (intended project outcomes), and then to map the actual state of what the Challenges were delivering at the midpoint (actual outcomes). Finally participants were asked to map expectations for the remainder of the programme, to meet the project ambitions (the desired outcomes). This final stage included a reflection on reasons for any differences between intended and actual outcomes.
To gauge potential impact, we hypothesised it would be useful for workshop participants to see how the OSF+ profiles changed across these three phases of the programme.
Our research involved a collaboration between the authors to develop OSF+ and a one‐day workshop with four representatives from 4 National Science Challenges. The workshop was jointly facilitated by the authors.
As well as testing the above innovations, several insights on applying OSF+ emerged from the workshop. Given the large‐scale, systemic and often slow‐moving nature of the issues the Challenges seek to address, the project boundary was interpreted temporally and highlighted the need for Challenges to plan for a legacy beyond the life of their programmes. Indeed, we found that where participants placed their outcome spaces in relation to the project boundary revealed quite different theories of change amongst the Challenges.
Participant reflections on OSF+
Participants were asked how useful they thought OSF+ could be for reflecting on and potentially evaluating the impact of their Challenge. All participants were very positive about OSF+. Features noted were:
- its synergy with their performance frameworks
- the way OSF+ forces thinking about what transdisciplinary means and how it could
- be applied in practice
- the way it focuses attention on where effort should or could be directed
- how effort contributes to overall outcomes of transdisciplinary research
- the way it makes explicit decisions on resources and trade‐offs
- how it can reveal implicit theories of change that exist behind where effort is placed across the outcome spaces.