Key points
Co-designed built interventions make an impact through their physical outputs and the processes that lead to their implementation.
The value for money of co-designed built interventions must be assessed by considering the multiple short- and long-term impacts, including the social impact of the process itself.
Allocating adequate time and resources for evaluating interventions and their long-term impact can contribute to learning.
Children and their communities must be part of the evaluation process.
This handbook has presented co-designed built interventions as complex processes involving multiple pathways for generating positive impacts. These can be broadly divided into two areas:
- Impact of the product: Impact generated by the physical features of the building itself;
- Impact of the process: Impact generated by the process leading to the implementation of the built intervention.
Impact of the product
As presented in part A, appropriate physical features such as colours, openings, and the height of furniture can impact children’s wellbeing. The built intervention itself and its aesthetic value can also positively impact on the local community. For example, a good quality building that is a source
of pride for the community can improve the self-esteem of its users and the community in which it is located. A properly constructed building can allow key educational activities to take place in a safe and conducive environment.
Impact of the process
As discussed, the process that leads to the implementation of a built intervention may generate positive impacts in multiple ways. In addition to the impact on children, caregivers and those involved in the process of building can generate a positive impact for the community. For example, choosing local materials and hiring and training local labour can improve the local economy and the livelihoods of vulnerable individuals.
Both types of impact can be measured in the short term immediately after project implementation, and in the long-term.
Drain et al. (2018) suggest a framework for the evaluation of participatory design projects based on three types of outcomes:
Insights:
the knowledge gained by the design team that can be leveraged for positive impact in the future;
Solutions:
the outcomes generated through the creation of material things (physical artefacts) and organisation, rules, and information flows (process changes);
Empowerment:
this can take many forms, including political empowerment in the workplace, creative capacity building, social empowerment for persons with disabilities, and technical training.
The evaluation should assess if outcomes in these three areas are aligned with the values underpinning a participatory design project.
Challenges for measuring long-term impact
Measuring long-term impact requires long-term engagement with the project and its location. Built interventions, especially in displacement contexts, often do not have funding allocated for evaluating long-term impact. This is partially due to the fact that built interventions are often conceived as physical responses to the need to create a space, hence the perceived impact is limited to having provided such space. In many cases, funding for a built intervention is project-based for a maximum of one to two years. Therefore, it is often difficult to allocate funds to carry out impact evaluations after the end of the project cycle, which frequently coincides with the end of the construction.
Despite these challenges, all of the actors involved can learn a lot from evaluations carried out long after the end of the project. An evaluation does not need to be time consuming or expensive. Simple evaluation processes can provide valuable feedback, which can be integrated into future projects. Evaluations that compare pre- to post-project implementation are particularly effective but require data collection before implementation (baseline).
A participatory approach
Both qualitative and quantitative data can be collected and evaluated for built interventions. Evaluating built interventions can involve feedback from users, which can be collected via interviews or questionnaires; observing the users; and by approaching children, caregivers, those who work in the interventions (e.g. teachers, social workers), and those who worked on the construction.
Using open questions may help identify unexpected impacts and the pathways through which the built intervention, and the process leading to it, have had an impact on different aspects of children’s and other residents’ wellbeing. For example,
- Did the project create a better space for a particular community/children’s needs?
- Do children behave differently?
- Has school attendance changed?
- Do caregivers feel differently about children playing in a playground or attending school? Have they seen positive changes in the children as a result of their participation and use of the intervention
- Have the workers found better employment opportunities with the new skills acquired
- Have new construction techniques been adopted locally in other construction projects
- How have relationships/interactions between different communities (e.g. different groups of refugees, refugees and hosts) changed?
In the spirit of participation, it is important to involve evaluation participants in the process of analysing the results. This is particularly important when looking at attribution of impact. Some changes can be linked to the intervention, while participants are more likely to be aware of the broader issues affecting their context which should also be considered. For example, a high-quality school built with children’s input may improve attendance, but attendance could also be affected by a safer neighbourhood, a school feeding programme, or a cash transfer programme conditional on school attendance. Making sense of unexpected and unintended impacts also requires evaluation participants.
Evaluations can also be conducted with children, and there are available tools to help practitioners achieve this. If questionnaires are used, the language needs to be simple and age appropriate.
Example of questions for children:
- Can you tell us about your typical day playing?
- Can you tell us how you were playing last year?
- What has changed in the way you play since this playground / school was built?
- What do you do in this place/building?
- What do you enjoy the most?
- What do you find challenging?
Value for money
The impact of co-designed built interventions on the wellbeing of children and their communities goes beyond the impact of the built outputs and the new activities they allow (see for example steps 7 & 9). However, particularly in humanitarian contexts, built projects are often rushed and quickly assessed only on their physical output. This approach fails to recognise the deeper, longer term impacts of co-designed interventions. With the increasingly protracted nature of displacement, interventions for children affected by displacement should adopt a more comprehensive evaluation framework.
Built interventions that are co-designed with children affected by displacement can offer excellent value for money, but only if the impact of the whole process is taken into consideration, and a longer time frame is adopted. For example, the benefits of the community having increased ownership of the intervention, and the improved local capacity for carrying out maintenance, show themselves only after the inauguration of a new built project. Sometimes budget constraints are used to justify interventions for children with limited or no participation. However, it is better to build something smaller through participatory design that empowers children and their community, as described in this handbook, than it is to create an intervention that children and their community do not need, that is in the wrong location, or that may not be fully used or looked after.
Proper evaluation processes that include assessments of long-term impacts are needed to demonstrate to funders and other stakeholders the value for money of co-design approaches. Moreover, given the complexity of these interventions, the need to adapt them to specific contexts, and the changing nature of displacement conditions, continuous learning from different projects and the actors involved is required. Such learning processes – of which this handbook forms part – should be based on rigorous and comprehensive evaluations to provide evidence for the continuous improvement of interventions for the benefit of children affected by displacement. Therefore, actors and funders should invest sufficient resources into project evaluation.
