Evaluation, particularly in relation to energy solidarity, may be new territory for energy communities (ECs). It is vital to demonstrating success and can support financing.
Academics from the University of Birmingham led the evaluation aspect of the CEES Project and are preparing extensive reporting. Within the Energy Solidarity Toolkit, they provided some fundamentals to get ECs started. While not aiming to be exhaustive, the following material clarifies some key principles and approaches to think about in evaluation.
Ethics in evaluation
Respect for project participants is core to ethics in evaluation. As such, ECs must invest time in developing a robust ethical framework that appropriately considers the following elements.
- Anonymity: At no point in the project reporting should it be possible to identify participating individuals.
- Informed consent: Any individual from whom an EC wants to collect data must be fully informed of what data will be collected, who it will be used by and for what purpose(s). This is especially critical for participating households but also applies to colleagues and external partners. Preparing a clear, concise agreement form on which the participants’ agreement can be recorded is vital.
- Data management and related systems: Protecting data, especially personal data (anyone’s personal data must be protected, not only that of vulnerable people), may also be new to ECs. An appropriate strategy must first comply with EU’s General Data Protection Regulation (GDPR) and/or any locally relevant frameworks. It should also set out protocols for data collection, storage, sharing and eventual disposal.
Five fundamental evaluation approaches
Ideally, evaluation will include efforts to assess both the processes applied and whether the intended impacts were achieved. In reality, these evaluations often overlap or have blurred lines. Evaluation undertaken during the project (formative) can inform whether some adaptations may be needed to achieve desired outcomes while after-the-fact evaluation (summative) can help determine what might be done differently in a next iteration of the project. To optimise their usefulness, both formative and summative evaluations should consider impacts and processes.
- Impact evaluation aims to understand – across all groups involved – the change produced by the project. Typically, it is carried out by asking participants to respond to a set of questions before and after their participation. Alternatively, it is possible to simply ask questions after the intervention, focusing, on aspects of change and experience of the intervention.
- Process evaluation seeks to understand what happened during the project implementation, including how various groups experienced it. Evaluating processes helps to identify which impacts were or were not achieved and to pinpoint where things went off track. In turn, this highlights where and how processes might be improved.
- Formative evaluation is carried out during the project, seeking to collect helpful data and feedback for the project delivery and/or management teams. This can help identify if things are not going as planned and signal the need for changes in direction or practices.
- Summative evaluation is carried out both during and after a project. It aims to provide information, data and learning for reporting and future project development.
- 360° evaluation prompts consideration of and engagement with all of the sources from which it might be helpful to gather information and data. In the context of an energy solidarity project, a 360° evaluation might include: participating households; the project delivery team (e.g. energy advisors, fund-raisers, volunteers); the project management team; project documentation; and external partners (e.g. energy agencies, local governments, health and social care services, civil society organisations).
Click through to the next blog on ‘Evaluation’: