Any project that takes place over an extended time period will experience change. A retrospective evaluation will be important in both evaluating the impact of unplanned change and in determining the need for planned change. The evaluation may include inputs from the quality and project management processes as well as metadata accumulated through the project. Outside evaluators may be useful or even required.
A retrospective evaluation may assess the degree to which goals were met. It may also consider changes in the project environment - administrative, physical, and intellectual (e.g., has someone invented better measurement methods?). Choosing to make mid-course improvements will always have to be balanced against the possibility of confounding study results. For a more detailed discussion see Greenfield et al.
9.1 ESTABLISH EVALUATION CRITERIA
Criteria will need to be established for determining which planned or unplanned changes are significant, whether external events need to be documented, and whether established goals are being met. All of the lifecycle elements are candidates for scrutiny. Design, sampling, data collection, data processing, analysis, and the retrospective evaluation process itself all should be evaluated. Comparison of expenditures against the budget and the rate of progress against the planned timeline may also be important.
9.2 GATHER EVALUATION INPUTS
The criteria for evaluation will guide the selection of inputs for the evaluation process. Some inputs may be collected as the earlier phases of the study progresses.
9.3 CONDUCT EVALUATION
Some components of the evaluation may occur as the study progresses. Some information about the data collection process will be available as a round of collection finishes. More may be revealed as data processing and analysis proceed. Analysis of patterns of missing data, for example, may uncover flaws in the collection process.
9.4 DETERMINE FUTURE ACTIONS
Evaluation may suggest changes to the project. Once again, the decision as to whether to implement those changes will entail a careful consideration of the impact on the ability to achieve projects goals. Some changes may have to wait for a future study. Choosing to make some changes may initiate a revisiting of some of the earlier steps in the process model. A change to a variable may necessitate redesign of an instrument and so on.