Members of the
World Bank Group
CAO is expanding the way in which data from cases is captured, mined, and structured, to support analysis that will enhance
transparency and improve the performance and outcomes of our work. To that end, CAO in Numbers aims to consolidate historical CAO data while considering changes in CAO’s practice. This methodological attachment provides more information on the data made available and used throughout the piece and clarifies some of the key methodological decisions made along the way.
For any further information, please contact CAO at CAO@worldbankgroup.org
As CAO has evolved, changes in its policy and practice impact its data. This means that CAO’s data is not always easily comparable over time. For example, the 2007 edition of Operational Guidelines marked a significant shift in CAO’s complaint-handling process towards a more distinct Assessment, Dispute Resolution, and Compliance case handling structure. The most recent review of CAO’s policy by the Board has led to another significant change in the movement from Operational Guidelines to the CAO Policy, applicable since 1 July 2021. The following table outlines the key policy and practice changes thought CAO’s history.
After a significant shift in CAO’s operational process in 2007 (see Table 1), CAO’s process has remained relatively consistent. This
consistency enables us to present data from 2008 onwards when showing how we handle complaints (How We Handle Complaints, Dispute Resolution, Compliance sections). We can use the full set of data from CAO’s inception in 2000 when reporting information about the complaints we received (Complaints to CAO, Complainants, Outreach, Threats and Reprisals sections
The table below highlights key methodological notes that should be considered in interpreting the data presented in this piece. These are broken down by section of the full CAO in Numbers website.
Year-on-year count (flow) of all eligible complaints received (n=203) and triggered cases (n=10) for the total number of complaints taken in by CAO each year.
To create data from complaints, CAO has coded text drawn from publicly disclosed complaints and CAO reporting. The information contained is classified using different indicators (issues) that track the frequency of the environmental & social issues reported as they align thematically with the requirements of the performance standards. When CAO receives complaints that raise issues that span multiple performance standards or are articulated in ways that do not clearly fit a singular performance standard, CAO captures these as cross-cutting issues, to preserve the fidelity of concerns raised. Timeseries—percentage of complaints received* year-on-year that raise the indicator reported (among others) CAO Sector—aggregated count of relevant complaints as they relate to each sector Assessment Outcome—percentage of relevant complaints broken down by assessment outcomes—closed, dispute resolution or compliance Note that the data in this section excludes complaints that are triggered.
Countries—aggregated count (stock at End of Year - EoY) of all eligible complaints received and cases triggered Regions—percentage of caseload (stock at EoY) of all eligible complaints received and those triggered. IFC updated the structure of its regions in July 2021. Regions represented in this publication do not reflect this shift given the historical nature of the data.
Percentage of caseload (stock at EoY) of all eligible complaints received and cases triggered
Percentage of caseload (stock at EoY) of all eligible complaints received and cases triggered. Note, when multiple E&S categories (through multiple projects) are tied to a complaint, the highest risk/impact category is captured by CAO.
Percentage of ineligible complaints received between 2014 and 2021— in 2014 CAO started capturing and analyzing data on why cases were found ineligible, hence the cutoff.
Percentage of caseload (stock at EoY) of all eligible complaints received and cases triggered
Percentage of respondents in relevant category—complainants, community (n=136). Data reports individual responses from the post-assessment stakeholder feedback survey.
Percentage of respondents in relevant category—complainants, community (n=137). Data reports individual responses from the post-assessment stakeholder feedback survey.
Percentage of respondents in relevant category—complainants, community (n=116). Data reports individual responses from the post-assessment stakeholder feedback survey.
Total number of countries reached (participants from the country attended (n=78), and total number of countries where CAO has held or participated in an outreach event (n=62).
Trend—percentage of all cases handled* each year (flow) since 2018, when CAO started capturing this information. Note that the data in this section reports on cases handled, and hence are counted multiple times if active over multiple years. Sources of Threat—percentage of threats and reprisals raised between 2018 and 2021, broken down by perceived source of threat (as related by the complainant)
Percentage of complaints that have closed in assessment between 2008 and 2021. Note: data from hereon only includes observations from 2008 onwards, to align with the implementation of the 2007 Operational Guidelines, which marks the most significant change in existing practice—see the CAO Governance section of this methodological attachment for more detail.
Percentage of complaints that have closed in assessment and proceeded straight to compliance between 2012 and 2021. This data is consistently available from 2012 onwards, hence the cutoff.
Percentage of respondents in relevant category—complainants, community (n=126); company representatives (n=43). Data reports individual responses from the post-assessment stakeholder feedback survey.
Total number of complaints handled in dispute resolution between 2008 and 2021.
Frequency count of process elements publicly reported in a total of 41 complaints concluded* in dispute resolution since 2008. The process elements are captured based on the language included in the publicly disclosed Close-out/Conclusion report, and hence may be underreported.* Closed here means the dispute resolution process has closed (settled, partly settled, or fully transferred to compliance), and not that the case has closed in dispute resolution.
Frequency count of outcomes observed in a total of 41 complaints closed* in dispute resolution since 2008. The outcomes are captured based on the language included in the publicly disclosed Close-out/Conclusion report, and hence may be underreported. * Closed here means the dispute resolution process has closed (settled, partly settled, or fully transferred to compliance) and not that the case has closed in dispute resolution.
Percentage of respondents in relevant category—post-assessment: complainants, community (n=119) and company representatives (n=30) post-dispute resolution: complainants, community (n=94) and company representatives (n=28); Data combines individual responses from both the post-assessment and post-dispute resolution stakeholder feedback survey.
Percentage of complaints that have closed in dispute resolution between 2008 and 2021. Note: data here drops cases that are undergoing dispute resolution or dispute resolution monitoring. * Closed here means the dispute resolution process has closed (settled, partly settled, or fully transferred to compliance) and not that the case has closed in dispute resolution.
Total number of complaints handled in compliance between 2008 and 2021 (n = 111). When parallel complaints are combined at appraisal, 90 cases have been handed in compliance over the same time range.
Percentage compliance appraisals that proceeded to investigation, again including two versions: the first uses complaints as the unit of measure and the second uses complaints merged into cases as the unit of measure. Note data here drops cases where the appraisal report is pending.
Percentage of compliance cases rated in the MATR, aggregated over six years, excluding observations rated Too Early to Tell. IFC/MIGA response is rated both at the project level and at the systems level and given one of the following ratings for each case: Highly Satisfactory, Satisfactory, Partly Unsatisfactory, Unsatisfactory, Too Early to Tell, or Not Applicable.
2010 Performance Standards Review—Percentage of recommendations (n=40) made by CAO leading to substantive changes, as noted in IFC’s Response. GRM Toolkit Survey—Percentage of respondents (n=29) from the voluntary survey conducted
.png)
.png)