As someone who is actively talking about, teaching and writing about value creation, and most importantly, someone who has published a book on how to build value-focused marketing strategy, the need for me to understand exactly how sustainability frameworks measure, account for, and report on value creation (or destruction) has been vital for my work.
Luckily in late 2020, I received a research grant from Doshisha University so that I could hire a small team of MBA student research assistants to help me figure all of this out. Because of this, my research team and I have spent more than 600 hours poring through 15 of the world's leading ESG and Sustainability reporting frameworks and collected more than 350 individual impact measurements. Our goal? Trying to organize these into a value measurement framework that anyone, in any company of any size, industry or location can begin implementing immediately in order to contribute to a more sustainable future.
The exciting news is that we were able to synthesize these 350+ ways to measure value into 80 goals across seven stakeholder categories. While we have published a streamlined version of our white paper here, I've written this blog post to give a basic overview of the process we followed.
The Frameworks we used
While there are an endless number of sustainability reporting frameworks in existence that help define how value can be measured across various stakeholder groups, we focused on these, including (1) The Sustainable Development Goals (SDGs), (2) Global Reporting Initiative (GRI), (3) SASB, (4) B-Lab's Business Impact Assessment (BIA), (5) Global Impact Investing Network (GIIN), (6) Impact Reporting and Investment Standards (IRIS+), (7) Natural Capital Protocol, (8) Social & Human Capital Protocol*, (9) McKinsey's Psychological Safety model, (10), International Future Living Institute's JUST 2.0, (11) The B Team, (12) Cradle to Cradle certification, (13) RBL's Organizational Guidance System, (14) The National TOMs Framework, and (15) The Common Approach to impact measurement.
What we did
Step 1: Collect all impact measurements
First, our research team collected all of these impact measurements and organized them into one spreadsheet in Excel. In total we collected 357 of these indicators which we came to call "micro-indicators" and kept the link to the source data to make sure that we understood exactly where it came from and the language that was being used to describe that specific indicator. Here is an example from the beginning of our spreadsheet that shows a number of impact measurements from the BIA framework.
Figure 1: Collecting Micro-indicators
Click Here for a full list of the impact measurements we collected
Step 2: Categorize into Stakeholder Groups
The research team then read through every micro-indicator and assigned it to at least one specific stakeholder category that the Business Roundtable and World Economic Forum's Davos Manifesto had outlined in 2019, including (1) employees, (2) environment, (3) society, (4) the firm itself, (5) customers, (6) partners, and of course (7) shareholders.
One early surprise: In these efforts, our research showed that while there were multiple ways to measure value for the first six stakeholders listed in the paragraph right above this one, shockingly (for me at least), none of these frameworks emphasized shareholder value. After thinking this over for a little while with the entire team though, this turned out to not be such a big surprise at all. The reality is that the accounting and finance worlds have already developed a clear set of indicators for measuring shareholder value, and all of these ESG or sustainability metrics were meant to help companies report or disclose information beyond what they were already doing related to shareholders.
Step 3: Categorize into Macro-Indicator Groups
Now that we had all of the micro-indicators organized into stakeholder groups, we then began the process of grouping these within larger themes. This is exactly how researchers who work with qualitative data (like rich text data) begin to add structure to their work, starting with raw interview transcripts, then coding these, and then grouping codes into code families. We followed this same approach and ultimately arrived at 27 macro-indicator categories as outlined below:
Figure 2: Creating Macro-indicator Groups
Step 4: Score micro indicators within each macro-indicator category
Within each of these 27 macro-indicator groups, we then reviewed every micro-indicator that was listed and gave it a score of between 0-5 points based on its ability to satisfy the following criteria. Specifically, we asked if each micro-indicator that we had collected and categorized was:
Goal-based: We tried to understand if there was a clearly stated goal or not. For those that had clearly stated goals, for example this one from the JUST 2.0 framework "Organization must have a gender equity pay-scale with a maximum variance in pay of 5 percent between genders within each of the organization’s pay scale classes," we allocated 1 point. For those that didn't, such as this one in the same Diversity & Equity macro-indicator category from The B Team, Businesses uphold gender balance, diversity and inclusion not only as the right thing to do, but as a driver of shifting norms and delivering better business performance as well as economic growth", we allocated 0 points. We are not judging the intentions or thinking behind these indicators, but instead simply assessing whether or not there was a clearly stated goal included.
Objectively Measured: We next tried to understand if each micro-indicator could be objectively measured, allocating 1 point if it could, and zero points if it could not. For example, GRI disclosure 305-5 requires companies to report on their greenhouse gas emissions as follows: (a) GHG emissions reduced as a direct result of reduction initiatives, in metric ton, (b) Gases included in the calculation (CO2, CH4, N2O, HFCs, PFCs, SF6, NF3, or all), (c) Base year/Baseline (the rationale for choosing it), (d) Scope in which reduction occurred (Scope 1,2,3), (e) Standards, methodologies, and assumptions used. Because these emissions can be measured from the same location by two different sensors and receive the same results, these types of indicators are considered objective. Others that allow for flexibility in measurement based on the individual evaluator received zero points. For example, the Organizational Guidance System's "Rate on a scale from 1-5 the current state of your business about positive relation between employees and customers/community," received zero points in this assessment category.
Independently Checkable (Transparency): In addition to whether or not an indicator was objective, the question remained as to whether or not an independent outside 3rd party could (easily) check and confirm that what was reported by a company was matched by actual data. Our definition of this rating was that the information could be “checked by using transparent data that an outsider could access or obtain.” We allocated zero points to those micro-indicators that were not independently checkable, 1 point for those that could be independently checkable, but that we did not have proof that this checking was actually being done, and 2 points for those indicators that were in fact independently checkable and we had evidence that this feedback loop was healthily in place.
Varies: One of the surprises of our initial efforts in Step #1 was the number of micro-indicators that simply required companies to file a report, with points given for simply filling in details rather than its contents. For example, within some of the frameworks we’ve studied it is possible to receive a positive score for reporting on the percentage of women on a company’s board of directors, even if that number is zero. The fact that this is reported satisfies the disclosure requirement with no qualification of the actual behaviors undertaken by that company. This was true across a number of different themes and topics, and we therefore added this final rating of an indicator based on its ability to highlight finer details beyond black and white, yes-no answers. We specifically defined this rating as receiving zero points if “the scale that is used for this measurement is a nominal, binary, or a yes/no question,” and eligible to receive 1 point if the micro-indicator was measured using an ordinal, interval or ratio scale.
The results of our assessment of all of these 357 micro-indicators returned the following results:
Figure 3: Scoring all micro-indicators
Based on this analysis of the indicators used across all the assessment frameworks we studied, as shown in Figure 3 we found that 0% (n=0) indicators achieved a full five-point score as we have explained above, and that only 5.3% achieved a 4-point score. This left 94.7% of the indicators that we rated (n=341) with scores of 3 or less points, with 30.8% (n=110) scoring 3 points, 16.8% (n=60) scoring 2 points, 23.5% (n=84) scoring 1 point, and 23.5% (n=84) scoring zero points. It is an understatement to say that we were surprised by these results, especially that nearly a quarter of all value measurements achieved zero points, meaning that they were yes/no variables without goals that could be objectively and transparently measured, and that nearly another quarter of all value measurements received credit for one of these factors.
One broad conclusion from our analysis is that a significant amount of work needs to be done by the global ESG and sustainability communities to tighten their impact measures to have clearly defined goals, establish methods for the objective and transparent reporting on these, and to ensure that these measures go beyond simple yes/no, present/not-present binary answers. Without such rigor put in place, too many loopholes exist such that any savvy marketing, advertising or PR expert could “re-brand” the company’s clearly negative actions into positive ones.
In this step, we also circled back to the original sources of our impact measurement data to confirm that these macro-indicators were for the most part supported by more than one organization. As shown in Figure 4 below, except for the two macro-indicator categories that we have created for Shareholders and Firm Capability (that we will discuss in detail in a separate blog post), all macro-indicators were derived from at least one of the frameworks we had studied and in most instances there were a number of frameworks focused on exactly the same issue.
Figure 4: Coverage of Macro-indicators across assessment frameworks
Step 5: Set Objective goals based on micro-indicators
With each micro-indicator allocated within one macro-indicator category, we then continued to organize micro-indicators within each of these categories so that we could clearly define exactly what each macro-indicator was there to achieve, and to operationalize these definitions into specific, measurable goals. As outlined above, while most individual indicators were poor at achieving a full five-point scale based on our assessment criteria, when joining these together into larger goals, it became possible to develop clear goals through aggregation around these specific themes.
In total 80 goals emerged from this process of which two were added by the research team as measures for shareholder value and management capability. I'll be writing more blog posts shortly to help explain each of these 80 goals as well as the 27 macro-indicator categories they are housed within.
If you'd like to go into deeper details of our work, please feel free to either download the Executive Summary of our white paper here, in English and 日本語：
You can download the full white paper here in either English or Japanese (but you'll need to input your name and email address to download them).
Or, if you'd like to read the entire 150+ page discussion paper, that includes all of our data inputs and a more detailed background on our research approach, please email me (psugai[at]mail.doshisha.ac.jp] and I'll send you the full version.
Comentários