8. Reviewing the Results of a Validated 10-10 Project
The project report will become available after a project is validated by CII and included in the dataset used to generate quartiles. The system will send an automatic notification to the project coordinator once the report becomes available. Validated projects are represented in green as shown in the figure below.
The Project Coordinator can then use the chart icon in the Manage Surveys box to access the final assessment report (Figure 8.1). The report will be opened in a new window or tab. A description of the final report is given below. Note that, if you try to access the project report before validation, all input and output measures will be unavailable (the character ‘C’ will be shown in place of each chart) and only the responses to the Input Section questions will be available.
Figure 8.1 - Validated Survey
The 'Project General Information' section (Figure 8.2) includes the project original and normalized cost, project type, capacity, and duration (actual and forecasted), which are relevant for the output measure presented in the report. All projects are normalized by Chicago in 2013 Dollars (as of Jun 2015) unless the project 'cost location city' or the 'phase midpoint' is not available. The 'cost location city' is assigned by CII Account Managers during project validation based on the reported project location.
Figure 8.2 - The Project Description Section
The Input Measures Section
For each of the ten boxplots included in 'Input Measures' section, the best performing projects are in the fourth quartile at the top to the chart and are represented in green (Figure 8.3).
Figure 8.3 - Input measure boxplot color legend
Each boxplot provides the project score (black dot within the boxplot), which can be compared to the four quartiles in order to indicate the project position with respect to N similar projects. For instance, the Supply Chain boxplot in Figure 8.4 indicates that the project is in the first quartile of the group of 46 similar projects (N=46). This means that the project is amongst the 25 percent worst performing projects in terms of Supply Chain related questions. Below the chart, numeric values of the quartiles are provided, along with the project score and the number of projects included in the quartile.
Figure 8.4 - 10 Input Measures
After observing a measure score and quartiles, the user might seek to identify which questions contributed to a given score. The question-measure map allows users to identify the questions related to any given measure (there is one map per sector-phase). The map allows the identification of the questions that negatively influenced a measure score and might help to identify problems that require corrections.
For instance, a shaded cell in the Supply Chain column indicates that a question influences the Supply Chain Score (questions 15 and 17 in Figure 8.5) while a white cell indicate that a given question does not influence the score of that measure.
Figure 8.5 - Supply Chain question to input measure map.
The ‘Input Measure Comparison Criteria’ section presents the criteria for selecting projects to be included in each input measure chart. All charts will contain projects within the same industry group and phase. Other criteria might apply depending on the availability of projects since only charts with 8 or more projects submitted by at least 3 companies are included. For instance, as shown in Figure 8.6, the Planning measure quartile includes projects with similar Respondent (contractors), which means that that are more than 8 projects submitted by 3 contractors in that quartile. In the absence of 8 or more projects submitted by at least 3 companies, a ‘C’ character is displayed indicating that the chart is not available due to CII confidentiality policy.
Figure 8.6 - Input Measure Comparison Criteria
The Output Measures Section
For output measures, the color scheme adopted for the boxplot chart indicates that the best performing projects are in the first quartile at the bottom of the chart, which is represented in green (Figure 8.7). Note that this is the inverse of the scheme adopted in the Input Measures section.
Figure 8.7- Output measure boxplot color legend
The number of output measures charts varies from phase to phase. As in the 'Input Measures' section, the project scores are displayed and, below the chart, the number N of projects and the specific values of quartiles are provided. For all the capacity-based measures, which are the ones with ‘efficiency’ in their names, the number of projects is considerably reduced because only projects with similar project type and capacity unit are included in the quartile. For common units such as ‘Barrels per Day’ and ‘MW’, more projects will be retrieved compared to less common units. The symbol ‘C’ indicates that a chart cannot be displayed because either the total number of project for the quartile is lower than eight or because there are less than three companies included in the quartile (as per CII’s confidentiality policy).
Figure 8.8 - Output Measure Charts
The ‘Output Measure Comparison Criteria’ section presents the criteria for selecting projects to be included in each output measure chart. All charts will contain projects with the same industry group and phase. All capacity-based charts will use project type and capacity as criteria. The definition of ‘N/A’ here is "Not Applicable"for the associate metrics. As shown in Figure 8.9, for instance, for the Engineering Project Cost Efficiency measure, the following criteria are used: Chemical Manufacturing project type; short tons per day as capacity unit, and both contractor and owner projects are included. Note that the Complexity criterion is not used for the Engineering Project Cost Efficiency.
Figure 8.9 - Output Measures Comparison Criteria
The Individual Questions Section
The last part of the report, the 'Individual Questions' section shows the aggregated responses from several team members to the Input Section. Questions are sorted in descending order according to the average score of responses (Figure 8.10). These scores are used to calculate the input measure scores.
Figure 8.10 - The question-level section of the 10-10 report