Archive for March, 2012


Should Formal Appraisals be Considered Harmful?

March 13, 2012

1 Context

A while back I made a proposal to a Chinese consultancy customer, a long-standing customer, through which we could combine training in CMMI and quality / process improvement, team training, readiness review and on-site appraisals through a programme that would minimize travel costs. The answer was something I knew, which he repeated again: the Chinese market is not really interested in quality or improvement; they are interested in having maturity levels cheaply.

This was, more or less, the last straw in leading me to believe that maybe we needed to talk about the value of formal appraisals.

A debate was organized on the motion “This house believes that formal appraisals are a waste of resources and harmful to the organization”.

I presented why this was the case, Kieran Doyle argued why he disagreed. We had a vote before and after the debate. Before the debate, the votes were shared evenly between those that disagreed with the motion (i.e. who believed appraisals to be useful) and those who did not know or care (and who were obviously coming to be informed).

The debate was useful; we covered a lot of topics. At the end, we had half the participants agreeing with the motion (appraisals are harmful); the other half was shared evenly between those who disagreed or did not know either way. Even though, I was defending the motion and therefore can be considered as having won the debate, I admit that this was not the result which I wanted.

2 Arguments put forth

2.1 Accepting the minimum

The focus of a majority of appraisals is “certification”, i.e. getting an approval, a rubber stamp, a maturity level that will be used for marketing purposes – this is demonstrated through the fact that a vast majority of reported appraisals are using the “Staged” representation (focused on maturity level) rather than the “Continuous” representation (focused on continuous improvement).

The training for team members has for a long time included the statement that you need to consider what is “reasonable” without considering the “goodness” of the artefacts identified. While this statement has now been removed from the main body of the slide, the recommendation remains in the notes. This means that there is a difference between the fact that participants have received training (reasonable) and have actually learnt what they should be doing, why and how (good).

Another example of this attitude is the large number of organizations which present a “MS Project” file (Gantt chart) to demonstrate that they have created a Work Breakdown Structure. They do not understand the purpose of the WBS, they have not taken the time to create a WBS that will allow them to enter valid data into the MS Project file, however, they have broken the project down into some form of hierarchical breakdown of activities which facilitates estimating, etc.

In conclusion, many organizations focus on the short-term satisfaction of the appraisal rather than the long-term benefit to the customers.

2.2 Coverage and Scope

The appraisal methodology has been trying to improve the manner in which coverage can become more representative of the work being performed. However, it is not possible today to deal with the management of a company that hides teams or projects in order to avoid not getting the result they want.

Frequently, achieving a maturity level or certification is critical for marketing reasons and therefore the people in charge are willing to do whatever can be done in order to ensure success. However, they are usually not willing to make the investment or take the time required to actually improve the work practices and focus on delivering higher quality or better efficiency (lower costs, reduced time to market): it is so much easier and faster to tamper with the evidence. It is very hard for an external appraiser to understand the difference which may exist between the department they are being shown and the full extent of the organization which will be advertising their “success”.

2.3 Team members

The team members should be trained to understand what is reasonable and that the letter of the law is not necessarily sufficient to achieve the results or the spirit of the model. The team includes a mix of internal and external members who are seeking to identify what is in place and where the strengths and weaknesses are.

Through discussions and seeking consensus, the team works on identifying what is best and what could or should be done for the long-term improvement. However a significant amount of trust must be placed on the internal team members who will assist in identifying the artefacts and ensuring the right people are interviewed.

Some managers understand this and offer the team members a bonus based on the “success” of the appraisal, ensuring that there is a financial reason for them to push the discussions, interviews and evidence in the right direction. As they are given time before the on-site appraisal to identify all the documentation necessary, they are also given the time to create the missing artefacts and post-date them wherever necessary. They can also take time to teach the interviewees what the correct answers should be.

When we get to 2.00 am, how long will the lead appraiser be happy to continue arguing that something is not sufficient when the whole team appears convinced that it is?

2.4 Formality

The formality of an appraisal ensures that the results are (apparently) of a similar level, whoever performs the appraisal. By having a clear communication of the purpose of the business, the teams and lead appraisers should be able to clearly identify what should or could be done in order to guarantee reasonable results.

However, once again, those results are valid only in the case when the lead appraiser and the organization being appraised are honest in their objectives and not trying to falsify the results for marketing purposes. The formality itself presents a down-side in the fact that the people being appraised can have a very good and precise understanding of what is required and can therefore bend the rules more easily.

2.5 Cost

Finally, the cost of the appraisal is such that it is in itself encouraging people to game the results in order to avoid having the additional cost of a second appraisal. In addition to the cost of a certified lead appraiser, training, travel and lodging, organizations need to factor in the cost of the disruption to their staff, the time for project managers and key staff to participate in interviews and various presentations. Finally, the appraisal team needs to ensure that the necessary artefacts are found and classified. This large expense is frequently forgotten in the bids made by consultants.

There are some 359 practices to be considered for a maturity level 3 appraisal. If you consider that you will need to identify, locate, review, verify and validate for three projects (base units, teams) and assume that this takes an average of one hour per practice, you are talking about 269 eight-hour work days – 13.5 man months just for the documentation! Of course you want to game the results as much as possible so as not to have to do that again!

2.6 Consultants

One approach to reducing the effort and increase the likelihood of success is to hire a firm of consultants to come and “do CMMI to you”.

Some consultants work on ensuring the long-term benefit to their customer’s business. They help identify the weaknesses, the risks to the organization, the causes and consequences of the practices in place and train them to understand what they are supposed to be doing and why so as to provide long-term improvement. They may or not participate in the appraisal team and are usually fair in helping identify both the strengths and the risks within the organization.

However, there are many consultancy groups out there who are willing to create all your processes for their customers, delivering a ready-made set of practices, including coaching of the members of staff in how to answer the questions. I remember one appraisal where the local members of staff did not know where any of the documents or standards were to be found or how they were used. However we had on the team the consultant who had been assisting them and he knew perfectly well where to find things and how to answer all the questions. It very rapidly became apparent that the local consultant had put in place a fictional set of processes and templates, then called in a lead appraiser, certain that I would not realise that the locals had no understanding.

Finally, there are a significant number of consultancy organizations who will assist in the definition and implementation of standard processes and templates, and will then use their own lead appraiser to review the result. In this case, they do not even need to train the staff, because everything is an internal job (even if the appraiser officially works for a different company).

Of course, all appraisers have signed a code of conduct which prohibits this kind of behavioud, but currently there is no body with the authority and the resources to audit everything.

3 Conclusion

The extensive cost of a formal appraisal could be invested in real improvements, producing results that are focused on satisfying customers and staff directly. While the formal appraisal has many rules built into it aiming at reducing cheating, there is no way that this is (or can be) effectively policed.

%d bloggers like this: