Sampling: Does It Work?

April 22, 2011

When using the new version of SCAMPI appraisals (v.1.3), there is a new sampling methodology that aims to ensure the selected projects being analysed are actually representative of the organization. Previously, the appraiser needed to select some three (or four) projects that were considered as representative of what the organization did. By demonstrating that these projects did a good job, as they were representative of the whole, they naturally showed that the whole organization always did a good job. Of course, companies were eager to present only what they did best, there are many techniques that can be used to hide the projects you don’t want to see.
With the new system, one needs to check all the factors that may influence the way work-practices (processes) may be implemented in different manners, consider the various possible combinations, and select a proportional sample of projects from each possible combination. The number of projects to select is influenced by the total number of projects in the organization vs the number of projects included in the subset.
Let’s consider a commercial software development organization, not a big multinational, but a running concern, which develops and sells products to a variety of customers. They have a number of different projects running, they are all for different customers, related to different types of products. Some of them are implemented according to Agile principles, others are more “waterfall” and a few are a home-made combination of the two. Some projects have heavy involvement by the customer, who can control the requirements, do the project management or even run the testing; others are completely hands-off, depending on the customer. Some of the developments are quite large, others are quite small. This is just a software development company, they work to customer requirements.
The MDD requires that the following factors be considered: location, customer, size, organizational structure and type of work, as well as any other factors that may influence the manner in which the processes are being implemented. In my (theoretical) example above, we have a different customer for each project, the organizational structure depends on the level of interference/control allowed to the customer, the sizes of the projects vary widely, they use different life-cycles and a variety of languages. Very rapidly, when doing the calculations for the sampling, we can discover that every single project is potentially in a subset that is peculiar to that one project, meaning that in order to get a true representative sample of the projects in the organization, there is an implicit need to review every project in detail! At this point in time, the natural tendency will be to determine that maybe this factor does not really have that much influence, perhaps the other is not really a variation, and the number of projects to consider is creatively rationalized down to something realistic, honestly and in good faith.
Of course, a series of small projects, run by the same team, using the same processes do not implement them in different manners, every time a parameter changes in the environment, so the appraiser is required to analyse which of the parameters have an impact on how the processes are implemented. How do you determine this? You can discuss it with the appraisal sponsor and participants, who then have the opportunity of making the claims they want to make, once again, focusing attention on the best projects and hiding the others. Otherwise, the appraiser will have to perform a detailed analysis of the differences in implementation, and that would be done through a mini appraisal of all the projects in order to identify what are the factors that really matter – so one would need to perform a full appraisal of all projects in order to determine how to select the factors to sample the projects to review. Not an economically viable solution.
Once the subsets of projects have been identified, there is a requirement to review one project from A to Z, and to collect “artefacts or affirmations” from another project within the same group, for “at least” one process area. So a second project can be reviewed and approved based only on affirmations in a single area – and this guarantees that everything else is done correctly.
Shame, when I first heard about the sampling factors, I thought this was a good idea; unfortunately, it seems that a choice needs to be made in implementation: go to a bureaucratic, extensive and expensive appraisal of everything, or select to play the numbers with as much ease and facility as previously.
The consequence of this: lead appraisers (and organizations) that believe in quality and want to do a good job, will continue to provide reliable results; those who want to give away (or receive) the highest maturity level possible, will continue to falsify data.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: