This article is still relevant for any startup or company involved in software development. It contains useful informations - just like “Clean Code” by Robert C Martin - details within this article are timeless.

I first came across this article around 2016, when I noticed a large disparity between the usual junior-mid-senior scaling of developers. This scaling system, unlike others, is universal, easy to understand, and the bear hunting analogy used in the original article by Meilir Page-Jones makes it a humorous and enjoyable read.

Since then, the original page has been taken down (an archive can be found here). With the permission of the author, I am allowed to distribute the original article.

I made some slight changes by adding additional titles and styling. Since there is more behind the seven stages, it is likely to be referenced frequently.

We hope you enjoy reading this article and that it helps your next projects be successful.

The Seven Stages of Expertise in SWE

Meilir Page-Jones © Copyright 1998, Wayland Systems Inc. All rights reserved.

(In this article, an indefinite “he” means “he or she”.)

Intro

For almost two decades we at Wayland Systems Inc. have toiled to transfer the latest developments in software engineering to practitioners at large and small companies around the world. However, until recently we didn’t do a very good job of quantifying the long-term field results of our efforts. So we decided to ask a question reminiscent of the rhetoric of Ed Koch (the former mayor of New York City): “How’re we doin’?”

We surveyed some of our clients to whom we’d provided education and consulting over the years and discovered an interesting but very disturbing fact. Over 50% of the shops into which we’d introduced Software Engineering techniques had either abandoned the use of the techniques or had let their band of active practitioners shrink to a dwindling core of diehards. What was going wrong here?

We turned to self-examination and reviewed our courses and consulting practices. But, whatever the problem was, it didn’t seem limited to Wayland Systems: Informal discussions with our competitors revealed that they too were having similar problems. The attrition rate across our whole industry was painfully high.

Factors Affecting Productivity

External examination revealed several factors at work, many of which were to do with corporate management habits. Examples of these were:

• unrealistic management expectations
• a lack of commitment to quality
• an obsession with coding (which we termed WIMP — “Why isn’t Marvin Programming?” — management)
• pernicious deadlines
• and a misallocation of skills to tasks

Other factors included: engineers’ resistance to change; engineers’ reluctance to participate in team approaches; and the then-unavailability of useful CASE tools.

The above factors were significant and worthy of articles in their own right. However, there was another factor, which is more important and fundamental than the others. This factor is the way that people are able to absorb sophisticated new techniques and then to apply them to their work — in other words the stages of expertise through which we all pass as we learn and use new and powerful disciplines.

At first we were aware only that there were two types of person: novice and expert. That was hardly earth-shattering news. However, further research revealed that there are actually seven stages of expertise through which a person may pass on the journey from total ignorance to world-class knowledge. With devastating acumen we numbered these seven stages Stages 1 through 7. (We also gave them names: Innocent, Exposed, Apprentice, Practitioner, Journeyman, Master and Expert.) As I’ll discuss below, these seven stages have a deep impact on the successful — or otherwise — introduction of software engineering into an organization.

A participant at a conference asked me how universal these seven stages are.

• “Very universal,” I answered.
• “You mean that I could even apply them to the skill of bear-hunting?” he retorted unexpectedly.
• “Yes,” I said.

So, if you’ll bear with me, I should like to illustrate the Seven Stages of Expertise with reference to hunting (although I’m most definitely not a fan of such an activity). Following that illustration I’ll outline the application of the Seven Stages to our own business of Software Engineering.

The 7 Stages in Bear-Hunting

Stage 1: Innocent — A Stage-1 person has never seen or heard of bears. It would not occur to a Stage-1, if he encountered a bear, that the bear could be hunted. Neither would he realize that a bear is a potential source of danger.

Stage 2: Exposed — A Stage-2 person has seen an occasional bear and has read articles in airline magazines suggesting that bears may be hunted. Moreover, a Stage-2 probably has friends that have hunted bears and has learned some desultory but intriguing facts about bears and their habits. He is motivated to learn more.

Stage 3: Apprentice — A Stage-3 has attended a 5-day seminar on bear hunting. During this seminar, the participants form into teams of three or four and practice hunting very small bears under the ever-watchful eye of the instructor. After a few interim setbacks, by Friday afternoon all the teams have successfully hunted their bears. They fill out evaluation forms attesting that “bear hunting is very useful and relevant to my job.” However, they are barely prepared for the world of real bears.

Stage 4: Practitioner — The Stage-4, having completed formal bear-hunting education, is full of confidence. He’s ready to transcend the minuscule bears of the 5-day workshop and go out for real bears, larger bears, fierce bears. He’s ready for Ursa Major. His manager is also keen to send him out with the latest bear-hunting techniques because the users want fur and they want it yesterday. Unfortunately, in the resulting scramble the budding bear hunter may be sent out without a map and with the wrong calibre arrow in his longbow. In the heat of ursine confrontation, the Stage-4 may also forget or misinterpret his classroom instruction and precipitate disaster. It’s typical that some Stage-4s get some bears; but it’s also typical that some bears get some Stage-4s.

Stage 5: Journeyman — The Stage-5 has survived the traumas of Stage-4 and has bear hunting down cold. The Stage-5 uses modern bear-hunting techniques naturally and automatically; in fact, he can’t imagine how he ever got along without them. He is accurate and productive: The Steering Committee merely points out the bear and he hunts it within both budget and deadline. The Stage-5 is the exemplary modern hunter that salespeople of bear-hunting seminars refer to in their brochures.

Stage 6: Master — Stage-6 bear hunters have internalised not only the mechanics of bear hunting but also the principles underlying the techniques. Stage-6s know more than rules: They know why the rules exist and even when it’s permissible to break them. For example, a Stage-3 or 4 may stand upwind of a bear accidentally and scare off the bear. However, a Stage-6 may know that by wearing Yogi-Spray Deodorant he can stand upwind without being detected and can thus surprise the bear from an unexpected quarter. Because of their deep knowledge, Stage-6s are very capable of training others in hunting techniques.

Stage 7: Researcher — Stage-7s are asked to write books and give talks at bear-hunting user groups. They are also engaged in extending and generalizing bear-hunting techniques to solve new problems. For example, a Stage-7 may extend bear hunting to work also on Big Foot or he may even develop the ultimate Yeti-Oriented Hunting Methodology.

The 7 Stages in SWE

Now let’s come back to the world of software engineering and see how the Seven Stages of Expertise apply to us.

Stage 1: Innocent

A Stage-1 may not have heard of Software-Engineering techniques. Or — more likely nowadays — he may be vaguely aware of their existence but may not see their possible relevance to his situation. Indeed, he may be only dimly aware that there are any software-development problems in his shop. You may find it incredible that Stage-1s could exist in the 1990s, but they do. And the reason stems from the way in which software complexity evolved.

Software became insidiously more and more complex in the 1970s and 1980s as users demanded more and more sophisticated systems be installed on the more and more powerful hardware that became available. Yet there was no sharp transition. The earth was not hit by a Complexity Asteroid in 1975 that suddenly made software three orders of magnitude more complex and cast our reptilian development techniques into extinction.

I call the way in which software complexity actually increased “Frog in the Pan.” This is because although a frog will jump out of a pan of hot water, a frog that is placed in a pan of cold water and slowly heated will fail to leap forth and will actually boil to death. The temperature gradient is so gradual that there will never be a point at which the frog declares: “Boy, it’s suddenly gotten hot in here! I think I should hop out.” (Before I get into deeper trouble from animal-rights folks, I hasten to add that this analogy is apocryphal. I’ve never tried the experiment and I don’t recommend that you do so either!)

Many Stage-1s are experiencing “Frog in the Pan” and are trying to tackle problems of the 1990s with approaches of the 1960s and 1970s, without realizing that the problems they’re facing are the very ones that modern Software-Engineering techniques were created to alleviate.

Stage 2: Exposed

Stage-2s have noticed that the water is getting decidedly warm, if not downright hot. So they are actively seeking Software-Engineering techniques that will get them out of the pan or at least reduce the heat. Stage-2s may survey magazines, confer with colleagues or attend one-day overviews of the techniques. Their interest level is high but their knowledge level is low, being limited to a few terms and definitions and not based on any practical Software-Engineering experience.

Stage 3: Apprentice

Stage-3s have attended one or two 5-day workshops on Software-Engineering techniques. In these workshops they tackled small but realistic case studies that resembled their own projects in miniature. The case studies provided valuable kinaesthetic reinforcement of the formal lecture material and were thus indispensable. However, the case studies’ apparent realism conveyed to the Stage-3 a confidence that is often unwarranted.

If a Stage-3 absorbs everything from a seminar, then he is minimally equipped to tackle a true, full-sized project in the corporate jungle. Usually, however, a Stage-3 does not grasp everything or has difficulty scaling the techniques up from a case study to a real project. You could say that most Stage-3s know just enough to be dangerous!

Stage 4: Practitioner

The rite of passage to Stage-4 is the use of Software-Engineering techniques on at least one significant project. Achieving “Stage-4-hood” is for many people the most difficult transition of the six transitions between stages. The fledgling Stage-4 is asked to take untried (by him) techniques and apply them to a corporate project with the usual demonic cocktail of politics, deadlines and changing requirements. At the same time, he is attempting to recall what he learned in class and scale up the examples 10- or 100-fold. He continually needs consulting advice, without which he will encounter a series of minor setbacks or major failures. Since many people throw up their hands at this point and revert to their old mediocre but familiar ways, a large proportion of Stage-3s never make it to Stage-4. If an entire project is peopled with Stage-3s, then it’s highly likely that the project itself will fail and the Software-Engineering techniques will be publicly pilloried and then abandoned.

Stage 5: Journeyman

Stage-5s have made it. Their experience of Software Engineering is “latched” in place and there is little risk of their reverting to the past. In the Stage-5 the Software Engineering techniques yield for the first time the productivity the marketing folks promised; and on each successive project a Stage-5 further hones his skill and enhances his productivity. A Stage-5 is self-sufficient - more often the source of Software-Engineering advice than its recipient.

Stage 6: Master

The Stage-6 not only is an adept technician, but also possesses a profound methodological foundation. Beyond the “whats” and “hows”, the Stage-6 knows the “whys” of Software Engineering. This depth allows him sometimes to break a surface rule, while adhering to a more fundamental methodological principle. The Stage-6 is a good instructor because his theoretical and practical knowledge give him the wherewithal to tackle difficult student questions.

Stage 7: Researcher

The Stage-7 is concerned with delivering the latest developments in Software Engineering to a wider audience, via books, articles and conference appearances. The Stage-7 looks out for flaws in contemporary Software-Engineering techniques and for consequent ways to improve the techniques. He also scans the horizon for new problems towards whose solution Software Engineering can be extended and generalized. The Stage-7 is at a methodological pinnacle.

Behind the Seven Stages (Productivity, Starting Projects, …)

These Seven Stages of Expertise are valuable in their own right. You might think about a Software-Engineering technique that you know and consider which of the stages you are in with respect to that technique. You may also decide what (if anything) you should do to progress to the next stage.

However, there are organizational implications behind the Seven Stages. Below I discuss four of these:

• The Productivity Curve
• Pilot Projects
• The Critical Consulting Core and Ephemeral Technology.

A. The Productivity Curve

Vendors who sell Software-Engineering techniques do so by convincing management of their boon to productivity. Although vendors rarely say so explicitly — we at Wayland Systems never used to — they compare the productivity of Stage-5s to that of Stage-2s. Unfortunately, this presentation (see Fig. 1) is not a complete picture. Between Stage-3 and Stage-4 there’s a hair-raising “J-curve” of productivity (see Fig. 2), which represents the difficult rite of passage of the incipient Stage-4.

This J-curve has led more than one manager to abandon all hope in Software-Engineering techniques. Such a manager is not merely impatient; no one warned him of the J-curve and so when it manifests itself he can only assume that the much-touted Software-Engineering techniques were mere hokum.

A couple of our clients measured the length of time from Stages 3 to 5. Depending on the individual person, Stage-3 to Stage-4 took 6 to 18 months (about the length of a project or phase) and Stage-4 to Stage-5 took 18 to 36 months. (These times are very approximate because of the subjective nature of the stages.) Except for a few outstanding people, the shortest time from Software-Engineering seminar to exquisite software productivity is 24 months. When managers tell me: “We just can’t wait that long,” I retort: “The 2 years will pass anyway, whether you do anything or not. Act now!”

(By the way, notice that Stage-7s have lower project productivity than Stage-6s. This is because a portion of Stage-7s’ energy is directed away from the specific project and towards general methodological research.)

B. Pilot Projects

But the J-curve has a facet nastier than simple delay in productivity. This is the “Who will bell the cat?” problem. In the fable of that title, a group of mice that’d been persecuted by a cat decided to hang a bell around the cat’s neck that would warn them of its approach. All agreed that this was a wonderful idea … but there was one minor difficulty: None of the mice wanted to be the one to risk walking up to the sleeping cat to tie a bell around its neck. This impasse is known formally as the Technology Trap.

Most project managers would agree that Software-Engineering techniques are a Good Thing. But which of them with bone-crushing deadlines and axe-wielding users wants to be the first to experience the depths of the J-curve. (Me neither!) Thus many shopfuls of intelligent people appear to be dinosaurs because the Technology Trap holds them in the 1970s.

If “who will bell the cat?” is a problem, then I recommend that you try Software-Engineering techniques on a pilot project first. The pilot will uncover all manner of pitfalls that occur during the Stage-3/4 transition as well as problems in the techniques themselves. The pilot should be short (6 months) and should yield a non-critical system. You should learn as much as you can about applying the new technology from the pilot project; this is one project where the learning experience is more important than the end product.

A pilot project will not eliminate J-curves on subsequent projects. People will still have to go from Stage-3 to Stage-4. But the pilot will make future J-curves shallower, as some causes of the curve’s nadir are removed. Indeed, although the delivered system is the most important product of most projects, you should never forget the learning by-product of every project. Review what you discovered about doing projects when you complete each project and share that information with others. In that sense, treat every project as a pilot.

C. Core of Internal / External Consultants

One of the surest recipes for disaster is to staff a project with only Stage-3s. Yet many managers do exactly that, pushing a group of analysts through a seminar one week and letting them loose on a crucial project the next week. (Jerry Weinberg refers to this as the “Sheep Dip” approach.)

Every project should have access to a Stage-5, or - better still — have a Stage-5 on the project team. Cultivate your own Stage-5s and Stage-6s in-house as part of your long-term staff-development policy and make them available to project teams. If you don’t yet have such people on board, then call in external consultants from Wayland Systems or one of our competitors. However, don’t become chronically reliant on external consultants, but instead continue to work on building up a pyramid of internal expertise with Stage-6s at the top that will ensure a lasting mastery of Software Engineering.

D. Learning Latency vs. Lifetime of Technique

One manager of a major organization recently complained to me that by the time a critical mass of his people had learned a Software-Engineering technique that technique had been left behind in the mud of obsolescence by the relentless march of progress. Although there is, of course, some truth in this complaint, it is also true that “plus ça change, plus c’est la même chose.” If you buy a power drill to replace a manual drill, the deep principles of drilling will be largely unchanged. Similarly, most new Software-Engineering methods are basically founded on familiar underlying methodological principles (such as the principle of “Safe Deferral” or of “Partitioning to Minimize Interfaces”).

Develop a large “capacitance” of Stage-6s in your organization in order to smooth the apparent high-frequency ripples in technology. These people will be able quickly to absorb a new technique and to assess its applicability to the shop’s needs. They will also be able to compare and contrast the new technique to current techniques and to steady the transition from old to new, because many of the new ways of addressing software are simply refinements of ways already known to Stages 3, 4 and 5.

Some Recommendations

• Be aware of the Seven Stages of Expertise and their effects on productivity under a specific Software-Engineering technique. Reconcile your expectations for Software Engineering with the realities of the Seven Stages.
• Take stock of where you are. Have each person assess his stage with respect to each Software-Engineering technique that your shop uses.
• Have each person state their medium- and long-term goals for expertise in each relevant technique and suggest a plan for achieving those goals.
• Establish similar medium- and long-term goals for the shop as whole. Determine the most appropriate blend of stages for the work that you do and set up a plan to achieve that blend.
• Never attempt a crucial project solely with Stage-3s and below. Seed each project with Stage-4s and allow access to Stage-5s and 6s if possible. If you don’t have a Stage-5, beg, borrow, steal, buy, kidnap… one. External consultants are valuable as a short-term measure.
• Stabilize the effects of technological change by nurturing an in-house expertise (Stage-6) in fundamental methodological principles. Try to formulate shop strategies (as much as you can) in non-technological terms. Superimpose technology on to the strategy in order to yield tactics.
• Have a mission that will never go away. For example, the mission of Wayland Systems is “Mastery in Software Engineering.” In other words, we seek to assist our clients in becoming self-sustaining at at least the Stage-6 level of Software Engineering. This mission will not evaporate for many decades yet, for software is becoming ever more complex and ever more critical. There will be much to learn — even for Stage-6s and Stage-7s — as we pass into the new century.