Deming is world famous, especially in the Agile/Lean world. During his life Deming taught top management how to improve design (and thus service), product quality, testing and sales. His name is synonymous to quality improvement and his philosophies are still seen as visionary.

One of  Deming’s principles is: “Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.” Sounds good, right? In stead of testing every last inch of your product, you increase it’s intrinsic quality which is more sustainable, has a direct benefit for the customer and eventually lowers cost.

Now how does that relate to software development? In software development we do quite a bit of inspection, not only by peer reviews but mostly by testing. We test documents, units of code, modules of code, systems, user interfaces, services etc. We test functionality, speed, error handling, ease of use, fit for purpose etc. etc. My interpretation of what Deming would say to us: focus less on creating more tests, focus more on product and process quality.

Let’s start out with the obvious, tests are useful. Even when you’re 100% Deming compliant™ you’ll still have tests, especially automated tests. But I’m not arguing to stop testing, the point I’m making is that tests lure you into a false sense of security and are not a guarantee for improving quality. A symptom of that is in software development QA people (often called testers) seem mainly concerned with ‘checking’ and spend very little time on ‘mistake proofing’.

Mistake proofing

John Hunter says tests are a form of prevention. They are a form of Poke Yoke, Poke Yoke means “mistake-proofing”. His reasoning is: with a test in place, you can’t introduce a bug in the software. But this is not true, the test detects the bug, but doesn’t prevent the bug from entering the code. A better example of Poke Yoke in software development is using high level language constructs, these prevent mistakes that are easily made with low level languages. The difference is that the mistake-proofing is done at the level of human actions where the mistakes occur and prevents the mistake being made. A test doesn’t prevent a mistake being made, it just signals that the product contains (at least) one bug. So even if your test servers as a Poke Yoke, it’s only detecting, not preventing.

Prevent Construction Errors

Tests and other forms of inspection have one fundamental limitation. I’ll try to clarify this with an analogy:

Suppose you’re a gardener who has to keep a big lawn. The worst enemy of the gardener is the mole and its molehills. Now suppose you’ve discovered a molehill. You don’t want those on your lawn. So you can remove it and place a heavy object (a slab of stone) where the hole was dug. You’ve made sure the mole can’t create a molehill at that particular place, but you haven’t prevented the next molehill from appearing.

After talking to your software tester friend, he helps you to find a way to strategically put the slabs in place before the mole arrives, preventing a lot of the molehills from appearing. You’ll soon end up with a garden covered in stone slabs and it keeps your lawn fairly green. However, the obvious question is, why aren’t you trying to get rid of the mole?

In software development, I’ve seen too few team getting rid of the mole. Too few discussions on the real cause of bugs. I would expect a Deming compliant team™ asking: “What preceded writing that particular bit of code that led to the bug? What would make it impossible to create this bug again? How can we increase quality across the board to prevent these kind of bugs, not just this single one?”

Prevention vs. Inspection

Donald Reinertsen says “Inspection makes economic sense if it is more cost-effective than prevention.”  Of course, this is a truism and automated tests are a relatively cheap way to do inspection. When inspection is your major instrument for assuring and improving quality, there’s a risk of test-addiction. Suppose you have lots of tests, but in production the customers finds a bug. What do you do? You create a test for it and that might not be the best way to handle the bug. If the actual cause is low software quality due to overly complex design or sloppy coding, it’s not enough to create a test.The opposite could even be true: the number of tests per quantity of software (LoC) should decrease over time because quality increases to such a level tests are no longer needed in large numbers.

Decrease the necessity of testing

In software development we need to fight the mole more often than patching the molehill. Do a root cause analysis with the team to uncover the cause of the bug and change your process, change they way you do design, introduce patterns, use pair programming, take a training course or do whatever prevents the mistake from occurring again, than will not only prevent that particular bug from re-occuring, it will prevent a whole slew of bugs because overall quality is higher. Your customer will thank you for it.

No it’s not. It’s a product development method (or framework). The good thing is that projects always have a product development part, so Scrum can be embedded in your project. But beware of the  ’Scrum organizational model’ which is geared mostly to product development companies. If your organization chooses to have projects for all new development, you’ll need to embed Scrum in a project management model.

What is a project?

A project has a start, a middle and a finish. If I tell you this project will take approximately one year, an interesting process starts, most people assume there will be a single release at the end. Even if the project will be done with an Agile process, it will still be ’steering’ to that single release. All work is planned backward, burndowns are drawn to show you’re going to make (or miss) the deadline. Anyway, most of us know what a project is.

What is product development?

The origins of Scrum point to a different model. The term product backlog says it all, it’s not ‘project’ backlog, but product backlog. The default case for Scrum assumes you’re working on releases of a product, which has a virtually infinite lifespan like MS Windows, Atlassian Jira, Patientkeeper etc.

Scrum is a method for making development part of your entire business. Or better yet, if development is an integral part of your business, use Scrum. You rarely start projects because new features (large of small) are just put on the backlog. Developing software is what you do. This differs from organizations that do projects to not only develop software.

Can we do projects with Scrum?

Using Scrum for projects is surely possible, but you can’t use Scrum for projects without some project management practices like tracking budget, planning resources and loads of extra communication. For instance, if you’re using Prince2 and Scrum, you’re mixing two models, the ongoing development model of Scrum and the special initiative (a.k.a. project) model of Prince2.

A pure Scrum implementation should be embedded in a big vision, when you’re working on a product that vision is clear, product managers are hired for that purpose. On the other hand, in a project the variations are much more extreme, the choices of technology, budget, features etc. are much harder to make. In many cases projects are done by people who aren’t familiar with the context and history of the company. So expect a preparation phase where the team at least gets acquainted with the environment.

The thing to watch out for when implementing Scrum is that you don’t try to force the product development model onto your organization. If you use projects, accept that there is a ramp-up at the start of the project. And you might need a different process for different phases of the project. For instance, when your project is a one-off special campaign launching project, you can’t expect the whole organization to react quickly (just-in-time) to your impediments. It’s a matter of economics: any process changes they make for your project will be obsolete by the time your project finishes, which is not always worth the effort.

So what are you saying?

You can use Scrum in many contexts, it fits product development companies like a glove. Organizations that are project-driven can also benefit from Scrum, but not without some extra project practices. Expect a phased process and accept that in a project some impediments are simply not worth solving. Much of the frustration that goes along with Scrum implementations boils down to forcing the product development model (dogma) on your project organization. It’s like forcing a cube in a triangular hole, you’ll get frustrated and it won’t fit, no matter how hard you try. A project management method can be the adapter that makes Scrum fit.

Het indammen van inhuur en drastisch verlagen van tarieven heeft gevolgen. De markt zal zich aanpassen aan deze verandering in de vraag. Een kort overzicht van wat dit op korte termijn kan betekenen, ervan uitgaande dat er geen structurele verandering komt in de vraag:

Wat ik nog niet kan bedenken is, wat voor effecten heeft dit op langere termijn? Wat gebeurt er als de vraag weer aantrekt? Is er dan voldoende interne capaciteit opgebouwd om de nieuwe projecten te bemannen? Of zal er opnieuw een beroep worden gedaan op externen en begint het liedje weer van voor af aan? Graag hoor ik wat jij hiervan denkt.

Open Source

Hippo is open source. This has a number of benefits, such as a lower risk of vendor lock in. Hippo’s commercial success is a result the Dutch government’s preference for open source software and Hippo’s professional, local and Dutch support, a unique combination.

For the developer open source projects have the benefit of community support. Obvious examples such as Apache and a CMS like Wordpress show that open source can lead to solid products with tremendous support (books, articles, plugins, forums).

Alas, Hippo still has a fairly small community. Their mailing list is mostly used by it’s own employees. The amount of plugins is small and mostly created as spin off from commercial projects. Perhaps Java is one of the limiting factors because hosting Java is expensive, which prevents individual developers to use Hippo for their own blog or website (a limitation Wordpress or Joomla don’t have). Without a large community, customers would still have only Hippo to maintain their code because their knowledge of the system is not free.

User Experience

The user interface is another Achilles’ heel. The competition is also appalling in this respect (Tridion, Sharepoint), but user experience is on the rise because of it’s high competitive value. Wordpress, for example  has a large amount of goodwill simply because of the good user interface for editing content.

However, it’s not just the user interface, it’s also the type of information supplied. A professional content editor would like to know how people rate the content, where it’s used, how often is it read, what content is obsolete etc. Hippo does not provide this and thus cannot be used to share knowledge and be the single content repository for the customer. There is great opportunity here to beat systems like Sharepoint.

Writing plug ins

The documentation for contributing to Hippo is quite minimalistic. There is one tutorial, which didn’t work when I tried it because it was based on an old version. There are other weird things about Hippo’s documentation. The site contains quite a few dead links, not very reassuring. An even weirder issue is that the site changes while you’re reading it. The tutorial I read on Friday morning was replaces by a tutorial on Friday afternoon. It’s like the site is in constant motion and you never know when a page might disappear. It seems like testing is not part of Hippo’s publishing process and the CMS itself doesn’t have functionality to check dead links on the site.

Creating add-ons means you’re extending the CMS, which means you cannot drop in your plugin into a running CMS, you need to bundle it with the application, in essence making the add-on an integral part of the CMS. Any change in the internal API or storage system will result in rewriting add-ons. Thus the learning curve for add-ons is steep because you have to  know the entire architecture of Hippo before creating them. I would prefer a plugin architecture that makes it easy to drop in functionality and creates visible results quickly.

If you want community contributions, plugins should be easy to create, loosely coupled to the underlying architecture and easy to exchange and install. Google and Yahoo have created such great APIs that developer will use them just because they’re fun and easy.  And although a good plugin architecture is always hard to master, Eclipse has great tooling and support for writing plugins.

Wrap up

Hippo has a great crew and  it’s organizational structure (people, vibe) resembles a company like TomTom. I’m convinced Hippo can create great products and service. With this Forge Friday it has made a serious attempt to enlarge the developer community. But it needs to step up its effort if it wants a real community and a solid product. Open source without a community is just as useless as closed source products because the customer still depends on the vendor for maintenance. And a CMS without serious tools for managing content and its meta data is not worthy of managing any other content than pages on a website.

My tips for Hippo:

• Create a plugin architecture that makes it fun and easy to add functionality to Hippo.
• Make testing an integral part of the CMS (such as an automatic dead links checker)
• In future versions build the user interface of the CMS from the ground up with the content editor in mind. Make it the slickest tool to inspect, report on and publish company content (hire an information specialist).

After explain the usual method I’ll give you some examples which you can probably identify with. The next post will discuss the small solutions and their gripes.

To clarify, in this post ‘fixing problems’ means fixing them in real life, not just analyzing the problem. The process that leads to big solutions is something like this:

1. Get a clear overview of the situation
2. Simplify the situation
3. Fix the problem if it hasn’t fixed itself already (more on this later)

To illustrate let’s imagine the problem is picking all the green circles from this pile:

Now, the usual process:

1. Get an overview: place all the circles in a line:

2. Simplify the situation: Group the circles by color:

3. Fix the problem: it has almost solved itself as the green circles are on a separate pile. Now you might notice your mind altering between seeing all the different circles and seeing 4 piles. Just  by sorting the circles by color the problem was fixed. Isn’t that neat? And now we can reason about the piles in stead of all the different circles, which makes life easier still.

Abstractions

A pile is an abstraction of the circles on that pile. In most situations it’s far easier to talk abstractions than the actual things you’re dealing with because otherwise the amount of detail would be overwhelming. That’s why we like big solutions, when restructuring the context of the problem, it becomes easier to reason about. For instance if everyone uses the same operating system it’s easier to reason about PCs in your workplace because you can forget about the detail of the operating system.

To make abstractions, there has to be some visible patterns in the situation, otherwise it’s just a big mess (like the single pile of circles). To get to a point where simple abstractions can be made, people go very far in restructuring the situation, often destroying value in the process. One important thing to realize is people don’t favor big solutions, big solutions are a result of our desire to abstract and simplify, it’s a natural tendency with adverse effects (in some cases).

This is the issue with big solutions. It’s not a bad strategy to deal with complicated problems, but big solutions don’t solve the problem very quickly, they start paying of usually months or even years later.

When simplifying the context of the problem, you could for instance introduce a new method: throw out all existing social structures and procedures and replace it with a full implementation of a method that is ‘consistent and correct’ throughout all layers of the organization. The big problem here is destroying the existing structures that worked very well or were even specially tailored to the domain. The biggest issue with restructuring is the new structure will have problems of its own. So by fixing one problem you could be introducing several others that come along with your new method.

I’ll give you some other examples of big solutions from my recent experience:

• IT department is slow and expensive, we’ll replace the entire IT infrastructure with an off-the-shelf product.
• Our process is too slow and complicated in places, we’ll redesign and document all business processes and put them all in a BPM tool.
• The use of one technology doesn’t fit certain problems in our application, we’ll refactor all modules in all applications to use a new technology.
• We’ve got a big pile of legacy systems, let’s define a new architecture (SOA) and demand all projects to conform to that architecture from now on.

I also tend to come up with big solutions. These big solutions, when you first image them, always have sense of elegance, simplicity and control. When struggling with the current system architecture I come up with things like: “If we could only replace the entire back end with technology X” or “If we could only get 6 months to make tests for all the legacy systems”.

But while you’re restructuring, training people in new methods and drawing up conceptual charts you’re not actually fixing anything. You’re laying ground work to fix the problem in one big sweep. It might actually work too, the big solutions are not wrong per se, they just distract you from the problem and a smaller solution would have had a far higher return on investment.

Do you explicitly think when to use a big or small solution? How effective are big solutions? In my next post I’ll talk about ’small’ solutions and how they are a great alternative when you find yourself drawn to the big solution.

One major topic of any process optimization is the management of bottlenecks. One intriguing statement is made, it urges to go beyond the popular idea that ‘the capacity of the bottleneck controls system flow’. It’s not only the bottleneck, it also very important to look at the flow through the bottleneck.

The example given in the book discribes a traffic situation where three lanes are reduced to two. Combining two lanes into one causes turbulence and you don’t want this turbulence at the bottleneck, which is why the narrowing process starts miles before the actual bottleneck.

Although this is better to leave the narrowing totally unmanaged, in practice it often doesn’t work like this. On a recent car trip I found there is a better way of controlling flow which works much better than the regular solution, I’ll explain.

First the basic problem found very often on the Dutch highway, three lanes need to be narrowed to two:

The regular solution is to urge the car driver to start ‘turbulenting’ a few hundred meters before the third road ends. This usually doesn’t happen, what does happen is people cram their car into the bottleneck, usually with high speed. Their motivation is to slip through (escape) the bottleneck as quickly as possible. This decreases flow sometimes to a standstill.

An alternative solution, I recently experienced, is much more focussed on flow. This solution takes a two step approach, it splits the three lanes into two before combining them. This way the driver in the middle lane has two options when merging with another lane, to the left or right lane. This spreads the turbulence more evenly over the two other lanes. It also makes it undesirable to drive up to the road block as fast as possible because you haven’t ‘escaped through the bottleneck’ yet because the real bottleneck is still a few hundred meters ahead.

I’m not sure if the road workers made this arrangement on purpose to increase flow, but it’s a brilliant way of dealing with a bottleneck and minimizing it’s impact.

The relevance to organizations is to keep in mind that widening a bottleneck might not have the highest value, optimizing flow through the bottleneck can also reap huge benefits.

Suppose you have too few testers in your team. One thing you can do is hire more testers. But instead you can make sure the flow of work through the testers is optimal. In stead of the tester testing a feature as a whole after (or even before) it’s developed, involve the tester a few short times during development. This way he/she can detect testing issues before the full test and spread the ‘testing turbulence’ over a longer period of time, reducing the impact of tester-shortage. No extra testers, but increased flow. This is how I prefer to tackle bottleneck issues in my team, when increase in capacity is not at hand.

The problem lies not only in measuring the wrong objectives, keeping an eye on the budget is not wrong in itself, it’s using quantified objectives to measure success that has some big issues. Reading Demings arguments against management by objectives (MBO) makes me think twice before using cost or deadline to measure a projects’ success. Deming: “It is human nature to assume that there is something better about random events that meet arbitrary objectives, and assign their superiority to a non-existent special cause”. This is the heart of the matter: when a customer sets a deadline and his supplier meets this deadline, it’s human nature to assume the supplier did something special to meet the deadline. But there is no cause-and-effect here, meeting a deadline does not make you a good or bad supplier, it doesn’t mean anything.

Most proponents of Agile still use these same old arguments, arguing Agile will bring your project in on time and on budget. But this is only a superficial answer to the customers (real) objectives. Because variation of scope is used to attain these goals, you are actually doing exactly what Deming warned us about: cheating the system to meet the objectives. Why is this cheating? Because you’re still making the numeric objectives the goal of the project. What about the consequences of this behaviour? Are projects that are on time and within budget better than projects that are not. Perhaps the projects overshooting their deadline actually have a higher ROI. More likely there is no relation between ROI and staying within budget.

One way to get out of this objective trap, is to measure up. Don’t measure progress towards the deadline or the budget but focus on things like customer satisfaction, increase in sales or actual reduction in cost or process cycle time. It will free up your project managers to lead teams to do the right thing and include the complexity of attaining actual ROI into the project. Scrum originated from this holistic approach (pdf), but project management hijacked the burn down chart and inserted the classic objectives again.

One note of warning, I’ve never seen this happen myself. I do know of companies with good track records in this regard, but they are not stricly IT companies. Do you have the courage to add business value to your burn down chart and measuring up?

This post is on the ‘pre-iteration-1′ phase, before the project has even started. I’ll explain the contradiction that I’ve found in my experience with starting Agile projects. Preparing a project is not the same as executing a project and therefore the Agile principles don’t apply cleanly to the pre-project phase (oh, dear).

Starting a project is a delicate and risky endeavor. Some projects don’t turn out too well, much money is lost on starting the wrong projects. And yet, projects are the way forward for many organizations, the way to really get things done. So there is a fine line between starting too many risky projects and not starting projects at all (or only doing the ones that have a high chance of success). What would be an Agile way to deal with starting projects? To my knowledge there is not a lot of Agile information on starting or preparing projects, but two principles that apply here come to mind:

1. Take action (start the project)
2. Decide as late as possible (don’t start the project, yet)

The first principle says: “You’ll never know exactly how the project go, so just start and find out.” The second principle says: “If you can, wait for more information and then start the project (or not)”.

In a project context these principles wouldn’t conflict because they’re about the same thing: making decisions based on result, not speculation. Principle 2 comes from Lean and implies using an options-based approach to decision making. So when during your project you are faced with a complex problem, apply principle #1 and using real experience, do #2. But you can’t take action when you haven’t started your project yet, no team and no software means no real experience to prove viability of the project.

I find principle #1 is the favourite amongst my Agile colleagues, they have an eagerness to get the Agile process started. It has obvious benefits: shorten the project cycle time, get measurements on real world experience etc. So, when the decision to start a project seems too complex, the Scrum principles is applied (quoted):

[...] the values integral to Scrum, such as empiricism, self-organization, and action

But this can-do spirit has a pitfall: you might start too many projects that has no chance of success.

The heart of the matter is, can you make a viable prediction of a projects’ success without actually starting it? Starting projects just to find out how viable they are, is a costly approach. The Agile ‘ready, fire, aim’ principle, driven by the high number of developers pushing for Agile, might not be the best option in all cases. But the alternative might sound all too ‘waterfally‘. But I don’t think it’s the same thing as writing down all your requirements in detail up front. It’s actually not a project yet, so not all Waterfall vs. Agile flamewars apply.

Let’s say we have two options for the project preparation phase:

1. Very little preparation followed by starting the project. “Just do it and learn as you go”.
2. Doing research on the ROI, the needs of the user, getting buy-in from executives, getting a clear set of requirements, setting up reporting and project management processes etc. And only after getting a green light on all of these, start the project.

I will not say which is best on all cases, there is no such thing. But you might find it interessting to know that the Standish Group, in the book ‘My life is Failure‘ recommends option #2. And yet, they also recommend using an Agile process. The business case and the project plan should be sound and there’s quite a bit of groundwork to be done before even deciding on the start of a project.

You are partly in the land of theory-building, making plans, doing ROI estimates, asking what the user wants. All these things can change when you start the project. The benefit of this phase is that it doesn’t take a whole team and focussed organisation to execute this phase. When you decide not to do the project, only a few weeks of work is lost, but a huge los on a failed project is avoided.

My main insight here is: “Preparing to start a project, is simply not the same as executing a project”. A big difference is the preparation phase does not have a working product as its main deliverable, only plans and decisions. This means not all Agile principles apply and they should be forced onto this phase. The Standish Group gives us a lot of patterns for dealing with this phase, but they are differ from the Agile patterns. So pick the right pattern for the right phase and context. The preparation phase needs its own patterns for success.

I like to view it as a creative endeavor where I need freedom to shout boisterously with my fellow developers while writing great (if not brilliant) software. Non-developers will probably view this style of working as chaotic and most of all loud. Some of the projects I’ve done had a high noise level and some pieces of software really cannot be modified in silence. This style of development fits me pretty well, but I don’t know if it’s the best way to develop for everyone.

Quite another way of looking at it, is to view the developer as a craftsman. This invokes images of carpenters and sculptors that pass on their wisdom from generation to generation. The love of the trade and the product is central. Good tools and quality of the product. A long day of coding does sometimes feel like sculpting, carving out (deleting) large pieces of material from a massive brick (legacy code) to reveal a deceptively simple result.

Anyway you look at it, a good software developer has a love for technology and systems. There would be no other way she/he would endure all the fiddly syntax issues, obscure error messages and paradoxical problems. It’s like trying to write a sentence, but while you keep trying to end the sentence, the computer keeps saying ‘wrong syntax’. Most developer problems have to deal with this frustrating trail and error cycle. Somehow developers can endure this and find fulfillment in fixing the problem after a long struggle. The worst thing you can ask a developer after such an endeavor is: “Why didn’t you just do it right the first time?”.

What I haven’t figured out yet, is whether software development really needs making lots of mistakes. Do we need to debate the software product and examine it from all different angles? Or is development much more a matter of technology, tools and discipline, the perfect developer being someone who can turn a specification into working software without making mistakes?

I believe software development is a weird combination of creativity and discipline. The software we create needs to work in complex environments with unpredictable scenarios and users. The computer, on the other hand, is nothing like its environment, it’s ruthlessly strict. One misplaced bracket or statement can be enough to bring down an entire system. The world of the system is  linear, where everything is black or white, true or false and nothing in between.

Bridging the gap between the organic world where the problem lies and the computer system where (part of) the solution lies is the essence of a developers life. Not many people understand this nature of software development, including developers themselves. Software development will always be a profession made up of conflicts and challenges. I will need to be creative to find a simple solution for hard problems. But I will also long for that perfect program or technology  that will be able to solve all problems and situations.