Continuous Delivery

How does the topic Continuous Delivery fit in a blog which is mainly on code architecture and code crafting?

In this post I hope to convince you that it fits quite well.

I am working on a team which has successfully practiced continuous delivery for some time and it turned out that pretty much everything we did affected our ability to actually deliver continuously.

In this context everything literally means everything in the software development life-cycle, from stated business goals to provable value for the customers. This does not only involve crafting the code but also deriving scope from business goals in a way that can guide the further process, which includes various kinds of testing and verification in parallel with coding, as well as operations tasks. Continuously.

The bad part is that if you wish to practice continuous delivery you need to rethink everything you do and you need to break quite a few habits. The good part is that it does not involve radically new practices, you only need to take existing and well-known practices more seriously and apply them consistently. Continuously.

It is common knowledge that the way we structure code, the way we break down complexity into manageable bits and the way we apply well-understood and agreed upon patterns consistently greatly affects our ability to deliver. So it is not a surprise that when we need to continuously deliver with a short cycle time, then all this becomes even more important.

What Is Continuous Delivery?

The short answer to that is look it up or read the book.

If you ask me, continuous delivery is really Lean Manufacturing principles applied to software development. The principles from physical manufacturing have been modified slightly in order to make sense in the software development world.

Do Lean Manufacturing Principles and Processes Really Fit With Software Development Ditto?

It is surprising how many physical manufacturing processes fit nicely with software development processes.

In lean manufacturing you want to have a short cycle time, meaning the time from feeding raw material into the factory and until the product is finished (and by then the product is hopefully valuable to the target user). This sounds awfully similar to what we want to achieve with continuous delivery, doesn’t it?

One way to achieve a short cycle time is to produce small batches, i.e. producing only a small number of items of a given type before switching to making another type of item. The challenge with small batches is that it takes significant time to set-up machines between two batches. And does it make sense to produce, say, 10 items in two hours, then spend 2 hours setting up a machine in order to produce 10 items of another type in two hours etc.? Wouldn’t it be better to produce 1000 items per batch, thus making the set-up time relatively small? The answer is that, yes it probably makes sense to have a small batch size, and no building up a large inventory is probably not a good idea. Optimize the process of setting up machines rather than increasing the batch size thereby avoiding the large amount of Work In Process (WIP) at any given time. The rationale is really quite simple and the logic makes a lot of sense. If you don’t believe me, I suggest you read the novel The Gold Mine: A Novel of Lean Turnaround. Yes, it’s a novel so it doesn’t really feel like working when you read it. But it will give you a gentle introduction to lean principles in manufacturing. (If you get really fired up on this topic I suggest you also read The Goal: A Process of Ongoing Improvement. It will teach you about the importance of focusing on bottle necks in the process.)

In the world of software development we have a similar challenge with batch sizes, the batch size in this context being the amount of code that we deliver (or anything else we deliver, but let’s focus on code for now).

We see the batch size challenge at multiple levels. At the highest level, the business would like us to turn the business goals into value for the customer as fast as possible, and one way to do that is to initially focus narrowly on minimal functionality. That’s a no-brainer, you say, but the business will never accept it – they always want it all and the want it yesterday, right? Well, it can be done. As Eric Ries describes in The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses it makes a lot of sense to quickly make a Minimal Viable Product (MVP) that can be validated and used in the further process. That is a small batch size which is used to achieve a short cycle time in software development!

We also see the batch size challenge for individual check-ins into a version control system and when merging change sets among branches. If you are really continuous you do continuous deployment and each check-in will be deployed directly to the live production system. This is not as risky as you might think if you have automated as much as possible, including tests, and if you can live with occasional hick-ups which you need to address fast. In my team we do not deploy each check-in, although we do deploy often after a short manual verification process that augments all our automated processes.

My advice here is to always strive at delivering baby steps, meaning small focused check-ins and small focused features. Any process which is in the way of doing baby steps must be optimized. If gated check-in takes 6 hours on a good day, then find another way to check the code. If code reviews have response times of several hours or days you need to look into that part of the process. If testing is a bottle-neck you need to address that, probably by adding resources in the short-term and doing more automation in the long-term (so that manual testing can be focused on new and UI centric features). Our goal must be to have as little WIP as possible, which in this context means code that we have spent time on but which has not yet been fully verified and turned into value for actual users.

That’s It

It is really that simple, deliver baby steps quickly, optimize any process that prevents you from doing that, automating as much as possible on the way.

But even though it is simple to state, it is not always easy to do it. I am writing primarily to code craftsmen, but before we go deep into core coding topics, you should convince those who control your process to read on Lean Manufacturing and Lean Startup. And if they get really fired up, they should also read Specification by Example: How Successful Teams Deliver the Right Software. In fact you, the code craftsman, and your testers should also read it – it could potentially help all three disciplines to work better together.

For true continuous delivery to work, developers must accept to be part of the full process, so a developer must accept to also partly work with operations. Read The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win and accept that we are no longer mere developers – from now on we are DevOps.

This was a long introduction. In my next post I will go deeper into how to craft code in the world of continuous delivery. I hope that by now you agree that successful continuous delivery requires that we think differently about the entire software development process. After my next few posts I hope you agree that we also need to think about code structure differently.

Then again, maybe you already do. After all, the coding practices I am going to describe are all based on existing knowledge and generally accepted practices, so maybe you do it all already.

Even More Asserts in a Single Unit Test Method

In my last post I stated that sometimes it is OK to have multiple asserts in a single unit test method and I devised a helper class MultiAssert for that.

Please do not get me wrong – I am a great believer in keeping unit tests readable and maintainable, and restricting a unit test to have exactly one assert is one way to achieve that.

But despite that I am now going to argue that sometimes I find it to be OK to have multiple asserts in a unit test method, even without packing all the asserts up using MultiAssert.Aggregate.

Remember that “The arguments against multiple asserts are multiple, a main one being that if one assert fails the rest will not be executed” (from my last post).

However, what if I in this case explicitly want the rest of the unit test not to execute – is it then OK to have one assert ensure that another is not executed? I think so. Read on and I will explain.

The other day, I looked into a unit test that failed randomly. I knew that a person whose unit testing skills I do not usually question wrote it. Still, it turned out that it was surprisingly tricky for me to figure out why it failed.

The following is a simplification of the original unit test,

public void DoThis_MustCallDoThatOnFooWithExpectedParameters_WhenCalled()
_target.Initialize(_foo, "first", "second");
f => f.DoThat(
Arg<string>.Is.Equal("first"), Arg<string>.Is.Equal("second")));

In order to figure out why this unit test failed, I started making assumptions.

My first assumption was that DoThat was called with at least one of the two parameters having an unexpected value, so I added a line to the unit test to let Rhino tell me what the actual parameter values for the DoThat call was,

var actualArgs = _foo.GetArgumentsForCallsMadeOn(f => f.DoThat("", ""));
() => Assert.AreEqual("first", actualArgs[0][0]),
() => Assert.AreEqual("second", actualArgs[0][1]));

This did not help me, as inspecting actualArgs only caused an index was out of range exception to be thrown.

My second assumption was that DoThat was not called at all. To test this hypothesis I tried the following,

_foo.AssertWasCalled(f => f.DoThat(Arg<string>.Is.Anything, Arg<string>.Is.Anything));

Bingo! The unit test still failed (randomly) with this assert, which showed me that DoThat was not called.

So the single assert of the original unit test was actually multiple asserts behind the scene – one assert to state that DoThat was called and two asserts to state that each of the two parameters were as expected. This is one case of having multiple asserts that I do not like!

With this knowledge, it was quite easy to track down the root cause of the failure. Somebody had checked in a parallel implementation of _target.Initialize so that the initialization of _target randomly made it to completion before _target.DoThis was called. While I realize that it is important to figure out what this new parallel code will do in production code, for now I will keep focus on the correct implementation of this unit test method.

Since we have three asserts in this unit test, some would say that it is obvious to split it into three separate unit tests? Well, I would go for one or, perhaps, two. Read on and I will explain.

If we want to split into three unit tests, that would be

  • One to check if DoThat was called at all,
  • One to check that firstParam had the expected value, and
  • One to check that secondParam had the expected value

It could be argued that this is a lot of repeated setup that should be avoided. I do not agree – common setup can be put in auxiliary methods and execution will be very fast because I have mocked out external dependencies to the code under test.

My problem with the tree unit test approach is rather that the first unit test is a prerequisite to the two other unit tests; if the first fails, the two other will certainly also fail (barring random behaviour). I believe that if you have an assert that, if failed, will make another assert fail with certainty, then these two asserts can, and often must, be packed together into a single unit test method.

So does that mean that the correct implementation of this unit test is to split it into two unit tests?

  • One that checks if DoThat was called, and also checks that firstParam had the expected value, and
  • One that checks if DoThat was called, and also checks that secondParam had the expected value.

I believe that the most maintainable way to implement this is to have a single unit test.

  • One that checks if DoThat was called, and also uses MultiAssert to verify that both firstParam and SecondParam had the expected values.

All in all, this means that we have changed the original – hidden – multiple asserts into two explicit asserts. This is better than the original, as the first assert is a prerequisite of the second, which means that it makes no sense to execute the second if the first fails.

Here is the resulting implementation,

public void DoThis_MustCallDoThatOnFooWithExpectedParameters_WhenCalled()
_target.Initialize(_foo, "first", "second");


_foo.AssertWasCalled(f => f.DoThat(Arg<string>.Is.Anything, Arg<string>.Is.Anything));
var actualArgs = _foo.GetArgumentsForCallsMadeOn(f => f.DoThat("", ""));
() => Assert.AreEqual("first", actualArgs[0][0]),
() => Assert.AreEqual("second", actualArgs[0][1]));


Multiple Asserts in a Single Unit Test method

The title of this post is a provocation to many people who have read and love Roy Osherove’s brilliant book, The Art of Unit Testing. In this book Roy clearly states that one of the pillars of good tests is to avoid multiple asserts in a unit test.

The arguments against multiple asserts are multiple, a main one being that if one assert fails the rest will not be executed, which means that the state of the code under unit test is really unknown. Another argument is that if you find a need for having multiple asserts, it is probably because you are testing multiple things in a single unit test method. This will break the principle of single responsibility and maintainability will suffer.

I am a great believer in having maintainable and readable unit tests and I have always tried to follow the single assert advise myself. I am also a great believer in the principle of single responsibility, although I am often forced to be pragmatic when working on legacy code. When I want to test several outcomes from a single object I can choose to implement Equals, or maybe ToString in order to do direct comparisons of whole objects. Sometimes I will try to make a utility method or class that will allow me to compare several values in a way that will fit a single assert. While some people do not like adding to the code base for unit testing purposes only, most people object to having too many utilities creeping up in the unit test projects.

Recently I had discussions on unit testing with my colleagues and the reasoning behind single asserts came up – and also some arguments against it.

Let’s have a look at one of Roy’s examples,

public void CheckVariousSumResults()
    Assert.AreEqual(3, this.Sum(1001, 1, 2));
    Assert.AreEqual(3, this.Sum(1, 1001, 2));
    Assert.AreEqual(3, this.Sum(1, 2, 1001));

The problem here is that if one assertion fails, the rest will not be run and we do not know if they would fail if run.

There are a number of solutions to this.

The first solution: Create a separate test for each assert

This is easy and it only takes a few seconds to write those unit tests,

public void Sum_1001AsFirstParam_Returns3()
    Assert.AreEqual(3, this.Sum(1001, 1, 2));
public void Sum_1001AsMiddleParam_Returns3()
    Assert.AreEqual(3, this.Sum(1, 1001, 2));
public void Sum_1001AsThirdParam_Returns3()
    Assert.AreEqual(3, this.Sum(1, 2, 1001));

What is the problem with this solution?

Well, although the example may be slightly contrived it is easy to imagine that the three cases are somewhat correlated. By putting all three asserts in a single method we have signaled that these must be considered as a whole in order to be understood, while if we create separate unit test we have lost this information. And imagine that there were more than three cases, say 42? If a fundamental bug in the Sum method creeps in so that all 42 unit tests fail, would you prefer to have 42 unit tests fail or would you prefer to have a single unit test fail?

Another problem is maintainability. It is correct that it only takes a few seconds to write these three unit tests, but someone needs to maintain them in all future and the task can become daunting due to the sheer number of unit tests.

Both problems can to a certain extend be overcome with proper naming and with true single responsibility of units under test as well as each unit test method, but that is not always the reality – especially when you try to put legacy code under unit test.

The Second Solution: Use Parameterized Tests

In many cases I would prefer to use parameterized tests. However, currently my unit testing environment is Visual Studio 2010 and it does not support such a feature!

The Third Solution: Use try-catch

Since an assertion failure means that an exception is thrown, at least in the unit test frameworks I have used so far, we can simply catch such exceptions, do some intelligent processing, and then allow the next assertion to fail or succeed. That solves our problem with having multiple asserts.

Even though Roy is my unit testing hero, I think he is a bit too hasty to simply abandon the try-catch solution with a statement like "Some people think it’s a good idea to use a try-catch block […] I think using parameterized tests is a far better way of achieving the same thing."

A Simple Solution Using try-catch

Since I cannot write parameterized unit tests with my unit testing environment, I had to come up with an alternative solution. My solution is to introduce a new MultiAssert class which will accept delegates to multiple assert statements but only fail at most once. This new class seems to be a logical addition to the existing family of assert classes along with e.g. CollectionAssert and StringAssert.

Here is the above example in a single unit test with a single assertion,

public void CheckVariousSumResults()
        () => Assert.AreEqual(3, this.Sum(1001, 1, 2)), 
        () => Assert.AreEqual(3, this.Sum(1, 1001, 2)), 
        () => Assert.AreEqual(3, this.Sum(1, 2, 1001)));

MultiAssert.Aggregate can even be used in situations that do not fit parameterized unit tests easily.

Here is the implementation of MultiAssert.

public static class MultiAssert
    public static void Aggregate(params Action[] actions)
        var exceptions = new List<AssertFailedException>();

        foreach (var action in actions)
            catch (AssertFailedException ex)

        var assertionTexts = 
            exceptions.Select(assertFailedException => assertFailedException.Message);
        if (0 != assertionTexts.Count())
            throw new
                    (aggregatedMessage, next) => aggregatedMessage + Environment.NewLine + next));

Using or Abusing Multiple Asserts

MultiAssert can be abused, it is not meant as a universal excuse for cramming a lot of assertions into any unit test method. Remember that maintainability and readability of unit tests must still be a top priority and you should only use MultiAssert when this can be achieved.

One situation in which I recommend the use of MultiAssert is when it makes sense to assert both pre- and post-conditions in a unit test method. In this context, a post-condition is simply a (single) assert that states something about the state of the world after the Act part of the unit test method. However, if you assert that something has the value 42, how do you know that this was not already true right after the Arrange part of the unit test? After all, the Assert part of your unit test must assert what was supposed to happen as a consequence of the Act part of the unit test method.

So one nice usage of MultiAssert is to assert both pre- and post-conditions in unit tests.

public void Foo()
    // Arrange
    var underTest = ;
    bool preCondition = underTest.TheFoo() != 42;

    // Act
    int actual = underTest.TheFoo();

    // Assert
        () => Assert.IsTrue(preCondition),
        () => Assert.AreEqual(42, actual));

Setting the Windows Phone 7 SMTP Port

I am now the happy owner of a Windows Phone 7 device, Samsung’s Omnia 7.

I am a happy user of Samsung’s hardware as well as Microsoft’s software, including the new Metro user interface, My Phone’s ability to locate my phone, the Zune software etc. And I am thrilled about the developer story – that apps can be developed using Silverlight and C# with Visual Studio and other familiar tools.

I intend to blog about developing apps as soon as I get proper hands-on experience.

One end-user problem struck me immediately when I tried to setup the phone. I use a hosted email solution which I can use to read my email through POP3/IMAP and I can send email through SMTP, all of which WP7 supports nicely.

Well, except that there seems to be no way to specify the SMTP port to be used, which is odd since it is quite common to have the default port 25 blocked?

As usual, there is a simple solution to this problem – simply specify the port after the SMTP server name. For example, if the server name is ‘’ and the port to use is 42, specify this as ‘’.

The Silverlight 3 SelectedIndex bug

There is a Silverlight bug which bothers a lot of users worldwide – if you bing the issue you will find quite a few hits.

The Problem

The problem is simple to state. Create a combo box and bind to the SelectedIndex and ItemsSource properties and you get an out of bounds error when a value is selected.

<ComboBox SelectedIndex="{Binding Path=IndexOfCurrentValue}" ItemsSource="{Binding Values}">

One of the sources that describe the problem is This post includes a work around but I wanted to find a simpler solution.

I asked the Silverlight product team and the answer was that this bug is very unfortunate, but that it may not be trivial to fix.

In my repro project, I bound SelectedIndex before binding ItemsSource, so when the binding engine sets the SelectedIndex property, items are not set yet and the value 0 is not correct.

The Solution

The workaround is simple; simply switch the order of bindings in XAML and bind ItemsSource before binding SelectedIndex.

This is a reported bug and it is still open at the time of writing this. I hope it will be fixed by Silverlight 4.

Silverlight and WPF Single-Sourcing – XAML namespaces

I have found that WPF and Silverlight XAML are sufficiently different to make single-sourcing difficult.

At first I did not know if it was possible at all, but I knew one thing – the differences in usage of namespaces look like something very annoying for which there must be an elegant solution.

The Problem

The first issue was with the standard controls which are in WPF 4.0 but not in Silverlight 3.0 (only in the toolkit), e.g. the Expander is declared like this in WPF,

<Expander … />

But it is declared like this in Silverlight,

… xmlns:Controls1="clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls.Toolkit" …
Controls1:Expander … />

What I really did not understand was that there is a similar problem when the standard controls which are in WPF 4.0 also are in Silverlight 3.0; for example, the HierarchicalDataTemplate is declared like this in WPF,

<HierarchicalDataTemplate … />

But it is declared like this in Silverlight,

… xmlns:Windows="clr-namespace:System.Windows;assembly=System.Windows.Controls" …
Windows:HierarchicalDataTemplate … />

Clearly the rules for defaulting and declaring namespaces are different.

The First Solution

I searched the Internet and found a few suggestions. One suggestion is to have two sets of name space references. This works because you are allowed to use name spaces on WPF although it is not mandatory. I did not like it because it still meant maintaining different files since XAML does not allow #IFDEF-like constructs.

My colleague David Anson found a surprisingly simple solution which he describes in a blog post (see the last bullet in the notes at the end of the post). The main idea is to subclass standard controls, effectively bringing them into the same namespace across Silverlight and WPF.

It looks like magic – and ReSharper also claims that this is not valid – but it works. Well, it almost works.

The Problem with the First Solution

So far I have found one problem with the first solution – the HierarchicalDataTemplate.

If I used David’s trick,

public class HierarchicalDataTemplate : System.Windows.HierarchicalDataTemplate { }


41: <Std:HierarchicalDataTemplate x:Key="hierarchicalDataTemplate" ItemsSource="{Binding Path=Subcomponents}">
42:    <StackPanel Orientation="Horizontal">
43:        <
Image Source="{Binding Path=TreeIcon}"/>
44:        <TextBlock Text="{Binding Path=Name}"/>
45:    </StackPanel>
46: </

Then it compiled in Silverlight and WPF but the StackPanel caused problems at run-time in WPF. The error was

‘StackPanel’ object cannot be added to ‘HierarchicalDataTemplate’. Object of type ‘System.Windows.Controls.StackPanel’ cannot be converted to type ‘System.Windows.FrameworkElementFactory’.  Error at object ‘System.Windows.Controls.StackPanel’ in markup file ‘Microsoft.Dynamics.Ax.Frameworks.Controls;component/singlesourcetest.xaml’ Line 42 Position 26.

It did not help to use David’s trick again and add StackPanel to the Std namespace.

This is a WPF bug. It has not been fixed in the WPF 4.0 beta I am using but I hope it will be fixed eventually in the RTM.

The Second Solution

David worked hard on this problem and got another brilliant idea; if this is a problem with WPF only then why not subclass in Silverlight only? This requires a little trick that left David "a little slimy" – see his blog post for the details – but it works. Well, it almost works.

The Problem with the Second Solution

I took the sample project from the blog post for a spin and found that it would not work if I added x:Name="myTreeview" to the TreeView in XAML. This is the error I got,

The type ‘System.Windows.Controls.TreeView’ exists in both ‘c:\Program Files\Microsoft SDKs\Silverlight\v3.0\Libraries\Client\System.Windows.Controls.dll’ and ‘e: \SharingXamlSilverlightWpf\PresentationFramework\Bin\Debug\PresentationFramework.dll’                e:\enlistments\axmain6\source\frameworks\controls\sharingxamlsilverlightwpf\sharingxamlsilverlightwpf-sl\obj\debug\mainpage.g.cs      38           42           SharingXamlSilverlightWpf-SL

Clearly there is a problem with disambiguating identical names across assemblies.

I needed to reference the name of the tree from a details view so I could not use the second solution for at least the controls that I needed to name.

But since the second solution seemed to only be needed for HierarchicalDataTemplate anyway, I am now using the first solution for everything but the HierarchicalDataTemplate.

And this works. Well, it almost works. But the remaining problems seem to be unrelated to this single-sourcing approach so I will defer blogging about it till later.

WPF: How to mark an input field as not valid

I wanted to add to my input controls a visual clue that tells if the current content of the control is valid.

The visual clue I had in mind was a red squiggly line, similar to the way MS Word underlines my typos.

Why this particular visual clue? Because this is how it is done in MS Dynamics AX,


This left me with two tasks,

1. How to trigger the actual validation.

2. How to change the default WPF visual clue (a red border around the control) to the desired red squiggly line.

How to trigger validation

I thought about using validation rules similar to what Nigel Spencer blogged about. But validation is only triggered when the binding value has changed, while I wanted the visual clue to work even at start-up.

I was not alone with this problem. Willem Meints blogged about this as well as about several other validation problems. He actually wrote a complete validation framework that I found would be too much of a good thing for me. He also wrote that he did not use IDataErrorInfo as it does not provide enough flexibility.

I like to keep things simple, so I opted to use IDataErrorInfo.

Basically, I let the business object I bind to report an error if it is not happy about what the user typed so far. It can be augmented by validation rules, e.g. a user defined NumberRangeValue.

For the simple case, meaning the check for mandatory fields having a value, I use code similar to the following,

public string this[string columnName]
        if (string.IsNullOrEmpty(columnName))
            return string.Empty;

        if (<this is a mandatory field and has no value>)
            // This is never shown in the UI and does not
            // have to be localized; but if we ever choose
            // to show this text it must be localized!
            return "This field is mandatory.";

        return string.Empty;

Now I need to figure out how to trick the visual of controls to show a squiggly red line instead of the default red border.

How to get the red squiggly line

The TextBlock control can actually do the trick with the red squiggly line with its built-in spell checker. Alas, there is no API to trigger this at will.

Instead I found that the following XAML placed in a Resources section would do the trick,

<DrawingBrush x:Key="squiggleBrush" TileMode="Tile"
        Viewbox="0,0,4,4" ViewboxUnits="Absolute"
        Viewport="0,0,4,4" ViewportUnits="Absolute">
        <GeometryDrawing Geometry="M 0,2 L 1,1 3,3 4,2">
                <Pen Brush="Red" Thickness="1"
                StartLineCap="Square" EndLineCap="Square"/>
<ControlTemplate x:Key="SquiggleError">
<StackPanel HorizontalAlignment="Center" VerticalAlignment=
        <Rectangle Height="4" Fill=
            "{StaticResource squiggleBrush}"/>
For e.g. a combo box I apply this template as follows,
<ComboBox >
DynamicResource ResourceKey=”SquiggleError”/> </Validation.ErrorTemplate>
All this ensures that the red squiggly line is drawn when I want it, and it looks close enough to the real thing,