MelGrubb.ToBlog()

Code Generation

Code generation is a great way to apply patterns consistently across a solution or to create multiple similar classes based on some outside input file, or even other classes in the same solution. The tooling has changed over the years, but the result is the same… a lot of code that I don’t have to write by hand, or update when patterns change and evolve.

I’ve been generating source code since the early days of .Net 1.0 when we used some plug-ins to generate source code based on class diagrams drawn in Visio. It worked, but it was a one-way trip because .Net didn’t have partial classes yet. We left that behind pretty quickly for a tool from Borland called “Together” that kept our code and diagrams in sync as you worked directly on either half. It was pretty awesome, but Together kind of died out when the price suddenly skyrocketed for version 2.0.

Then, somewhere around the Visual Studio 2008 timeframe I discovered T4 templates. A coworker had written a command-line program to generate mockable testing shims around Linq to SQL contexts, and I adapted it to run inside a T4 template so that we wouldn’t have to keep running it by hand and copying the results into our solution by hand every time we made a change to our models. These templates were stored as artifacts in our solution, and added code directly to our projects which was a big win. I was hooked, and used this approach to eliminate other repetitive coding tasks such as creating Builder classes for testing support, which is still my primary use case today.

T4 templates became untenable with the release of .Net Core, or at least the way I was using them, because the reflection code that I relied on no longer worked. T4 templates run under the full framework and cannot reflect over .Net Core code, or at least they couldn’t at the time.

I moved on to Roslyn templates and a project called “Scripty.msbuild” to transform them at build time. The .csx files that drove this approach did their job very well, and I was happy with how they worked. I could keep my reflection-based code mostly intact because I was just swapping out the runner. Unfortunately, Scripty was more or less abandoned in 2017, and when it started having some problems as well, I needed a new alternative.

Next, I tried using TypeWriter to generate my Builder code. TypeWriter was meant for creating javascript equivalent DTOs out of .Net code, but it can be used to create any kind of text file. Still, it’s clearly focused on emitting javascript and TypeScript, so more work was needed to get it to generate .Net code.

Most recently, I was back to using T4 again, but without the reflection part. The T4 engine still works just fine. We just need a different input than reflection if we’re going to use it on a .Net Core project. My current project is using an external designer to create the business entities, much like Visio and Together did back in the day. The XML file from this designer is used as the input for the templates instead of reflection, so T4 seems to be working just fine.

That’s great, and solves our problem in newer code, but the writing on the wall seems clear that T4 templates are not a thing Microsoft is interested in developing any further, and I imagine they’ll just go out of style completely at some point now that we have .Net Source Generators.

Source Generators run inside Visual Studio and can generate code in real time as you work, which makes them a lot easier to work with than the systems that came before them. They’re implemented as Visual Studio analyzers, which you’d normally use to do things like putting squiggly underlines in the code to suggest changes, and maybe even implementing “quick fixes” to make those changes for you. Source Generators take this a step further and let you create entirely new code that becomes part of the project, and not just at compile time. The code that generators create is available immediately. You can reference the generated code in your own classes, even in the same project, without having to recompile everything first.

It’s not all perfect though. The mechanism is still pretty young, and the Visual Studio experience is in its early days, so the process of creating a Source Generator isn’t totally smooth yet, but once you’ve written your generator, the act of using it is pretty painless.

Planning

The first step in generating code is, counterintuitively, to write that code by hand. Start by trying to write the thing you intend to generate and make sure it works the way you expect. Work out any kinks or problems and take it for a test drive before trying to automate it. Hash the idea out before creating a factory to mass produce it. It’s usually easier to write code than it is to write code that writes code.

The ideal candidate for generated code is a pattern that you’ve nailed down already and grown tired of writing over and over again. If you find yourself saying “I never want to write another {thing}” again, then that thing might be an ideal candidate for generation.

As you progress along, compare your generated output to the example class you started with in order to identify the parts that aren’t coming out quite right or that you haven’t finished yet. Tackle a feature at a time, and when your generator can output the same (or similar-enough) code to what you wrote by hand, then you’ll know you’re done.

For this demo, I won’t make you watch the entire process of evolving the whole system in real-time, so I’ll be jumping ahead quite a bit as we go. Just know that this is how my example was developed, one piece at a time.

First Steps

Create a new class library project to hold the source generator itself. I’m calling mine “BuilderGenerator”. This project doesn’t need to target a specific framework version, so either choose “netstandard2.0” when creating the project, or alter the TargetFramework node in the csproj file as follows. At the time I wrote this, you still had to specify the language version as “Preview” as well, although that may not be the case by the time you read this.

In addition, you’ll need references to the Microsoft.CodeAnalysis.CSharp and Microsoft.CodeAnalysis.Analyzers packages. You can do this through Visual Studio’s NuGet package manager which might find a newer version, or you can copy and paste these in from the example below and then go look for updates. The choice is yours.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <LangVersion>preview</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.2" PrivateAssets="All" />
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.9.0" PrivateAssets="All" />
  </ItemGroup>

</Project>

Remember to save your project file before continuing, or Visual Studio may get confused. This project can now be used as a source generator. We just need to fill in something for it to generate. Let’s create our “BuilderGenerator” class. You can call this class anything you want, but I’ll name it the same as the project and solution just to keep things simple. Decorate it with the “Microsoft.CodeAnalysis.Generator” attribute, implement the “ISourceGenerator” interface, and you should end up with something that looks like this.

using System;
using Microsoft.CodeAnalysis;

namespace BuilderGenerator
{
    [Generator]
    public class BuilderGenerator : ISourceGenerator
    {
        public void Initialize(GeneratorInitializationContext context)
        {
            throw new NotImplementedException();
        }

        public void Execute(GeneratorExecutionContext context)
        {
            throw new NotImplementedException();
        }
    }
}

Rather than throwing exceptions at runtime, which won’t get us very far, let’s start building out some code. We’ll start off small, and generate a class that says “Hello World!”.

using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Text;

namespace BuilderGenerator
{
    [Generator]
    public class BuilderGenerator : ISourceGenerator
    {
        public void Initialize(GeneratorInitializationContext context)
        {
            // Nothing to do... yet.
        }

        public void Execute(GeneratorExecutionContext context)
        {
            var greetings = @"using System;

namespace Generated
{
    public class Greetings
    {
        public static HelloWorld()
        {
            return ""Hello World!"";
        }
    }
}";

            context.AddSource("Greetings.cs", SourceText.From(greetings, Encoding.UTF8));
        }
    }
}

The work is done in the “Execute” method which is simply outputting a string containing the entire Greetings class contents to a file. This is the simplest form of code generation there is, simply adding pre-defined contents to the project that’s using the generator.

Using the Source Generator

Now that we have a source generator, albeit a boring one, let’s add it to a project to see it in action. Add a second project to the solution. I’m calling mine “Domain”. It will be taking the place of a typical domain layer and define some sample entities. This project can safely target a specific core framework version as usual. Edit the resulting csproj file to look like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <LangVersion>preview</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\BuilderGenerator\BuilderGenerator.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
  </ItemGroup>

</Project>

Note that the reference to the BuilderGenerator project has some additional properties on it. The first is what tells Visual Studio that the referenced project is an analyzer, and not a library. The second property, “ReferenceOutputAssembly = ‘false'” states this even more explicitly. As of now, you can’t reference a project as both an analyzer and a library at the same time, although I’m hoping this might change in the future because it would make things like base classes or attributes easier to implement.

This new analyzer will generate code that lives in the Domain project itself, although your ability to see it may be limited. Support for viewing generated code is improving all the time, so depending on the vintage of your Visual Studio installation, it may or may not let you browse to the generated code easily. You can, however, make references to it from other classes and/or projects, so let’s see if we can find it. Add a unit test project to the solution. I’ll call mine “UnitTests”. Add a reference to the Domain project, and then create our first test, called “GreetingsTests”

using NUnit.Framework;

namespace UnitTests
{
    [TestFixture]
    public class GreetingsTests
    {
        [Test]
        public void SayHello_returns_expected_string()
        {
            Assert.AreEqual("Hello World!", Generated.Greetings.HelloWorld());
        }
    }
}

Depending again on your version of Visual Studio, you may see some Intellisense errors at this point, and may have to restart Visual Studio before continuing. It appears to me that Visual Studio loads up the analyzer once, when loading the solution, and will keep using that analyzer for the rest of that session. This may or may not accurately describe what is really going on behind the scenes, but it’s certainly the effect I observed. Any time I made a change to the generator itself, I’d have to restart Visual Studio to take advantage of the changes.

Once the generator itself is finished, you should be able to go about using it “live” without any problems, but the process of working on the generator itself may be a bit of a chore. If you run into this problem, restart Visual Studio and Intellisense should start working again, including auto-complete. Hopefully these errors will go away in later releases and by the time you read this, it may have been solved once and for all.

Generating Real Code

Now that we have a working source generator, let’s make it do something more useful than returning “Hello World”. I’ve posted several times in the past about my love of the Builder pattern, and how it can help make your tests easier to write and also easier to read, communicating their intent more clearly to other developers. Since this is a pattern that I have implemented using just about every available source generation system in .Net, it seems like a natural choice for my first Source-Generator-based project.

Let’s start simply. For each entity class (e.g. User), we want to generate a corresponding “Builder” class (e.g. UserBuilder). The contents aren’t important yet, just that we can generate the builder class, and that we can refer to it from other classes within the same project, or from the tests.

First, we’ll create an attribute to mark the classes we want to generate builders for. Unfortunately, we can’t just define the attribute in the BuilderGenerator project itself. Our Domain project is referencing the BuilderGenerator project, but it’s doing it as an Analyzer, not as a library, so we can’t make direct use of the classes defined there. Instead, we can either create another project to house the attribute and other common elements such as base classes, or we can have the generator simply output the attribute class the same way we generated the Greetings class above. In fact, we can simply replace the existing contents of the BuilderGenerator class to emit this new attribute class instead.

using System.Text;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Text;

namespace BuilderGenerator
{
    [Generator]
    public class BuilderGenerator : ISourceGenerator
    {
        public void Initialize(GeneratorInitializationContext context)
        {
            // Nothing to do... yet.
        }

        public void Execute(GeneratorExecutionContext context)
        {
            var attribute = @"namespace BuilderGenerator
{
    [System.AttributeUsage(System.AttributeTargets.Class)]
    public class GenerateBuilderAttribute : System.Attribute
    {
    }
}";

            context.AddSource("GenerateBuilderAttribute.cs", SourceText.From(attribute, Encoding.UTF8));
        }
    }
}

We’ll reuse this same mechanism to generate any other attributes or base classes that we need.

Next, in the Domain project, create a folder to hold our entity definitions. I’ll call mine “Entities”. Create a new entity called “User” in the “Entities” folder, give it a couple properties, and decorate it with the “GenerateBuilder” attribute which should already be available if everything is working properly.

using BuilderGenerator;

namespace Domain.Entities
{
    [GenerateBuilder]
    public class User
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
    }
}

Again, the contents are not important at this stage. We just want something to generate a builder for. Since this is not a tutorial on Roslyn syntax, I won’t get too deep into the workings here, but we’re going to want what’s called a “SyntaxReceiver”, which acts as a kind of filter, sorting through the syntax tree that represents all of the code for our project, and forwarding the parts we’re interested in to our generator. Add a new class called “BuilderGeneratorSyntaxReceiver” with the following contents.

using System.Collections.Generic;
using System.Linq;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Syntax;

namespace Generator
{
    internal class BuilderGeneratorSyntaxReceiver : ISyntaxReceiver
    {
        public List<ClassDeclarationSyntax> Classes { get; } = new();

        public void OnVisitSyntaxNode(SyntaxNode syntaxNode)
        {
            if (syntaxNode is ClassDeclarationSyntax classDeclaration && classDeclaration.AttributeLists.Any(x => x.Attributes.Any(a => a.Name + "Attribute" == "GenerateBuilderAttribute"))) Classes.Add(classDeclaration);
        }
    }
}

The name isn’t actually important, but we’ll name it for the generator it serves to keep it clear. This filter looks for any classes in the syntax tree with the GenerateBuilder attribute on them, and adds them to a list called “Classes”.

Replace the contents of the BuilderGenerator.Execute method with the following:

        public void Execute(GeneratorExecutionContext context)
        {
            if (context.SyntaxReceiver is not BuilderGeneratorSyntaxReceiver receiver) return;

            foreach (var targetClass in receiver.Classes.Where(x => x != null))
            {
                var targetClassName = targetClass.Identifier.Text;
                var targetClassFullName = targetClass.FullName();

                var builderClassNamespace = targetClass.Namespace() + ".Builders";
                var builderClassName = $"{targetClassName}Builder";
                var builderClassUsingBlock = ((CompilationUnitSyntax) targetClass.SyntaxTree.GetRoot()).Usings.ToString();
                var builder = $@"using System;
using System.CodeDom.Compiler;
{builderClassUsingBlock}
using Domain.Entities;

namespace {builderClassNamespace}
{{
    public partial class {builderClassName} : Builder<{targetClassName}>
    {{
        // TODO: Write the actual Builder
    }}
}}";

                context.AddSource($"{targetClassName}Builder.cs", SourceText.From(builder, Encoding.UTF8));
            }
        }

There are a few things going on here. First, we make sure that we’re only paying attention to the classes that have passed through the SyntaxReceiver we defined earlier. For each class that passes the filter, which I’ve called a “target class”, we extract some basic information, and inject it into a template in order to create a builder class, which is then added it to the consuming project. That class is empty for now because our first goal is to simply generate a builder for each decorated entity class.

Since we’ve made changes to the generator itself, you may need to restart Visual Studio before it starts using the new version, but if everything is working properly, a new class called UserGenerator be added to the Domain project. Just like with the attribute, this file won’t appear in the Solution Explorer, but is should appear in the Class View, under Domain.Entities.Builders, and if you double-click to open it, you should see a new “UserBuilder” class with a comment that says “TODO: Write the actual Builder”.

You can already reference this class in your other code such as in unit tests. Delete the old GreetingsTests class, and replace it with a new “BuilderTests” class that looks like this:

using Domain.Entities.Builders;
using NUnit.Framework;

namespace UnitTests
{
    [TestFixture]
    public class BuilderTests
    {
        [Test]
        public void UserBuilder_exists()
        {
            var actual = new UserBuilder();
            Assert.IsInstanceOf<UserBuilder>(actual);
        }
    }
}

Now that we have the skeleton of our builder being generated, we’ll want to start filling in the parts that make it work, starting with the backing properties for the target class’ public properties. We start by pulling the list of settable properties from the target class, and adding a property to the builder for each one. Add the following just after the declaration of targetClassFullName in the Execute method.

var targetClassProperties = targetClass.DescendantNodes()
    .OfType<PropertyDeclarationSyntax>()
    .Where(x => x.IsInstance() && x.HasSetter())
    .ToArray();

Then, replace the TODO comment with a call to a new method called “BuildProperties”.

    public partial class {builderClassName} : Builder<{targetClassName}>
    {{
        {BuildProperties(targetClassProperties)}
    }}

And finally, define the BuildProperties method as follows.

        private static string BuildProperties(IEnumerable<PropertyDeclarationSyntax> properties)
        {
            var result = string.Join(Environment.NewLine,
                properties.Select(x =>
                    {
                        var propertyName = x.Identifier.ToString();
                        var propertyType = x.Type.ToString();

                        return @"        public Lazy<{propertyType}> {propertyName} = new Lazy<{propertyType}>(() => default({propertyType}));"
                    }));

            return result;
        }

I like to make my backing properties Lazy<T> because they don’t just hold the final property value of the completed object. Builders represent the plan for how to build that object, and plans are subject to change. Using a lazy value means that the code to set the final value won’t run until the object is built at runtime.

Next Steps

I’m not going to make you read through the entire evolution of my Builder pattern here because it’s only one specific use case, but the next step would be to add a method called “Build” that creates the final object, and a set of convenience methods to support a fluent interface. If you want to see the code that makes up my own BuilderGenerator, you can find the project source on its GitHub page.

There are many different examples of Builder classes out there, and some are more complicated than others, but that’s the beauty of code generation. If I expand on my pattern, to add new features, I only have to update the template, and the changes will be reflected across all my classes, or even across all my projects.

What I’ve shown here demonstrates the basic scaffolding of a Source Generator, and how to drive that generator’s output using the existing code in a project. From here, you would iteratively add a feature at a time until the generated code matches the hand-written example you started from. You can follow this pattern to generate any kind of highly-repetitive code that you want in your projects.

“Don’t Repeat Yourself”

Developers should all be familiar with this mantra, but we tend to think of it only as it relates to the code, and not the coder. It’s the second part that I want to talk about, so skip ahead if you want to, but I’m going to talk about the traditional DRY (Don’t Repeat Yourself) principle first.

Don’t Repeat Your Code

We factor out code all the time. We move things to a base class, or extract it to a helper so that we only have one copy to maintain. If we find a flaw, we fix it once, in one place, and the rest of the code reaps the benefits.

This is infinitely better than the copy/paste reuse model in which we’d have to spend not only the time to fix the problem, but to hunt down all of the other places where this code exists and fix them too. If, that is, you are lucky enough to work on a team that values craftsmanship and long-term maintainability of the codebase. All too often, you find yourself in a situation where some piece of code has been copied and pasted into a few locations before anyone realized that it was becoming into a pattern. One developer wrote something he needed, and then someone else needed to do either the same thing, or something very similar. It’s the “very similar” where things start to fall apart.

The problem is that there are now two copies of very similar code, but they’re not quite identical. If a flaw is found, someone has to find all the instances of that flaw and correct them. This is more time-consuming than fixing one single shared instance of that code. It gets worse though. Since the instances aren’t guaranteed to be identical, how do you know if you found them all? If the variable names are different in each instance, what are you even going to search for? Maybe you’ll get lucky and there are a couple keywords in a row that aren’t instance-specific, and you can search on those to find all of the copies. Hopefully, you recognize the pattern and extract it before it becomes widespread.

Once is unique, twice is coincidence, three times is a pattern. The “Rule of Three” states that we should have refactored out the first two instances as soon as we needed the same code a third time. Hopefully you don’t work in a control-freak environment where you are prevented from touching the first two instances because they’re not “your code”, or they are locked down because the original features they were written for are considered “done” and are not to be touched in any way ever again.

Don’t laugh. I know places like that.

That’s why we have unit tests in the first place. I want to be able to prove that my change didn’t break someone else’s code. I can only do that if I have adequate coverage of that code in the first place. With good test coverage, I can refactor with wild abandon, and know that I haven’t changed any of our documented assumptions about how the code should behave.

Don’t Repeat Your Mistakes

This is what I’m really here to talk about today. I want developers to start thinking about how the DRY principle relates to their daily activities, and in particular how it relates to our mistakes. We’re human, we forget things, we slip up, we communicate poorly with our team members, and things fall through the cracks as a result. When we find the resulting flaws, our instinct is to fix them and move on. The next time you find a problem, I’d like you to stop and think about whether you can do anything to prevent it from happening again in the future. Can you stop anyone else from making the same mistake?

One way I like to combat this problem is through unit testing, but not in the way you’re thinking. Developers who are “test infected” like to put tests around everything, but most stop at the immediate feature at hand. When a bug is discovered, we write a test that demonstrates it by failing (Red). We take our best stab at fixing the bug until the system works (Green). Finally, we clean up after ourselves, extract and refactor code to help out the next guy, and use the new test to check that we didn’t break the code in the process (Refactor).

Whenever possible, I like to write a “guard rail” test that lets me know whether anyone else has done the same thing. Here are just a few examples of this kind of test.

Missing Parameters

Here’s a unit test that uses reflection to let me know if any of my API controller endpoints have unused route parameters.

[Test]
public void Path_parameters_must_match_method_parameters()
{
    var apiAssembly = typeof(Controller).Assembly;
    var baseType = typeof(Controller).GetTypeInfo();
    var controllerTypes = apiAssembly.GetTypes()
        .Where(x => !x.IsGenericTypeDefinition
            && !x.IsAbstract && baseType.IsAssignableFrom(x));
    var violations = new List<string>();
    var regex = new Regex(@"(?<=\{)[^}]*(?=\})");

    foreach (var controllerType in controllerTypes)
    {
        var methods = controllerType.GetMethods(
            BindingFlags.Public | BindingFlags.Instance);
        foreach (var method in methods)
        {
            var attribute = method.GetCustomAttributes()
                .OfType<HttpMethodAttribute>().FirstOrDefault();
            if (attribute?.Template != null)
            {
                var routeParameters = regex.Matches(attribute.Template)
                    .Select(x => x.Value);
                var methodParameters = method.GetParameters()
                    .Select(x => x.Name);
                var unmatched = routeParameters.Except(methodParameters);
                violations.AddRange(unmatched.Select(x => 
                    $"{controllerType.Name}.{method.Name} - {x}"));
            }
        }
    }

    if (violations.Any())
    {
        Assert.Fail($"The following route parameters have no matching method parameters:\r\n  {string.Join("\r\n  ", violations)}");
    }
}

This particular test depends on the fact that we’re defining our routes directly in the HttpMethodAttribute (HttpGet, HttpPost, etc.), but it could easily be adjusted to account for explicit RouteAttribute usage as well.

The test extracts the route from the parameter, uses a RegEx to find all of the parameters in the format {param}, and then compares them to the method’s parameters. If there are any unused route parameters, the test fails.

Note: I probably could have collapsed the nested foreach loops into a single LINQ projection if I wanted to (I’m known for that), but in the case of this test, the readability was more important. I wanted other developers to be able to look at this and see exactly what was going on.

So what have we accomplished? Well, for one thing, no-one can accidentally leave out a route parameter without breaking the build. I’ve not only found and fixed my immediate a problem, but I’ve prevented myself or anyone else from repeating the mistake in the future.

Bad References

Here’s another example. Tools like ReSharper make it extremely easy to add missing references to a project, or “using” statements to a class. Tools make it so easy, in fact, that you can accidentally add a reference to something you shouldn’t. Here’s a test I made to make sure that no-one adds a reference from the API assembly to the Data assembly. There’s a service layer in between these two assemblies for a reason.

[Test]
public void The_Api_project_should_not_reference_the_data_project()
{
    var apiAssembly = typeof(Api.Startup).Assembly;
    var dataAssembly = typeof(Data.Startup).Assembly;
    var referencedAssemblies = apiAssembly.GetReferencedAssemblies();
    referencedAssemblies.Any(x => x.FullName == dataAssembly.FullName)
        .ShouldBeFalse("The Api assembly should not directly reference the data assembly.");
}

It’s simple, right, but it’s going to save me a huge headache caused by an errant “Alt-Enter-Enter”. I litter my test projects with these kinds of “structural” tests to raise a red flag if anyone repeats a mistake I’ve made in the past.

Right Concept, Wrong Place

The difference between Unit and Integration tests seems obvious to me. The former tests system components in isolation from one another, the latter tests the complete, fully-assembled, real-world system from end to end. It may do so against a fake database, or on a secondary network that simulates the real production system, but the important thing is that it’s the real code doing what the real code will really do. To make sure someone doesn’t get the testing patterns in the wrong place, here’s a one line test to make sure that the integration test assembly doesn’t touch the mocking framework.

[Test]
public void Integration_tests_should_not_reference_Moq()
{
    GetType().Assembly.GetReferencedAssemblies()
        .Any(x => x.Name == "Moq")
        .ShouldBeFalse("The Integraton test assembly should not reference Moq.");
}

A fresh, junior member of the team that’s new to the concepts and the differences between them will be reminded by this test if they are applying the right patterns to the wrong set of tests.

Right Attribute, Wrong Layer

In my most recent project, the DTOs (Data Transfer Objects) are returned from an API to a public consumer. They are, more or less, the ViewModels of this system, and as such, they need to be validated when they are posted back to an endpoint. We can do this easily through the use of the DataAnnotation attributes, but we need to make sure we’re using the right one. MaxLengthAttribute is used to control how Entity Framework will generate migrations, whereas StringLengthAttribute is used by MVC to validate models. They are very similar, but they are not the same. It’s very easy to slip up and use the wrong one, and then your validation won’t work.

These two tests work in tandem to make sure that you are using the StringLengthAttribute, and that you aren’t using the MaxLengthAttribute on any DTOs. Of course, this depends on your DTOs having a common base class.

[Test]
public void String_Dto_properties_should_not_use_MaxLengthAttribute()
{
    var baseType = typeof(Dto).GetTypeInfo();
    var types = baseType.Assembly.DefinedTypes
        .Where(x => !x.IsAbstract && baseType.IsAssignableFrom(x));
    var propertyTypes = types.SelectMany(x => x.GetProperties())
        .Where(x => x.PropertyType == typeof(string) 
            && x.GetCustomAttributes<MaxLengthAttribute>().Any())
        .Select(x => $"{x.DeclaringType.FullName}.{x.Name}");
    if (propertyTypes.Any())
    {
        Assert.Fail($"The MaxLengthAttribute is for controlling database generation. DTOs should use the StringLengthAttribute instead.\r\nThe following String DTO properties incorrectly use the MaxLength attribute:\r\n  {string.Join("\r\n  ", propertyTypes)}");
    }
}

[Test]
public void String_Dto_properties_should_specify_StringLength()
{
    var baseType = typeof(Dto).GetTypeInfo();
    var types = baseType.Assembly.DefinedTypes
        .Where(x => !x.IsAbstract && baseType.IsAssignableFrom(x));
    var propertyTypes = types.SelectMany(x => x.GetProperties())
        .Where(x => x.PropertyType == typeof(string) 
            && !x.GetCustomAttributes<StringLengthAttribute>().Any())
        .Select(x => $"{x.DeclaringType.FullName}.{x.Name}");
    if (propertyTypes.Any())
    {
        Assert.Fail($"The following String DTO properties have no StringLength attribute:\r\n  {string.Join("\r\n  ", propertyTypes)}");
    }
}

Check Yourself

These are just a few examples of kind of “structural” or “compliance” tests that I write on a typical project. If I can stop myself from making the same mistake twice, then that’s good, and will save me time down the road in the form of bugs I don’t have to investigate.

Every now and then I’ll be throwing some “Notes” out there. These are things that I’ve come across before, solved before, and then somehow forgotten about. Inevitably, the problem happens again someday, and I spend far too long searching the web for the solution that actually worked last time.

In the interests of helping my fellow programmers, and future me, I’ll just put these out there as part of my permanent record and maybe it’ll turn up in your search someday. Maybe that’s why you’re reading this right now. Maybe it’s why I’m reading this in the future. In that case; You’re welcome, future me.

Collecting code coverage in Visual Studio is pretty easy, as long as you have access to the Enterprise Edition. Fortunately, I do. I know not everyone does, and I think it’s ridiculous that Microsoft made code coverage an Enterprise feature. This is something we want to encourage the youngest of interns to absorb, not something reserved for senior Greybeards like me.

Anyway, inevitably, whenever I start analyzing coverage, I always run into the same problem. The test assemblies themselves have been counted in my totals. Since the tests all get run, the test coverage for those assemblies is nearly 100%, and it skews my overall results, making it look like my team’s coverage is better than it really is. What I want to do is exclude the test assemblies from the coverage numbers, but the ExcludeFromCodeCoverageAttribute can’t be applied to entire assemblies, and I don’t feel like adding it to every individual test class, so I fire up a web search and end up at the Microsoft documentation for .runsettings files. I follow the advice in the documentation, create a .runsettings file with nothing but a ModulePath Exclude section for and the next time I gather coverage, I see a bunch of assemblies that aren’t even mine. I’m talking about Nunit, JSON.net, etc.

The problem is that as soon as you specify modules to exclude, everything else gets included by default. You need to specify an Include section as well, but the syntax can be a little weird because it’s a RegEx, and now we have two problems. I try to write a simple RegEx to match only the files in my project, but the problem is that the files that aren’t mine are located in folders named after my projects, so my “magic words” are still part of their path, and they therefore get caught up in the results. I need a RegEx that will look for my root namespace as part of the filename, but not as part of the complete file path.

Enough already. What’s the answer?
Okay, here are the magic words. Assuming my root namespace, perhaps the name of the client, is “Foo”, my runsettings file will look like this:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
        <MaxCpuCount>1</MaxCpuCount>
        <ResultsDirectory>.\TestResults</ResultsDirectory>
        <TargetPlatform>x86</TargetPlatform>
        <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
    </RunConfiguration>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <Configuration>
                    <CodeCoverage>
                        <ModulePaths>
                            <Include>
                                <!-- Include only our own (Foo) assemblies -->
                                <ModulePath>.*\\Foo\.[^\\]*\.dll$</ModulePath>
                            </Include>
                            <Exclude>
                                 <!--Exclude the test assemblies themselves -->
                                <ModulePath>.*Tests\.dll</ModulePath>
                            </Exclude>
                        </ModulePaths>
                    </CodeCoverage>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

It’s that middle part there that’s the important bit, especially the RegEx in the Include section. It’s looking for the word “Foo” followed by any number of characters that aren’t a backslash (because I’m on Windows), followed by “.dll”. Users of other operating systems may have to adjust the direction of the slashes accordingly… I haven’t looked yet

Then, I have a more regular-looking Exclude section to ignore any file that ends in “Tests.dll”, which is the convention I happen to be following (e.g. Foo.UnitTests.dll”, “Foo.IntegrationTests.dll”, etc.)

Hopefully this helps someone. If you’re a RegEx ninja, please don’t kick sand in my face. If you’d like to share a better way of expressing this, I’m open to suggestions, of course.

It looks like Code42 isn’t interested in home users anymore, and they’ve announced that they are shifting focus to enterprise users. Even if you were backing up to your own computers, your account is still going away, and with it your ability to back up your stuff.

Now is the time to start looking for alternatives. I don’t have a recommendation yet, but I’m looking into it. I’m interested in hearing what everyone else is using, and how it’s been working out. I’d like to be able to recommend a drop-in replacement for the CrashPlan workflow, but nothing has quite fit the bill just yet.

For the short-term, Windows 10 users like myself can use Windows’ built-in backup system with a network share living on a Raspberry Pi. You can also use Resilio Sync or SyncThing to mirror your important files to the Pi.

On Wednesday, Aug 16th 2017, a new major Raspbian OS version was released. The “Stretch” release replaced the previous “Jessie” version, and makes a number of changes that may or may not affect you. I’m editing this post as I go through the process of upgrading my servers to the Stretch release, to let you know what I saw.

This will likely be another “live” post for a little while until I’m sure everything is stable, so remember to check back now and then, and please mention any problems you’ve seen in the comments.

Known issues (so far)

This upgrade is not quite ready for everyone to just jump in and do. I’m finding that a few things aren’t working correctly already.

Samba shares

My shares were gone after the upgrade, and even purging and reinstalling Samba couldn’t bring them back. I tried for hours to figure out what the problem was before deciding to go whole hog and do a dist-upgrade, which I normally don’t recommend. A lot of things can go wrong with a dist-upgrade since you’ll sometimes get newer versions of packages that aren’t quite ready for the real world. In this case, it worked. My shares are back online. DO NOT try this without a full backup. You’ve been warned.

Network UPS Tools (NUT)

The first time I tried the upgrade process, I got a rather troubling error message at the end saying that there were errors while processing “nut-client”, “nut-server”, and “nut”, so I gave apt-get upgrade a second pass just to make sure everything else was updated properly. Only these three packages had failed to upgrade, and it appears it’s because the nut-client service failed to restart. This is because the configuration files were overwritten by the upgrade, and didn’t have any useful information in them. After I restored my configuration settings, I was able to complete the installation. See below for more details.

Take a Backup

I should even have to tell regular readers to do this. We’re about to make major changes to the OS itself. You should shut down the Pi, and take a backup of the SD card before going any further. If you’re booting from a hard drive, then you’ll want to attach that to another computer and backup the root partition as well. After all, that’s where most of your stuff actually lives.

Consider upgrading from the desktop

If you can boot to the desktop, then I’d consider doing that. There are a couple times during the upgrade process where I found it convenient to open a second terminal window to examine something the upgrade process was going to change. If you connect through SSH, then you can use multiple sessions to achieve the same effect. If you’re running Raspbian Lite, and only use a directly-attached monitor and keyboard, then you may want to keep a note pad nearby in case you need to take note of proposed config file changes so you can restore your customizations afterward.

Update APT sources

To get the new software packages, APT will need to know where they live first. All you need to do is edit the two “source” files to point to the new “stretch” repositories.

sudo nano /etc/apt/sources.list

Change all instances of “jessie” to “stretch” and save the file. You can do this by hand, or you can let nano do some of the work for you. To do a search and replace, press ctrl-\. That’s the control key and the backslash. You’ll be prompted for the text to search for (jessie), and what to replace it with (stretch). If you haven’t done much to these files, then you should only find two matches, one on the first line, and one on the last line, commented out. Press “Y” for each match, or “A” to replace them all at once.

When you’re done, close the file, saving your changes (ctrl-x, y, enter). Next, do the same thing again, for a second file.

sudo nano /etc/apt/sources.list.d/raspi.list

Change all the “jessie”s to “stretch”es, and save the file. Finally, do a standard update/upgrade.

sudo apt-get update
sudo apt-get upgrade

What to expect during the upgrade

This is a pretty massive update, so you can expect this to take a while. Don’t just leave and come back later though because you’ll be prompted for answers several times during the upgrade process. The first prompt will be to approve the proposed changes, just like every time you do an upgrade, but the list will be huge. I’m talking hundreds and hundreds of packages have updates. You’ll also be prompted when configuration files have updates. I got prompted for changes to the following files

• /etc/skel/.bashrc
  I’ve never made any changes to this file by hand, so I just took the new version by pressing “y” and then “enter” when prompted. Note, these kinds of changes will default to “n”, so fight the urge to just hit enter like you’re used to. If you’ve made any customizations to this file, then you might consider opening a copy of the file in another window, and then reapplying your changes by hand when the update is complete. See the dhcpcd.conf step below for details.
• /etc/login.defs
  I took the new version of this file as well since I’ve never touched it myself.
• Graphical prompt for the keyboard language
  I let this one “guess”, which is the default option.
• /etc/dhcpcd.conf
  Now this IS a file that I’ve messed with. It’s how you set up static IP addresses on the Pi these days, so I first used the “D” option to examine the differences between the proposed new version and what I currently have. There were changes to things other than defaults; things I had modified by hand. I opted to open a copy of the file in nano from a second command prompt, and then hand apply my customizations when the upgrade completed. Press “y” to take the new version of the file. Don’t forget to come back and apply your customizations later on, though.
• /etc/lightdm/lightdm-gtk-greeter.conf
  Another file I haven’t touched by hand. I took the new version.
• /etc/nut/nut.conf and /etc/nut/upsmon.conf’
  I’ve definitely customized these as part of installing the battery backup (see Network UPS Tools), but the customizations aren’t that extensive. I opened a couple new terminal windows, opened the files in nano, and then took the new version of the file (“Y” option). The installation will fail to complete, but once we restore the customizations to these files, you’ll be able to pick back up and complete the process.
  • /etc/nut/ups.conf
    The part you’re interested in is at the bottom, and it’s where you set up the driver for your particular UPS. Mine looks like this:

    [RPHS]
     driver = usbhid-ups
     port = auto
     desc = "CyberPower SX550G"

  • /etc/nut/upsmon.conf files.
    This sets up the UPS monitor that’s in charge of actually shutting down the Pi when the power goes out. There are a few sections you’ll need here.
    The first is the MONITOR section. Mine looks like this:

    MONITOR rphs@localhost 1 upsmon NOTMYREALPASSWORD master

    The second section is the NOTIFYCMD. You will only have touched this part if you set up email notifications for power events. Mine looks like this:

    NOTIFYCMD /etc/nut/upssched-cmd.sh

    Finally, there’s the NOTIFYFLAG section. This tells NUT which power events you’re interested in getting notifications for. Not just email notifications though, this includes “wall” messages. Mine looks like this:

    NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC
    NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC
    NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC
    # NOTIFYFLAG FSD SYSLOG+WALL
    # NOTIFYFLAG COMMOK SYSLOG+WALL
    # NOTIFYFLAG COMMBAD SYSLOG+WALL
    # NOTIFYFLAG SHUTDOWN SYSLOG+WALL
    # NOTIFYFLAG REPLBATT SYSLOG+WALL
    # NOTIFYFLAG NOCOMM SYSLOG+WALL
    # NOTIFYFLAG NOPARENT SYSLOG+WALL

    It’s not important that your configuration looks like mine, and it probably won’t. The important thing is that you’re writing down, saving off, or opening a second window with your customizations so that we can restore them later on.

• Graphical prompt for the “/etc/apt/apt.coasdfnf.d/50unattended-upgrades” file.
  I haven’t touched this one either, so I took the new version.

Removing PulseAudio

The Jessie release of Raspbian used the PulseAudio library for Bluetooth audio. If you’re not using it, you can safely remove it.

sudo apt-get -y purge pulseaudio*

Restoring NUT

I decided to take the new version of the configuration files simply because I don’t know what else has changed in the newer versions, and my own customizations aren’t that extensive. Taking the new files will cause the upgrade to fail because part of the upgrade involves restarting the services, but the new configuration files are missing all of the vital information about your UPS. We’ll restore these files one at a time.

/etc/nut/ups.conf

Scroll to the bottom and restore your UPS information from above.

sudo nano /etc/nut/upsmon.conf

Restore the MONITOR, NOTIFYCMD, and NOTIFYFLAG sections from above. Then, we’re ready to take another shot at completing the NUT upgrade. Pick up where we left off with the following command.

sudo dpkg --configure -a

Apt-get is just a polite shell around the dpkg command which is really doing all the work behind the scenes. This command tells dpkg to finish configuring any outstanding packages. You’ll get prompted again to keep or overwrite your files. We’ve already overwritten them once, and then reapplied our customizations, so this time, choose the default option of “N” to keep your configuration files the way they are now, and the installation should complete successfully this time.

Restoring Samba

As I mentioned above, my Samba shares stopped working after the upgrade, and the only thing that seems to have helped bring them back to life is this:

sudo apt-get dist-upgrade

Normally, I’d say don’t do this. I used to do it all the time until I got burned. dist-upgrades are the bleeding edge of upgrades. Not everything has been tested to make sure it gets along well. Most of the time you’re probably okay, but it’s that one time in ten that takes your machine down that you can avoid by only doing normal upgrades.

Checking one last time

Just to be sure nothing got left behind, I did another apt-get upgrade. I noticed a note about packages that were no longer needed. I decided to leave them alone for now. There is also an extensive list of packages that have been “kept back”. You can force these packages to update with a “sudo apt-get dist-upgrade”, but I advise against that. You can read more about it here, but the practical explanation is that dist-upgrade can leave your system pretty broken.

You’re welcome to try it if you’re feeling daring, but I’ve had bad luck with it in the past and generally avoid it. There’s definitely no way you should even consider this without a fresh backup. You’ve been warned.

Changes in Stretch

One that I’ve read about is a change to the way network interfaces are named. From what I’ve read, this only affects new installations, and upgrades will retain their previous naming scheme, so an upgrade should be safe. Previously, you could count on the Pi’s Ethernet port being named “eth0”. Much like hard drives though, if you happened to have more than one Ethernet port, there is the possibility that their names could end up in a different order on any given day. That’s a pretty rare case though. Most Pi’s are only ever going to have the one port that they came with.