You're doing it wrong

Attention software developers: you're doing it wrong.

To be more specific, you need to be doing these things, and you're not.

Don't block the UI thread!

No matter what the application is, not matter what you're doing, do not block the UI! A "Please Wait" dialog or screen is trivial to code and makes the user experience so much better. You can never assume that an operation will always execute quick enough not to block. It always will under some scenario.

Catch everything!

There is no excuse (at least in the .NET world) to allow an exception to bubble up so far that the application crashes. There is always a way to catch exceptions at the highest level. Even if there's no way to recover, exit gracefully.

And don't restart the app without warning. Visual Studio and SQL Server Management Studio are big offenders here. If recycling is the only way to recover, fine, but don't just do it without telling me!

Always format numbers.

I'm so sick of having to squint at 41095809 and mentally insert the commas (or whatever the delimiter is in your culture). Having the code format this kind of output is so trivial. Software is supposed to make our lives easier. If you don't do this you have failed.

Don't confuse a pattern with a function.

Okay this one's not quite so obvious. There are times when you have to do a similar thing in lots of places. The natural inclination is to make this into a function/method/class. This is a good instinct! Don't repeat yourself! But it can be taken too far. Some things look similar but have enough differences that trying to create a generic base for them just makes the code way more complicated.

To put it another way, you wouldn't try to write one ForEach() function that handles all loop cases. It would quickly become a monster. Looping is a pattern, not a function.


Let's be clear: I have committed all these sins and will again. Learn from my fail, and we can all stop doing it wrong.

Stop describing me!

Anyone who's ever worked in software knows how difficult it is to keep the code and the documentation in agreement. It's all too common to find errors in documentation, and not just typos, but misleading or even just false statements. This is (generally) not the result of malice, laziness, or stupidity, but is rather a natural consequence of trying to keep two unrelated things in sync with each other. Programming and natural languages are just not similar enough to make this a simple or automatic task. It's one reason experienced programmers are prone to say that all code comments are lies.

Additionally, documentation is far too often a crutch used to make up for the fact that the design is not intuitive. As Chip Camden once said, "Documentation helps, but documentation is not the solution - simplicity is." That doesn't mean that software can't be doing complex things - doing complex things is the whole point! What it means is that the interfaces to software, whether they be APIs or GUIs, have to make the complex thing simple.

Documentation then is typically a symptom of poor design and a difficult symptom to manage at that. But how then do we communicate the what and why of a software system?

The answer is self-documenting systems. "Self-documenting system" sounds like the tagline of some new tool somebody's trying to sell, but it's actually a design philosophy. It's an approach that says: make the endpoints, buttons, menu options, links, namespaces, class, method, parameter, and property names so obvious as to their purpose & function that, for the majority of them, documentation would be redundant.

The term "redundant" is used very deliberately here - it's not that documentation is never warranted, it's that we should seek to make it unnecessary, for the reasons mentioned above. There will still need to be a few bits of explanation here and there, but they'll be few, so maintaining them won't be a big burden, and they'll be very targeted, so it'll be very obvious when they need to be updated. (By targeted, I mean they'll be either high-level descriptions of the unit as a whole, or notes on some edge cases.)

Self-documenting systems lead us to the Principle of Minimum Necessary Documentation, which states:

There is an inverse relationship between how well designed something is and how much documentation it needs.

The term "inverse" is also used very deliberately. The inverse of infinity is not zero - it's darn near close to it, but it's not actually nothing. A perfect system needs no documentation, but there are no perfect systems, so there is no system that can have zero documentation, no matter how well designed it is. But for a well-designed system, a README file or some tooltips will probably suffice.

But do not assume from this that a project with no documentation except a README is well-designed, or that you can get away with just throwing together a README and taking no other thought for what's going on. Note the principle says "how much documentation it needs" not how much it has. If users have cause to say "the documentation has been skimped on", then the design is poor and no crutch was provided, which is even worse than a bad design band-aided with documentation. Intuitive, self-documenting design is the critical factor, not the documentation itself.

Now, there are a few caveats & clarifications that should be mentioned:

• 'Self-documenting' is often used in connection with test-driven development. There is a vein of thought that says the tests should be the mechanism for documenting the code. This is a very good approach to take, but tests cannot serve as the user documentation. If the project is a public one, expecting people to read the tests is not realistic, and even with an internal project, understanding the guts of how a component works is often beyond the scope of the task at hand. Tests can and should be the developer documentation, and complete test coverage is critically important to developing robust software, but a test suite is typically too large and specialized to be the sole documentation.

• Most systems have a fairly obvious surface. It may not be instantly obvious what a class structure or API or SDK or GUI does, but it's usually pretty easy to see what it has. There are interfaces though that don't have an easily reflectable surface, such as command-line tools. At first blush, it might seem that this is a place where more documentation is a necessary evil. Certainly there are a lot of tools that believe extensive documentation is preferable to abstraction (git, I'm looking at you). But the principle does still apply here. It is still incumbent on the developer to make the behavior intuitive (typically by following conventions), and to make the options easily discoverable. There are a myriad of ways to approach this, and it depends a lot on how flexible the tool needs to be. It requires more work, but that's our job: to make it easy to use.

It's as I've said before: deal with things directly. Don't build hedges around them, because it just takes more work and isn't as effective. Don't worry about documentation unless there truly is no other way to get the point across. Focus instead of making your code as intuitive and straightforward as possible, as this is the greatest programming good.

Learn the Rules so you can figure out who's breaking them

99.99 percent [of subatomic interactions] are explainable ... Spending your time exploring each particle trail will lead you to conclude that all the particles obey known physics, and there's nothing left to discover ... Most stars are boring; advances comes from studying the weirdies - the quasars, the pulsars, the gravitational lenses - that don't seem to fit into the models that you've grown up with ... Collect raw data and throw away the expected. What remains challenges your theories.

- Clifford Stoll, in The Cuckoo's Egg

The Cuckoo's Egg is an amazing book - it's a memoir, a techno-thriller, and a manifesto all in one. It recounts how, in the mid-1980s, an astronomer-turned-sysadmin at UC Berkley stumbled across a hacker in their system, and upon tracking down the intruder, ended up catching a ring of KGB agents in their first real stabs at cyber-espionage! It's a must-read for anyone who's ever going to write code, and just a darn good read for everyone else.

The quote above (from chapter 3) is a very insightful one because it tells us, in essence, that the really useful details are in the edge cases. Stoll's first hint that something was up was a $0.75 discrepancy in an accounting system. No one suspected foul-play, and in most scenarios, the bean-counters would just write this off and be done with it. But Stoll and his co-workers decided to dig, and found a rabbit hole so deep it literally did go all the way through to the other side of the world.

It's a principle that technology professionals need to keep in mind at all times. Yes, you will encounter that case eventually. The numbers will get that big. That reference will find some way to be null. There will be someone with 25 dependents. It's not enough to make sure it works for the most typical cases. Of course that's where to start coding, and where to focus the majority of the testing. But you can't ignore the weirdo edge cases - finding and understanding them can expose hidden problems, as well as opportunities. A lot of revenue can be lost by systems that are leaky around the edges, and a lot of unrealized revenue can be tapped by looking in the same places. Don't assume there aren't any outliers, because there always are. Find them, and either bring them in line, or exploit the opportunity they present you!

And the thing about outliers is that they can always pop back up. You think you've fixed that bug, only to see some more examples of it weeks after the release. You write off a 'harmless' variation each month, only to realize it added up to a pretty big loss by year's end. Again, always assume there are outliers, and put checks and balances in your systems to find them. Sometimes this mean making sure your test suite is robust enough; other times it means having an independent audit system.

This is especially important these days. I'm not suggesting that most weird events can be explained by a hacker, but you never know. Anyone who reads the news knows you can't be too careful. Don't assume your security is good enough! If something seems suspicious, figure it out! Outliers can just be bugs, but they can also be attacks.

Very few people are actually lazy, but we can all be lulled into a false sense of security. Just because the fire alarm isn't going off doesn't mean there aren't any fire hazards. We don't need to be paranoid, or rule-bound, but we do need to be vigilant and thorough. How one maintains such an attitude is probably the subject for a whole book and in the end is different for everyone, but it's a skill that technology professionals must develop.

WCF Data Member Order

One element of SOAP in WCF that is often overlooked is Data Member order. We need to stop and remind ourselves from time to time that SOAP is just XML, and that a data contract is not actually code, but rather an abstraction of XML (or JSON). The WCF Test Client can actually give us a good illustration of this. When you have a operation that you are going to call, the standard view is "Formatted", like so:

But there is also an "XML" view, which gives you this:

This is what actually gets transmitted to the service; this is the 'real' message.

WCF can actually be very picky about the order of the elements in a SOAP message: if the message being sent and the service receiving it don't agree on the data member order, the message won't deserialize properly. Worse, it won't give you any errors, but your object will only be partially populated. Now, if you publish your service, the client generates the service reference, and no one ever changes either, then this isn't an problem. But services do change - even adding a new element where you can accept the default value can change message order expectation. And if the client is not using WCF on their end (e.g., because they're using Java, or are a legacy system), then agreement between client and service cannot be assumed.

The good news is you can control the order of elements in a message.

The code for the previous examples is this:

[DataContract]
public class DeleteClaimRequest
{
 [DataMember]
 public int ClaimId { get; set; }

 [DataMember]
 public string RequestorUserName { get; set; }
}

Right now, the data member order is merely alphabetical (note: not the order of declaration). But let's say we wanted to enforce that the RequestorUserName came first. We can do that by modifying the code like so:

[DataContract]
public class DeleteClaimRequest
{
 [DataMember(Order = 2)]
 public int ClaimId { get; set; }

 [DataMember(Order = 1)]
 public string RequestorUserName { get; set; }
}

The message would then be formatted like so:

<s:Envelope 
   xmlns:a="http://www.w3.org/2005/08/addressing" 
   xmlns:s="http://www.w3.org/2003/05/soap-envelope">
  <s:Header>
    <a:Action s:mustUnderstand="1">
       http://company.com/service/2/IDataReceiver/DeleteClaim
    </a:Action>
  </s:Header>
  <s:Body>
    <DeleteClaim xmlns="http://company.com/service/2">
      <message xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <RequestorUserName>john</RequestorUserName>
        <ClaimId>123</ClaimId>
      </message>
    </DeleteClaim>
  </s:Body>
</s:Envelope>

Easy, right? Yes, but only because there's no inheritance involved. Consider this contract:

[DataContract]
public class SetCredentialsRequest : ManagementRequest
{
 [DataMember(Order = 1)]
 public string ProviderUserID { get; set; }

 [DataMember(Order = 2)]
 public string ProviderPassword { get; set; }
}
 
[DataContract]
public class ManagementRequest
{
 [DataMember(Order = 3)]
 public string AssociateUsername { get; set; }
}

The result is not what you expect. WCF will actually want the message to be in this format:

<s:Envelope 
    xmlns:a="http://www.w3.org/2005/08/addressing" 
    xmlns:s="http://www.w3.org/2003/05/soap-envelope">
  <s:Header>
    <a:Action s:mustUnderstand="1">
       http://company.com/service/2/IServiceManager/SetProviderCredentials
    </a:Action>
    <a:MessageID>urn:uuid:3c9a3661-0c33-461e-adca-08076696742d</a:MessageID>
    <a:ReplyTo>
      <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
  </s:Header>
  <s:Body>
    <SetProviderCredentials xmlns="http://company.com/service/2">
      <request xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <AssociateUsername>john</AssociateUsername>
        <ProviderUserID>user123</ProviderUserID>
        <ProviderPassword>41aer45b1</ProviderPassword>
      </request>
    </SetProviderCredentials>
  </s:Body>
</s:Envelope>

This is because WCF always puts the elements from a base class first (see this MSDN article). No matter what value you put as Order for AssociateUsername, WCF will always expect it to be first. And just to make things to even more confusing, the "Formatted" view of the WCF Test Client will show the message like so:

The solution? Use interfaces!

If you change the code like so:

[DataContract]
public class SetCredentialsRequest : IManagementRequest
{
 [DataMember(Order = 1)]
 public string ProviderUserID { get; set; }

 [DataMember(Order = 2)]
 public string ProviderPassword { get; set; }

 [DataMember(Order = 3)]
 public string AssociateUsername { get; set; }
}

public interface IManagementRequest
{
 string AssociateUsername { get; set; }
}

then it will produce the desired result:

<s:Envelope 
     xmlns:a="http://www.w3.org/2005/08/addressing" 
     xmlns:s="http://www.w3.org/2003/05/soap-envelope">
  <s:Header>
    <a:Action s:mustUnderstand="1">
       http://company.com/service/2/IServiceManager/SetProviderCredentials
    </a:Action>
    <a:MessageID>urn:uuid:67ddfa83-88a7-407e-a4b9-12c314a05905</a:MessageID>
    <a:ReplyTo>
      <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
  </s:Header>
  <s:Body>
    <SetProviderCredentials xmlns="http://company.com/service/2">
      <request xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <ProviderUserID>user123</ProviderUserID>
        <ProviderPassword>41aer45b1</ProviderPassword>
        <AssociateUsername>john</AssociateUsername>
      </request>
    </SetProviderCredentials>
  </s:Body>
</s:Envelope>

C# doesn't care what order the members of an interface are placed on the inheriting object, and WCF serialization doesn't deal with a base interface the way it deals with a base class, so you are free to specify the order explicitly and definitely. This way you're able to achieve type unity/inheritance while still being able to specify the necessary order. And because the order is specified in the derived object instead of by the interface, AssociateUsername could have a different order in every class that inherits from IManagementRequest. (Generated service reference code is smaller and less complex as well.) As an added bonus, interfaces generally make for better inheritance chains, as a class can implement many interfaces but only one base class. (Base classes have their place for separation of concerns in code, but again data contracts are not code!)

So, long story short:

• Always explicitly specify data member order to avoid serialization problems.
• Use interfaces when creating data contract inheritance chains.
• When you change a contract, be mindful of how member order changes may affect clients/impact message serialization.

Software Development Roles

In software development, there are four major stakeholders, or roles to play: the customer, the architect, the engineer, and the operator.

• The Customer: The customer is most typically the business analyst, executive, or client that is driving the production of software. They are the people who write the checks.
• The Architect: The architect is the person or persons responsible for designing how the system(s) work. They must take a holistic view and are responsible for guiding software to a place that is reliable, testable, efficient, and de-coupled.
• The Engineer: The engineers are those who actually build the software, who sit down and write code based on the customer's needs and the architectural designs.
• The Operator: Operators come in two very different classes: end-users and operations groups. If you release software for public consumption, your operator is the end-user. If however your software runs on company servers and/or workstations, then the systems personnel are the operators.

The ideal scenario, the sweet-spot where quality, sustainable software is developed, is where each of these different roles work in concert with all the others. The problem of course is that they often don't. The customer is impatient or bombastic, demanding features and deadlines that turn the architects and developers into slaves to his whims, too harried to do their jobs correctly. The architect is overbearing and tyrannical, over-planning the system to the point that it can never be completed and insisting all development pass through him. The engineers are slap-dash, churning out code without regard to system performance, readability, or bug count. The operators are too cheap, unwilling to provide enough servers to handle the load gracefully.

Hopefully no one has ever been in a position where all four of these sentences were true (or at least didn't have to work there long), but everyone's been involved in a job or project where at least one of them was. When one group wields too much say over the software development cycle, problems ensue. Each role should have autonomy in their own domain - their working environment should not be dictated to them by another role. However, for each role to enjoy such independence, there has to be a certain amount of give and take between them. For instance, if your engineers want to develop ASP.NET websites, the operators can't very well insist on Apache servers. If your operators are largely iMac owners, it would be unrealistic for the engineers to decide to write C# desktop apps. If the architect thinks it would be swell to use SharePoint, it's not his/her place to demand that the customer abandon the current CMS system they've come to know and love. Deciding how a piece of software (or software system) comes together requires negotiation and collaboration between the different groups.

One mistake commonly made is to assume that these roles have to be separate people. Naturally, in small companies, particularly start-ups, a small group of IT people will wear multiple hats. Freelancers, consultants, and one-man-IT-shops will often wear the architect, developer, and operator hats simultaneously and exclusively. There is a tendency, though, as the company grows, for these roles to become different departments. There's not necessarily anything wrong with that. But how a company sets up its reporting relationships should not determine how software development roles are filled. Software quality is improved and innovation is fostered when individuals who are engineers 90% of the time are allowed to be architects when the time is right. Architects who step into developer shoes produce more realistic systems. Operators who can be the customer sharpen system requirements. Fostering an environment where this kind of 'cross-pollination' is looked on favorably should certainly be a priority for all involved parties.

I once worked in the Information Systems Division of a Fortune 100 company. They had a lot of clumsy processes and a lot of the folks who worked there weren't standouts in their fields. But their environment was set up in such a way that each role had the autonomy they needed. An architectural group had created an overall vision for how the various systems should fit together, and the development groups were expected to follow that vision. However, inside their own domains, development teams had enormous flexibility in choosing the programming language, OS platform, and design patterns to use. The operators, the infrastructure teams that maintained the servers and terminals that ran this software, had a finite list of platforms and runtimes they would allow, but it was a long list, and anything on that list they would fully support.

My current job employs a much higher caliber of person, and has a much better development life cycle. They have, however, struggled with finding this correct balance between the stakeholders. Despite the flaws of that previous position, I find myself looking back at how they did things as a guide in this area. On paper, this sounds like a rather abstract and theoretical discussion, and in some IT shops it might be. But when this balance is off and/or these roles are not clearly defined, your workday can quickly transform into a series of turf wars. If you find yourself getting into that kind of situation, it's time to take a step back, define, and balance. We didn't, and it went badly: we thought we'd 'won' the turf war, only to have the problem come back at us sideways and make things worse. Don't let this happen to you!

Use this data, not that

The Salt Lake Tribune has recently been running a series of articles dealing with some privacy and security concerns that a fraud probe into a Utah prenatal health care program raised. The articles are primarily immigration-themed but they are also eye-opening from a software design perspective. One of the articles focuses on the fact that the software Utah uses required them to enter a Social Security Number (SSN) for patient identification. Some of the women coming into the clinics were unable or unwilling to provide this information, so the clinic staff would issue them 'dummy' SSNs to get them into the system. This eventually caused an issue because one of the dummy numbers entered happened to match the real SSN of a man in Maine. The end result was a case of accidental identity theft.

Anyone who's ever developed software will be un-surprised by the details the article gives about how the data ended up so muddled. The system required a nine-digit ID, so the staff used SSNs. When the SSN was unavailable, they'd make one up. For years they'd prepend a "V" or something to try and distinguish the reals from the fakes, but then an upgrade forced the values to become numeric only. Under both schemas ID duplication was occurring, a fact the staff was well aware of. Changing the ID field's parameters was too expensive, so they just lived with the mess. Investigations by the U.S. Social Security Administration (SSA) only prompted the helpful advice to use a different numerical prefix that the SSA doesn't use in SSNs. The state's processes have been modified to continue doing exactly what they've been doing all along, except now they have to keep a separate (most likely paper) log to be used to sort out any difficulties.

There are some very important lessons about software development that can be learned here: first, that using government-issued numbers as IDs is a very bad practice, and second, that good software design cannot ignore the human element.

SSNs are not used as IDs as much in software anymore, but I think some designers and developers don't really understand why this is the case. We may say "People don't want to give us that information" or "We don't want to be responsible for keeping that data private." While it's good to recognize the inherent privacy concerns, these reasons miss the point, plus most organizations that would use SSN in the first place do have valid reasons to collect it. The real reason SSNs make poor IDs because they cannot be changed, and because they are intended to be a private key.

An example to illustrate: I worked for an automotive shop where the mechanics would track the vehicle work via a touch-screen terminal. The mechanics would log in to said terminals using their SSN. The software running on the terminal communicated only with the server in the back room, and the shop had valid reasons to know the SSNs of the people their customers were entrusting their vehicular safety to. It all seemed like a reasonable setup. But then Employee B found out Employee A's SSN, and began to enter work under Employee A's login. I don't remember why firing Employee B was not an option, but it wasn't. We couldn't change Employee A's SSN without screwing up the payroll system, and we didn't have the resources to redo the terminal software (it was really, really bad code). This left Employee A entirely without recourse.

Using an SSN as a private key to eliminate duplication or provide positive identification for legal purposes is a perfectly valid thing to do. But to use SSN as a username or a public ID number is just wrong. It boxes you into logistical and ethical corners that can be very expensive to get out of.

The complaint is raised that we don't want our users to have to be responsible for yet another number or ID that they have to remember. This is a valid concern, and it has a simple solution: don't do it. Look the patient up by name when they come in the clinic. Issue them a card with the ID number printed on it (and include a barcode or magnetic stripe so it can just be scanned). Issue them an ID badge with an RFID chip. Let them choose a username - these are intended to be public, so people can reuse them ad infinitum. Require SSN as a element of account creation if you must, but store it privately (and securely) and map to it by the public ID of your/their choosing.

David Platt once said, "Your user is not you," and I don't think truer words have ever been spoken. Developers tend to have a certain mental block about this; they assume that because the field says "SSN" or "Email" or "Date of Birth" then that's what the user will enter. But we forget that to a user, a field is not a discreet, re-usable piece of information - it is a post-it note where they can write stuff till they need it again. Users will put information wherever they can fit it, regardless of categorization. A balance has to be struck between making forms daunting or too permissive. Validation goes a long way to helping with this. I work for a company that receives real-time (multiple per second) data feeds from the largest retail chain on the planet. One element in these feeds is email address. We get the data just as the store associate enters it, and since the software on their side does not require any validation at all - not even a check to be sure it includes a "@"! - the email addresses are not viable. A trivial regex would allow this information, which is invaluable for our marketing purposes, to be usable instead of dross. Validation is no magic bullet though - the most rigid validation in the world won't alert you to the fact that the patient's birth-date is not 1/1/1970. Unless for some exceptional reason you can verify the person's birth certificate, there's pretty much no way to independently verify that kind of data, and it would not be worth the effort for you to try. So in that instance the software should simply be aware that this value is not ironclad and may need to be treated with kid gloves.

The bottom line is that as computers and software become more and more ubiquitous, we have to avoid creating any further pitfalls like this.