Testing is a developer job

lack-of-testingMaybe you have heard about the various discussions about Test-Driven Development (TDD). Is it worth it? Does it lead to good design? … In this blog post I will not speak about this kind of practice, just “classical” tests.

When I started my software developer career, I knew nothing about automated testing (unit or not). I wish I did, it would have save me a lot of time and a lot of trouble back then.

The pain of legacy code

I started my life as a software developer in a small company, I had no experience and I was alone on the project, which was several years old. I had to deal with a “big ball of mud” where I was afraid of touching anything because I did not knew anything about the consequences it might have.

Yet, I had to fix bugs and to implement new functionalities in order to improve the application. Of course, I did not test much my changes, only that the bug is fixed or the new feature works as expected on my local machine. And every time it had unseen consequences because the code is highly coupled and changing one part of the source code change the behavior elsewhere.

I wish I had tests at that time to prevent me from working in fear, fear of breaking things, fear of regression. But I’m also guilty in this story because the number of tests I have added during this period is ZERO… My contribution was to make the whole thing worse by adding more legacy code.

I now realise that I was behaving un-professionally, legacy system is a real pain to work with and it is my job as a software developer to avoid creating this kind of mess. We have the tool and practices to make things better, we can add tests, we can refactor bad written code.

Whatever… QA will test it

Now I work for a larger company with several development teams, each one of them has a QA to validate the work done by the developers. I think that having QAs within the teams is a wonderful thing, they will check new feature and potential regression before a production release.

But sometimes I feel like that some developers see this situation as an excuse to be lazy. “I just code, I won’t test it, this is the job of the QA”. What?! Are you serious? Your code does not even compile! Sure the QA will test it and they will just say: “It doesn’t work”. They can’t even test a single feature because the entire system cannot be built.

This might look far-fetched but I’ve seen situations like this one, several times unfortunately.

Sometimes, the code “works” but what has been asked is not done, the code has been written and it compiles. Yet, when opening the page (example of a website), the new element is not present… It’s the developer job to open the site to make sure that it works as expected from end to end, at least locally. Again, I’ve seen it many times, with my own work as well.

In my opinion, QA should find nothing, if they do I have failed at some point. If the issue is technical, I made a fault and I need to fix it ASAP and learn from it. What do I’ve missed? How to prevent that from happening again? Is there a unit test I can write? If the problem is a business issue (not doing was it is supposed to do), then again: what did I missed? Is there an acceptance test that needs to be written? Did I know all the domain related details? If not, why?

Always learn from your mistakes and a feature not validated by QA is a mistake. QAs are not hired to piss developers off, they are paid to make sure the products are viable from a quality point of view. We are not paid to write code, we are paid to automate process, and make them work! QAs are here to help, not to do our job.

I’ve been down this road and this is why I make this blog post, to share my experience and my failures. I love my job as a software developer and I want to be proud of what I’m creating, I want to be considered as a real software professional. There are ways to improve how we work, from a technical point of view and an attitude point of view.

Testing is a developer job, unit testing, integration testing, manual testing, all of them. It’s our job to make sure everything works.

See you next time!


Image credits:

http://www.mobilisationlab.org/six-testing-ideas-for-your-next-email-campaign/

Are you a software Boy Scout ?

boy-scout

If you are a developer like me, you probably worked on a legacy code based application. If not, don’t worry (or do) it will certainly happen… We all know a project that we fear to open because it is just a huge mess without a single test and nobody really understand how it works, it just does. Every software team has to work in order to prevent the effects of technical debt.

Refactoring a whole application to make the code “cleaner” can take an enormous amount of time, it can be weeks or sometimes months, depending on the project size and complexity. Some organization will incorporate refactoring phases during the development life-cycle. But since during these periods, the team does not provide any new features (i.e. no value), having these refactoring phases is a hard sell. Time to become a Boy Scout then !

What the boys scouts have to do with software development ? The answer is found in their rule :

“Always leave the campground cleaner than you found it.”

This simple rule can be applied to code and especially legacy code, and it becomes :

“Leave the code better than you found it.”

I present you the Boy Scout Rule (BSR) of programming. Making the code “cleaner” can be done at any moment, and it can be done piece by piece, no need to wait for a “Big Bang Refactoring” phase.

If you have a legacy project it is likely that you will have some improvements or bug fixing to do in it. This is the perfect time to embrace the boy scout philosophy. Of course after your passage the application will still be legacy but a bit less, and the next time you will improve it again. The code will become better with time and one day you’ll stop considering the project as a legacy one.

For example it is possible to rename a variable with a more meaningful name. Given the following code to calculate a triangle area :

var res = b / 2 * h

After the BSR applied :

var triangleArea = base / 2 * height

This is not much but it will help the next developer (maybe you) to understand the code and its purpose. And next time you will see that this piece of code is duplicate in several parts of the code. Time to create a method then :

public int CalculateTriangleArea(int base, int height)
{
    return base / 2 * height;
}

You now have a method that can replace your duplicate code and that can be easily tested with your favorite automated testing framework ! I know that this example is really simple but I’m sure you’ll find these kinds of easy “cleaning” in your applications.

There is a type of code refactoring I often use to make my code more understandable and to ease maintainability : moving repeated magic number value into a single constant variable. For example, a few years back, the french VAT was equal to 19.6% and now it is 20%. I let you imagine the pain it could have been to change every “19.6” in some projects where it could have been far easier to use a single constant with a meaningful name.

There are a lot of refactoring techniques to improve your code base, Martin Fowler gives you a list of some of them here.

A software craftsman does not fear legacy code, by following the Boy Scout Rule he will improve his projects.

See you next time !

The Clean Coder : How it changed me

“The Clean Coder” is not a book about the code, it is about the coder. A software developer does not only write code for himself, he writes code to solve problems, to add value to his company. In this book Robert “Uncle Bob” Martin shares his experience on the mistakes he did and how he changed his behavior in order to act as a professional.

I really enjoyed “The Clean Coder” because it made me think of my own behavior. Do I behave as a professional ? What can I improve ? What should I stop doing ?

I discovered a definition of professionalism that I was not following, I learned to do “no harm”. I discovered what I implied when I said yes and that I should not be afraid to say no. I discovered how to stay focus while coding. I discovered the benefits of Test Driven Development (TDD), the benefits of Acceptance Testing and the benefits of having a good Testing Strategy. I discovered that practicing my skills is key to achieve mastery. I discovered how to manage my time in order to stay productive, how to avoid unnecessary pressure and making concrete estimations. I discovered that collaboration is key to build excellent software and that I have to work closely with my team to complete my projects. I discovered a whole new world of apprenticeship and mentoring : Software Craftsmanship.

This software development ideology suits me, it gave me a path to follow. I decided to sign the software craftsmanship manifesto to be committed and to ask more of myself. Being a craftsman is not an easy task, it is an attitude that has to be learnt. It is challenging and that’s fine to me. I like to be challenged, it allows me to improve.

I hope you liked this journey through “The Clean Coder”, see you next time !

The Clean Coder : Mentoring, Apprenticeship and Craftsmanship

Apprenticeship

Software development is a relatively new profession, new technologies and discoveries make it changes constantly. Thereby it has not been codify yet, there are several methodologies, principles, practices and patterns. And this is a good thing, we still have a lot to explore to master and contribute to our craft. Yet, this lack of codification allows us to do whatever we want and sometimes in a bad way. We’ve all seen teams that defined themself as “agile” because they do not have any single methodology and use the term “agile” as an excuse for chaos ! We’ve all seen a two or more years project that does not have a single test ! We’ve all seen programs that become unmaintainable after 6 months of coding ! We’ve all seen projects that have more bugs than lines of code ! Somehow this is the cost of our new non-codify profession. Fortunately there are a lot of well designed, covered with tests, maintainable software too. It is up to us to share our knowledge and best practices with each other, it is up to us to codify our own craft.

Even if software development is new, we all had mentors to teach us programming. We learned through teachers, colleagues, books, videos, articles or even friends. Forty years ago when programming was starting all these resources were scarce or non-existent. Uncle Bob (“The Clean Coder” author) had to learn programming the hard way, without all the resources we can find one mouse’s click away. If we work with senior developers or any experienced professional we can ask them to share their knowledge to teach us how to behave as a professional. And we can of course share our own experience with the younger ones.

After graduation, a medical student is not thrown into an operating rooms to perform brain surgery or open heart surgery even if he as the theoretical knowledges to do it. The medical profession oversees education through intense mentoring. Medical students spend a lot of their education time working with professionals to sharpen their skills. It takes a decade and thousands hours of practice to become a professional doctor. Are we shocked by this approach? Of course not, we cannot even conceive it otherwise. Their work is highly important and we expect them to act as professionals.

In the software industry, things are “slightly” different from the medical system. It is no surprise to see “teams” formed with freshly graduated programmers that are asked to build software even critical ones. Of course creating programs is not as crucial as surgery, there is no life at stake. But bad software can lead to colossal monetary loss, Sony is one example of many. Graduating in Computer Science (CS) gives us enough skills to work in the domain but schools can’t teach us everything about programming. Software development is a complex world that evolves day by day and offers us an environment with constant learning. Creating a doctor-like system for our professions is essential to avoid making the same mistakes over and over.

Software apprenticeship can be a three steps journey : starting from apprentice and moving to journeyman before becoming a master.

Masters have more than 10 years of experience and have worked on different systems, technologies and programming languages. They are able to lead and coordinate several teams. They are responsible for the technical aspects of the projects.

Journeymen are trained and competent programmers, they are professionals. They learn to work as teams and to become team leaders. Their experience levels vary among them, there are former apprentices with little experience and there are burgeoning masters.

Apprentices are programmers that just begin their career. They are closely supervised by journeymen in order to improve their skills and knowledges, pair programming is heavily recommended to do so. They learn how to behave as professionals.

This system is similar of the guilds organization during the medieval era. In the real world this system seems to exist, graduates are supervised by young team-leads who are supervised by project-leads. But most of the time there is almost no technical supervision.

When we build software, we are crafting them, programmers are craftsmen. Craftsmanship is the mindset help by craftsmen, it contains values, disciplines, techniques, attitudes and answers. A craftsman is able to work quickly but without rushing, he also know when to say “no” and to meet the commitments. A craftsman is a professional !

If craftsmanship is your way of life keep in mind that you cannot force other programmers to become craftsmen, and convincing them is difficult. If you want them to become craftsmen you’ll have to show them how it is done and the benefits of it. Then maybe they will join the movement.

This was the final chapter of “The Clean Coder : A Code of Conduct for Professional Programmers” by Robert C. Martin. My next article will be a conclusion about the book, explaining what I’ve learned.

The Clean Coder : Teams and Projects

team

Your organisation might have several teams and several projects. It is important to know how to manage these 2 areas of your IT department in order to get things done.

In a “project first” environment it can happen that some people will work on several projects at the same time but with different teams. A developer working in this type of environment will have to learn the methodology used for project A and the one used by the team on project B which can be completely different from the first one. He will also have to learn to work in an effective way with the people of project A and the ones of the other project. Making these switches over and over can be annoying or even frustrating. At the end this will result in a loss of focus for the people working on distinct projects.

Forming a team (a real team and not a group of individuals) is a long process. It takes time for the members to know each other, it takes time for them to know each other strengths and weaknesses, it takes time to understand each other motivation. The members will start to form relationships at some point, the team begins to gel. However having a gelled team is worth the waiting, the members will enter into a new dimension, they can do miracles. They will anticipate each other, they will support each other and they demand the best from each other, they make things happen.

In software development a gelled team is not only composed of programmers, the testers and the business analysts are part of this team as well, all of them working with a project manager. They will all plan together, face issues together and solve problems together, they work as a team. The analysts develop the requirements and are able to write the acceptance tests through user stories. The testers will also write acceptance tests from another perspective, they focus on correctness providing failure scenarios where analysts focus on the happy path. The project manager tracks the progress of the team and makes sure that the team is heading the right way.

Once your organisation has a gelled team you do not want to break it, you want to keep it alive. When the team’s project is reaching its end, give them a new one. They already know how to work as a team and they can focus on this new project right away at full speed. These kind of teams can even work on several projects simultaneously once they know their velocity. This velocity is shared amongst the projects depending on their importance and can be reallocate during crisis for example. It is easier to switch from a project to another with the same team than switching from a team to another.

Building teams is way harder than building projects, this is why it is important to persist them and make them work on a project after one another. In some cases they can even work on more than one at the same time. It takes time to build these teams but once it’s done they exceed the expectations you had.

You can read more about the gelled team on these articles :

See you next time for Mentoring, Apprenticeship and Craftsmanship !

The Clean Coder : Collaboration

collaboration

Software are made by teams and collaboration is needed in order to be efficient and to produce quality.

Sometimes we choose to work in programming to deal with the predictable behavior of a machine and not with the messy relationships between humans. But at some point dealing with others is mandatory in order to achieve our work especially with Agile methodologies.

Except if you work for yourself you will have employers and you have to keep in mind that their business is your business. They pay you so their problems are also your problems. Professional programmers collaborate with the managers, business analysts, testers and other team members to deeply understand the business goals. If you do not understand it how can you provide the best solution to the business problems ?

In a development team programmers have to work with other programmers. Do not fall into the “my code, my precious” attitude, it is not helping anybody. Maybe you think that your code is perfect and everyone should do the same. Perfect code does not exist ! Some developers will prefer a strong design over performance and some will prefer the other way around. It is just a matter of perspective. When you build walls around your code and your scope, you are doing harm to the entire system. It can not work as a whole, how can it move forward if one leg refuses to move ? The code of your application is owned by the team, not by the individuals. So code as a team : share your design ideas or your performance ideas, practice pair programming in order to benefit of all its advantages. Write the code in collaboration.

An open-space setup does not make you work as a team, especially if the members are sitting in corners with their backs to each other. Configure your work environment to face each other, this allows you to communicate with efficiency without making you roll your seat all over the open-space. You will also see when one of your teammate is struggling allowing you to offer your help.

Professional developers work with people whether they are from the business teams or their own team. They collaborate to answer the company’s needs and to produce great software.

See you next time of Teams and Projects !

The Clean Coder : Pressure

pressureWhen a surgeon is operating under pressure do you want him to stay calm and focused or to rush ? Professional developers are calm and decisive under pressure. They have best practices and principles to avoid making mistakes, they are made to be followed during pressure moments.

If you don’t like working under pressure, the first thing to do is to avoid the situations that cause it. As seen in the previous chapter (“Estimation”) do not force you into a stress situation by committing yourself to an unrealistic deadline you have to meet.

It is also important to “stay clean”, do not succumb to the temptation of the “quick & dirty” path. As it says it is dirty and you know you’ll have to deal with the consequences at some point. Furthermore it is not always that quick because the “quick & dirty” approach creates technical debt on your system and at some point your application is just a swamp and you are doing harm to your structure.

Stressful moments are not always avoidable, there are times when you will have to work under pressure. It these situations it is important not to panic, not to rush and stay focused. Rushing will in almost every case drive you deeper into the hole. Instead you should slow down and think of the problem you are facing in order to find the best path toward the solution.

Do not hesitate to communicate to your team that you are in trouble, look for input and guidance, avoid creating surprises for your teamates. You can practice pair programming to help you with your problem. Your partner will see the mistakes you are making, will have ideas you did not think of, will keep you from panicking and more. You are stronger when working as a team so do not hesitate to be the partner when someone else is struggling.

Relying on you disciplines is also important, you have them to give you guidance and they are the most useful during pressure moments. Do not abandon them at any cost, be even more dedicated toward them as usual. If you to TDD, write more tests. If you are a refactorer, do more refactoring. If you write small methods, make them even smaller. You practice these because you believe in them and you know it works, rely on your disciplines !

The best way to deal with pressure is to avoid it but when it is not possible you must remain calm and focused, communicate with others and trust your disciplines.

See you next time for “Collaboration” !

The Clean Coder : Estimation

target-estimation

Developers have to estimate the amount of time required to complete their features. This is at the same time a simple and a frightening task because the business depends on them. There is no ultimate way to make correct estimation every time but it is possible to aim for a good evaluation.

But first, what is an estimate in the software development environment ?

In some cases it is a commitment, you must achieve it ! You have to be absolutely certain (100%) to commit yourself into a deadline. Other people will make plan based on your commitments so you definitely don’t want to miss them. Otherwise you will look as a dishonest person and you are doing harm (see Professionalism chapter).

In other cases an estimate is… an estimate ! It is a guess, you do not promise anything and you do not commit yourself. If you already had to quantify the time a development task will take you certainly know how difficult it is. Most of the time there are too many variables in play to make a proper estimation. And most of all an estimate is not a number, it is a probability distribution ! The concept is explains with the following figure (fig. 1).

Probability distribution example
Probability distribution example (fig. 1)

In this example the task has 50% chance to take 3 days but it can also take 4-5 days or even 10 days (the Murphy’s law is never far away). With a distribution the business has data to rely on for their work and the development team does not have an inconsistent deadline. It is also important not to commit yourself without willing to do, especially if you use the verb “try” (see chapter 2).

Program Evaluation and Review Technique (PERT) offers an effective way to convert estimates into probability distributions suitable for the business team and for the managers. When estimating a task 3 numbers must be provided, it is a trivariate analysis :

O : Optimistic Estimate. Everything goes as planned, nothing as come into the way. In the chart example, O = 2.

N : Nominal Estimate. The value with the highest probability, in our example, N = 3.

P : Pessimistic Estimate. Everything goes wrong, very low percentage of chance to append. P = 10.

With these 3 values it’s possible to describe the probability distribution of the task with the following formula :

µ = (O + 4N + P) / 6

µ (mu) gives us the expected duration of the task. For the example this value is equal to 4 days. But this is just an average based on some coefficients and cannot be used as a deadline, it is a hint. Moreover the pessimistic estimate high value increase this average.

σ = (P – O) / 6

In this second formula, σ (sigma) represents the standard deviation of the task. It is a measure to specify how uncertain the task is, it is equal to 1.33 days with our sample data. The higher this value is the higher the risk is.

With the PERT technique the estimate for the task is the combination of the two values : 4 | 1.33. You cannot take one without the other.

Agile methodologies have given us several techniques to estimate tasks within a small development team. You can try a method called Wideband Delphi for example or the Planning poker approach or even the Affinity Estimating technique. All of them are based on agreement to provide meaningful estimates and you can combine them with PERT. To ease the process of estimating it is advisable to transform larger task into several atomic tasks, this way the team can be more accurate.

Professionals developers provide practical estimates, do not make promise they can’t keep and commitments that can’t be meet. An estimate is a probability and not a deadline !

See you next time for Pressure !

The Clean Coder : Time Management

time

Programmers should spend most of their time writing code for their projects, this is their job. But on a 8 hours work day how much time do you actually spend doing this ? In some organization meetings can take a huge amount of your time and you also have to control the amount of focus you have for the day.

Unless you work alone you certainly have to deal with meetings within your company, especially if your team uses an Agile approach for software development. But be careful not to spend your days in meetings, it can kill your daily productivity.

Attending meetings is important in order to follow the life cycle of your project but you are not required in every one of them. In this case you can politely decline the invitation if your presence is not mandatory. It’s no better to be present and play with your smart-phone because you’re bored or because you’re not involved.

There are also some cases where you can leave a meeting. I know it might look rude to do so but it can happen that a meeting goes not as planned and take much more time that you have anticipated. In a situation like this you can politely ask if your presence is still needed and negotiate your exit. There is nothing worst than a meeting without an agenda and/or without a goal, there is no better way to waste time and energy.

If you use an Agile methodology such as Scrum at work you certainly have to do stand-up meetings every day (mostly at the beginning of the day). Each member of the team should answer the 3 following questions :

  1. What did I do yesterday ?
  2. What am I going to do today ?
  3. What’s in my way ?

And no more, each person should be able to answer these questions in less than one minute. With this short and simple meeting you can easily know if your project is on track or no.

During an iteration planning meeting a development team select and reject backlog items for the new sprint/iteration. The estimates (our next chapter) should be done for every candidate item and if possible some of the acceptance tests. The chosen tasks should be clear and ready for the development phase (coding) and this meeting aims to allow the team to briefly discuss over the items.

Iteration retrospective meeting are designed to share what went wrong and what went right during the last iteration and to present demos to the clients/business. It should not extend 45 minutes, 20 for the retrospective and 25 for the demos which are prepared in advance.

Uncle Bob defines two truths about meetings, finding the correct mix between them for your team can be challenging :

  1. Meetings are necessary.
  2. Meetings are huge time wasters.

 

Writing code is an intellectual exercise that requires long periods of concentration and can be exhausting for your mind. But your focus is not infinite and can be depleted, if you are familiar with Role Playing Game (RPG) see this as an empty mana pool. Unfortunately unlike in RPGs you cannot drink a potion to recharge your concentration in a blink but you can refill it.

Sleeping is the best way to replenish your concentration, a good night of sleep (7~8 hours) can give you enough concentration for an entire day. Coffee is the developer’s best friend and can definitely help you regain a small amount of concentration for a short amount of time but don’t let this beverage send your focus in the wrong direction.

It’s also possible to partially recharge your mana batteries by taking breaks during your day, it allows you to de-focus. If the weather permits it you can go out for a walk, have a conversation with a friends, even meditate if you want. You can also practice a physical discipline, it also demands concentration but not intellectual focus : muscle focus. This type of focus can help you increase your mental focus and give you mana. Programming is a creative discipline then exposing yourself to other people’s creativity (books, comics, movies, etc…) is also helpful to boost your own creativity.

 

When producing code you will sometimes encounter “blind alleys”. It means that the path you’ve taken leads nowhere, in other word your algorithm does not what you want, your solution does not answer your need. It’s impossible to avoid every “blind alleys” but it’s important to realize when you are in one of them to back out.

What you definitely want to avoid are software “marshes”, “bogs” or “swamps”. Unlike “blind alleys” they don’t stop you, there is always a way forward that looks shorter than the way back but that is not. Sticking to a bad software design is a typical example of a swamp, the more you advance the harder it is to advance and at the end you end up with a colossal Technical Debt without noticing it. It kills a team’s productivity and can sometimes kills and entire project/company because maintenance has become overwhelming. If you discover that you are in a situation like this you should definitely turn back before it’s too late.

 

Clean Coders manage their time and their focus to keep their productivity up and running. They also know how to detect traps and know when it’s time to go back.

See you next time for Estimation !

The Clean Coder : Testing Strategies

Test Pyramid
The test automation pyramid

Professional developers test their application but not only with some unit tests and a few acceptance tests. These are components of a testing strategy.

Your company might have a Quality Assurance (QA) department or quality analysts (QAs) in each development team to track bugs before a release. As programmer our role is to let them find nothing, otherwise it means that in a way we failed somewhere during the development phase. Of course this goal is almost impossible to achieve every time but that is not an excuse for letting bugs slip through the code.

Am I saying that the developers and the QAs relationship is adversarial ? Absolutely not, they are parts of the same team. First, QAs are “specifiers”. It means that they are able to translate business requirements into test cases to let the programmers know how the system should behave (more in chapter Acceptance Testing). Secondly QAs are “characterizers”. They have the discipline to track incorrect behavior and to reproduce them in order to give proper feedback to development team and to the business.

Having a full test automation policy is a feature of professional development teams which is not only composed by unit tests and acceptance tests. The test automation pyramid figure show every test types and their proportion.

Unit tests are written by the programmers for the programmers to ensure that the code is working at the deepest/lowest level. They should execute in milliseconds and target a 100% code coverage (at least 90%).

Component tests are a part of the acceptance tests and check the behavior of individual component. A component encapsulate a specific set of business rules. These kind of tests should be very quick as well because they are decoupled from the other components and should cover about half the system.

Integration tests are required to check the communication between components in order to verify that the “plumbing” has been done correctly. They ensure that the architectural structure of the system is correct. About 20% of the system is covered by integration tests.

System tests are executed at the highest level of the system, from the UI to check the whole application and its construction (load tests are in this category for instance). They check about 10% of the system.

Manual/exploratory tests are done by humans to explore the application for unexpected behaviors. They need the human creativity to hunt possible hidden bugs.

An effective testing strategy is required to develop reliable applications. This comes from the relationship between QAs and programmers and from a solid automated test suite.

See you next time for Time Management !