Troubleshooting best practices for DevOps teams – strategies to recover quickly from downtime

It’s Saturday night – and your system is down, customers can’t access the application any more. And your key developers are not in reach. Sounds like a quite uncomfortable situation. Read here what you can do to prepare for such events – and recover as quickly as possible from outages and downtime events.

The usual steps to remedy problems are clear: understand the issue, fix the root cause. Sounds very straightforward. However, what if the person on call is not the experienced developer and doesn’t know right away what to do? In DevOps teams with shared responsibility and distributed 24/7 support that may happen sooner or later. To be on the safe side you need an approach that enables the people on call to remedy the most common problems without deep expert knowledge. How to prepare for that? Here are some best practices.

(1) Know the usual suspects

Chances are that this is not the first time that the service failed. That may be due to some known but not yet fixed problem, or due to some dependency to an service outside of the team’s control. Such potential “known” issues should be documented prominently, along with step-by-step instructions on how to get up and running again. Ideally this should be part of the ‘troubleshooting’ section of your runbook (see below).

(2) Provide quick diagnostics via monitoring boards

A good monitoring board is the starting point for efficient troubleshooting. Each service should have its ‘health’ board where the status of each major component is displayed, e.g. via green/red status panels. Make sure the overall situation can be perceived at a glance. Where finer grained information is available make it accessible as drill down from these panels. An example: for service latency you can use time series plots over the last few hours. For such display it may be helpful to display horizontal lines within the chart to indicate the ‘normal’ range of the value.

The board should also show the status for each required external service. This will immediately indicate if the own service or some external dependency – like e.g. a database – is the cause of the problem.

Building good monitoring boards takes time and effort. For each component you need to come up with a reliable status test. However the work will pay back sooner or later. Running production systems without such monitors is like driving a car at night without the lights on.

Grafana is a widely used open source tool for building such boards. There are also lots of other tools, including commercial systems that automatically take care of code instrumentation for health and latency measurement.

(3) Set up symptom based fix procedures

This is the most underrated approach to speed up system recovery. It will take some time and effort to prepare but will most likely provide good learnings for the team and put you in a much better position if problems occur. How does it work?

As engineers we are used to start reasoning about system behaviour from the viewpoint of individual components:

“if the database index is corrupt => service xyz will have high latency“

However in an outage situation such information is not very helpful, especially for non-experts. The guys will not see the database index problem – they will see the high service latency. And they want to know what to do about it. So lets analyse the system and set up instructions that start from exactly such observable symptoms. This is how it may look:

“high latency of service xyz may be caused by an overloaded database“

Imagine your 24/7 support had a complete list of possible system problems (symptoms) – and for each of them a corresponding fix procedure. Troubleshooting would be a lot easier. Of course there may be more than one potential root cause. Or additional checks may be required to find out which of the possible causes is the culprit. Here’s the approach to do this analysis in a systematic way. For best outcomes do it with the entire team:

Phase 1: Problem brainstorming

Brainstorm possible system problems and failure symptoms
Ask yourself: what can go wrong and how would that become visible in system behaviour?
Try to make this list as exhaustive as possible

Phase 2: Assign root causes and verification checks

For each symptom list the possible root causes
If required or useful: add instructions to verify or exclude the suspected root cause – these are the verification checks

Phase 3: Write down fix procedures

For each root cause write down the required steps to bring the system back up to normal operation
If possible include verification instructions – how would you check that the procedure solved the problem?

Congratulation: you just created a troubleshooting guideline : )

Do this exercise with the team, and repeat it every few weeks or months t o make it more complete over time – and to adapt it to modified system behaviour or new features. The troubleshooting guideline is also an essential part of the fourth best practice:

(4) Keep a runbook

Set up and maintain a runbook for each of your applications and services. The runbook contains the basic operational data for the service:

Name and short description
SLA (service level agreement or target)
Involved team members
Involved artefacts and libraries – and corresponding links to the repositories
Consumed external services (dependencies)
Build and deployment approach
Smoke tests (how would you quickly verify that the service is up and running?)
Monitoring KPI’s and strategies
Troubleshooting guideline (see above)
…and everything else that may be helpful

Keep the runbook up to date – and make sure it is easily accessible for whoever may need the related information.

And how about logging?

Logs are important, no doubt about that. However you should not rely only on logs to find out about your system’s health status. Set up monitoring boards for that purpose. And have your logs ready and easily accessible for verification checks – or situations where the approaches 1…3 did not help and you need to dive one level deeper.

January 16, 2019May 4, 2019

Fast software release cycles – how to avoid accidents at high speed

Why are fast release cycles so important for software development – and what strategies can help to avoid accidents although the team is producing at high speed.

Blurr - fast software releases — Photo by chuttersnap on Unsplash

Fast release cycles create customer value

The goal of every software development team should be to deliver new functionality to the users as soon as possible. Why? Finished software that sits in the shelf waiting for the next release is not usable. It is incomplete work, wasted effort and money. To add value you need to put that shiny new feature in the hands of the customer. Only then the new features make a difference in the real world. This means your software is only complete after the release and deployment. The entire process from development and testing to deployment needs to be optimized for speed.

Fast release cycles enable flexibility

Or think about a situation were your tests have discovered a security problem in your software. Now you need to be able to fix it quickly. Or you may need to adapt to a breaking change in some other consumed service that is not even in your own hands. Things happen, and in Cloud World you need to be flexible and able to adapt quickly. Once again – you need to be able to fix fast, but this only helps if you are also fast at testing and deployment. However nobody wants to be reckless. Jump and see how it goes? You want to be sure that your fix works.

Fast release cycles - but no reckless jump into the unknown — Photo by Victor Rodriguez on Unsplash

Why incremental changes are your friend

The good news is that you won’t change the entire product from one day to another. If planned accordingly the team can break down the work into small steps. Ideally these can be tested individually to get immediate feedback. It works? Great. There’s a new problem? Ok, we should know pretty well where it comes from since only a small number of changes occurred since the last good version. And the developers will have these changes fresh in their minds. Fixing the problem should be much easier compared to yesterday’s approach were many changes came to test a long timer after implementation -and all at the same time. So let’s assume the new small change is implemented and tested separately.

Incremental step wise changes help to shorten release cycles — Photo by Lindsay Henwood on Unsplash

The next and final step is to deploy this incremental change and we’re done? Sounds too good to be true and indeed… How can you assure that the small change didn’t cause any side effects and broke something within the existing overall system? This is called a regression.

The new bottleneck: regression testing

So you need to test for regressions. And this basically means that you need an overall test of the entire system, which often is a huge effort. If you want to be on the safe side you will have to repeat this exercise over and over again for each small incremental change. Now if such an overall tests would take days or weeks it kills the nice-and-small incremental approach. It would just be too slow and too expensive.

Software test lab — Photo by Ani Kolleshi on Unsplash

The only way out of this dilemma is…

Test automation – the enabler for high speed releases

Imagine a setup where you could prove with a click on a button that your software is doing what it is supposed to do. That today’s changes did not introduce any regression. Test automation aims at achieving this. Manually clicking through an application may still have its place within an overall testing concept. But in no way this should be your normal approach. Put the test procedures in code and execute them automatically. This is what enables quick feedback to code changes – and therefore fast release cycles. This automated approach has the added benefit of repeatability – and test status reports that your the test framework will create automatically if set up accordingly.

Does this mean that testers are not required any more? Not at all, rather the opposite is true. Test automation won’t save cost – this is about saving time and improving quality. Or in other words: about avoiding regressions although the team is going at high speed. This is where testers play a key role. However with test automation the tester’s focus and know-how profile changes completely. Yesterday testing meant manually executing test procedures over and over again. Today it means development and coding of tests. Testers are becoming developers – or developers take more and more responsibility for testing. Welcome once more to DevOps world. (more here).

Fast software release – mission accomplished?

So let’s assume you the team works with incremental changes. You have automation in place to quickly test the changes for functionality and regressions. We are good to go – now what needs to happen to put the new version to production – into the hands of the users? This will be covered in the next article about Deployment automation. Stay tuned.

January 2, 2019January 19, 2019

Cloud Native? What it means to develop software for the cloud

What is cloud native – and what does it mean to develop software for the cloud? Find out what makes software teams life very different in cloud world. What is so special about cloud software?

Is there anything special at all? If ‘cloud’ would only mean booking a virtual server and installing the application there – then the answer would be no. However this would only leverage a small part of the what-is-possible. Hidden within the term “cloud native” is the assumption that you want to realize on-demand scalability, flexible update capabilities, low administration overhead and be able to adapt infrastructure cost dynamically to performance demands – just to name a few. Sounds wonderful… however these things won’t come all by themselves. Your software needs to be designed, built and deployed according to…

The cloud-rules-of-the-game

Huh, what’s this? First of all don’t get me wrong: there is nothing bad about having a monolithic application residing on a single server or cluster – and from development perspective this may be the simplest way to realize it. However this monolith may get difficult to extend over time without side effects, it may be difficult to set it up for high availability and to scale it with growing number of users, data, or whatever may be the reason for your scaling needs. And it may be hard to update your monolith to a newer version without impacting the users during the update.

Now consider the same application based on what is called a microservice architecture: the monolith gets split up into smaller, decoupled services that interact with each other as providers and/or consumers of stable API’s.

What makes microservices well suited for cloud applications?

Let’s assume that each service may exist not only once but with multiple instances up and running simultaneously. And let’s assume the consuming services are fault tolerant and can handle situations where a provider service doesn’t respond to a call. Wouldn’t that be cool? The overall system would be robust, and it would be very well suited to run on cloud infrastructure.

Because now if service xyz is starting to become a bottleneck you can simply create more instances of that service to handle the extra load. Or even better, this would happen automatically according to rules that you have configured up-front. This approach is called “scaling out” (compared to the old-school “scale-up” approach where you would get a bigger server to handle more load).
Next up imagine that you need to update service xyz to a newer version. One way of doing this would be to create additional service instances with the new version and remove the old ones over time, an approach called “rolling update“.
Or you decide to add a new feature to your application. Since only 2 of your 5 services are impacted you will only need to update these 2. Less change is easier to handle and means less risk.

Getting microservice architecture right is not easy, but once you have it the advantages are huge. Note that microservices architecture as such has nothing to do with the cloud as such. However both go very well together because in a cloud environment you need to be prepared to micro outages of single services anyway. You don’t really control the hardware any more, at least you shouldn’t want to. In order to be cost efficient with a cloud approach each service should be enabled to run on commodity infrastructure. From that take and pay for as much as required.

Where will your services live? – serverless vs. containers

For hosting your workers or compute loads consider a serverless approach over a VM or container based one. This basically means that you only write the service code as such and determine when and how the logic will be triggered for execution. All the rest is handled by cloud infrastructure. At Amazons AWS this technology is called Lambda, Microsoft named it Azure Functions, Google calls it Cloud Functions. The principle is always the same. There’s no virtual machine any more, not even Docker or Kubernetes containers – means less things to manage and look after, means less operations and maintenance effort. And you only pay for the execution time. If nothing happens there’s no cost. If you suddenly require high compute performance your serverless approach will be scaled automatically – if you have done your architecture homework. Serverless will e.g. require that your services are stateless, means whatever information they need to keep between two executions of the service must be stored externally, e.g. in a cache or database service. As with microservices the advantages need to be earned. Serverless will not solve all problems, but make sure your team (at least the architect) understands the concepts and can make informed decisions about its use.

Other game changers in cloud world

What else has changed in a world where dedicated physical servers seem to have disappeared? Very trivial things need to be handled differently.

There’s no local hard drive any more. If your service runs in a VM or Docker container it may feel so, but remember: in cloud world machines are cattle. A VM/container may die or disappear and will be replaced by a new one. Bad luck if you had data on a local drive of that machine. Now you’ll need to think about alternative ways for storing away your data or settings. In cloud world you may want to use a storage service for that purpose, a central configuration service, a database, environment variables… the choice depends as always on the requirements. Make sure the team knows the available cloud building blocks.
If you are a Microsoft shop: there’s no registry any more. See above comments.
For logging there are no local files any more. These would anyway not make much sense in a world where services are distributed. You’ll rather send log output to a central logging service. That will consolidate all logs of the various services in one central place, making troubleshooting much easier. There are many open source solutions for logging, our you may just use the one your cloud provider provides.
And finally the term infrastructure will get a whole new meaning in cloud world. Infrastructure still exists, but now it needs to be managed very differently. You should strive to set it up automatically, based on scripts that you can re-run any number of times. This is crucial because you will need more than one cloud system. At least you should have one for development and testing which is separate from the real one – your production system. The two environments should be as identical as possible, otherwise your test results are not meaningful and you will chase after phantom problems that are just caused by some infrastructure misconfiguration. Those scripts will set up your required cloud resources. Means you describe the infrastructure in code just as your service logic. Infrastructure as Code (IaC) is the term for that. Check out this article for more.

So what is Cloud Native after all?

Hopefully it became clear that software needs quite a few specific considerations to feel comfortable in cloud world. Which is the reason why “lift and shift” for existing legacy software is a valid approach, but won’t leverage the full cloud potential. In order to run efficiently in the cloud, software must be designed for that purpose – this is the meaning of cloud native – software that is architected and optimized for a cloud environment. For legacy software that usually means: for efficient shifting to the cloud major refactoring or even a complete rewrite may be required.

So welcome to cloud world. Tremendous power and flexibility is at your disposal. However you’ll need to architect your software with the cloud and its building blocks in mind. Decompose your application into services. Consider serverless approaches to reduce operation effort and improve scalability and availability. Focus on your domain knowledge and your specific value-add. For the basics use existing building blocks wherever it makes sense.

Check out these related articles:

January 2, 2019January 19, 2019

Tool checklist for cloud development – set your team up for productivity

What tools are required for teams that are developing software for the cloud? Verify this checklist to find out if your team has the basics for productive software development in place.

Contents

Versioning system

You know that you
(1) have one, (2) it is set up correctly and (3) and your team is using it
… if you can answer the following questions with YES (in case you’re not sure if you need a versioning system read here)

Each team member is able to revert source code to a former version at any time.
You are 100% sure that this works because you have tried and tested it
You are able to quickly find out what has changed between versions
You are treating configuration files, documentation and any other artifacts like your source code – everything is under version control
Team members check in modified source code at a regular basis
Your team has agreed on a common branching strategy
If your versioning system would fail completely today there would be no panic. You would just set it up again from scratch and reload yesterdays backup. You know that this would work because you have tested it at least once.
If yesterdays backup was not created there is a notification in your team mailbox

Issue tracking

You know that you
(1) have one, (2) it is set up correctly and (3) and your team is using it
… if you can answer the following questions with YES (in case you’re not sure if you need an issue tracking system read here)

The team has a complete list of all currently known bugs and issues
The list is accessible to each team member and everybody is be able to work on issues (e.g. add comments)
Each team member can see “his/her” issues with one single click, and ideally get automatically notified about status changes for “his/her” issues
The team has a clear policy on issue status management: what issue status values exist and who is allowed to change issue states? An example: many teams follow the principle that an issue should be verified and closed by the person who initially opened it. Your team may want to handle that differently – just ensure that everybody agrees on the same standard.
Each issue in the list has at least:
- a clear title and a description that each team member can understand without need to ask whoever has written the issue
- a status and an owner within the team
- additional information required to tackle the issue (screenshots, steps to reproduce, logs etc.) and meta information that helps to manage the issue and understand its history (time created, component or service concerned, version, …)
Daily backups are created and stored on a separate system. The backup / restore procedure has been tested at least once. This is not required if you use the cloud service of some provider (see article)

Build and Deployment System

You know that you
(1) have one, (2) it is set up correctly and (3) and your team is using it
… if you can answer the following questions with YES (in case you’re not sure if you need a build / deployment system read here)

Each developer can create a new build with a single command or click
Build status is visualized and team members get notified about build problems
New builds can be deployed to the desired environment with a single command or click (at least for the productive environment most teams will set up rules regarding who and when this is is allowed)
You can always tell which version is installed on what environment
You have a track record of builds and deployments

Team Collaboration and Knowledge Base

You know that you have what you need if you can answer the following questions with YES (in case you’re not sure if you need tooling for team collaboration read here)

Each team member can access a common system and find the most relevant 5 documents via direct short link
Each team member can add or modify content
Each team member can search for information or documents via keywords

Check out these related articles:

January 2, 2019February 2, 2019

Tools for agile development? These are the must-haves for every dev team

What tools are essential for agile development? A code editor may be enough to get started. But what are the must-haves for every dev team?

(1) Software versioning

Never develop software without a versioning system. Why? sooner or later you will need to know what you changed since that last version that actually worked and did not have that strange bug you have now been tracking down for hours. Or you may need to fix a minor problem in the productive version but don’t want to roll out all your latest untested changes. Or you work in a team and several people contribute to a common product or service.

A version control system is a central code repository and provides what you need to combine work results in a controlled way. Don’t try to get along by copying your sources around in different folders or zip archives. There a many great tools for proper version management, pick one and use it. GitHub is one of the favorites. it even takes care of permanently and securely storing your sources. Left your notebook in the coffee shop and now it’s gone forever? Well bad luck, but at least you still have your source code available. And while we’re at it: put everything under version control, not only your source code. Version control your configuration files, your specs, your user documentation – whatever you and your team are producing. It’s just like a car insurance – sooner or later you will be grateful for having it.

(2) Issue tracking

Software comes with bugs, and you need a system to keep track of them. You’ll want to know what problems have been found in which version of your software, what has been fixed already. Team members will need to add comments, logs or screenshots to an incident. Ideally you may want to link the issue to the code in your central repository that fixes a the problem. And since you work in a team you want a system that supports simultaneous multi user access. Don’t waste your time trying to do all this with some spreadsheet software. Yes you could do that, but it won’t really work. There are many good incident management and bug tracking systems available. You may want to take a look at the issue tracking that comes along with GitHub, or check out Jira – these are probably the most popular ones.

(3) Build and deployment automation

Software is born on one or several developer machines, but this is (hopefully) not where it is going to be run for productive use. Various steps need to happen before the checked-in code arrives in production. Depending on your development environment, programming language, used components etc. the artifacts for the prod system need to be built and versioned. You may want to apply static code analysis. Deployment to a test environment comes next. If all tests have been passed you or product management or whoever is responsible for that may decide to put that shiny new version out for productive use by your customers or users.

Let’s note that most of these steps need to be automated in order to get to reproducible high quality results. The system behind this is often called the build pipeline because things happen sequentially in a predefined order. If any of the steps fails the pipeline stops – the team needs to fix the problem and restart the pipeline again. A basic build pipeline system is e.g. available as part of GitHub. Then there are many open source and commercial solutions out in the market, and each of the major cloud providers offers a solution. If you’re not sure what to use start with GitHub – you can always move to some other system later on. Just make sure that you have an automated build and deployment approach right from the beginning. Improve it over time.

(4) Team collaboration

This is a crucial topic and the perfect approach for team collaboration yet needs to be uncovered. And of course no tooling will ever replace person-to-person communication, willingness to engage, to share information, to work together. Check out this [] blog post to dive into this. Having said all that – collaboration is still an area where good tooling really can make a huge difference. What you want to have is a central place for all your team’s information sharing needs. Documents, drawings, ideas, team calendars, whatever. You will want to search for stuff, and create links between various pieces of information.

Of course you could just store stuff on some shared folder or Dropbox. However this will be far from good. Who will want to browse around in some obscure folder structure? Admit it – this does not even work for your own stuff on your notebook, am I right ? We are in the new millennium and Wiki’s have been invented – if you’re not using one already please start to do so now. Once you have used one you’ll wonder how a team could ever have survived without. It will of course only be accessible by your team, and each member can set up new pages and content with a few clicks. Documents become much more lightweight and can easily be interlinked. Updates are done in seconds and you’ll have one valid version per document only. Paradise. Who knows, maybe even your most no-documentation-required style people will start to add stuff to existing pages.

There are many hosted Wiki flavors out there, setting one up for your team is a matter of minutes. A good choice is Confluence (by Atlassian – no I’m not getting paid by them, I just like their software). If you happen to use their Jira for issue tracking you can take advantage of the integration between both. In my opinion a great choice and all you need to cover your team’s basic tooling needs.

(5) Is there point 5 ?

Well there’s many more tools out there for life-cycle management, test automation, code analysis, etc. etc. – but these may be somewhat more advanced topics and your needs will depend on the type of software you develop, on your programming languages, target system and so on. For now we wanted to stay with the indispensable basics. You can always add to your tool-set, just make sure you’re not missing out on versioning, build automation, issue tracking and a Wiki – the basic tools for efficient agile development.

Check out these related articles:

January 2, 2019February 2, 2019

What is DevOps? How development for the cloud changes a dev teams life

What DevOps means is quickly explained: Development + Operations together. ,But what does DevOps really mean for development teams and their day-to-day work? And what is ‘operations’ to begin with…?

“Operations” explained

What is operations, does all software need to be operated? To explain this let’s take your local Word and Excel, or whatever local software you have installed, as an example. It just sits on your notebook. Once in a while you’ll probably update it to a newer version but that’s it. It is your personal software on your own machine – no real operation involved.

Compare that to your email. Here again you may use some local client, or just the browser. No operations. But then there is your email provider and all that is required to manage your mail account and transmit your mails. This is done by services that run in some datacenter and you can count on a team of experts that look after that software. They’ll make sure that the system runs smoothly, they apply the latest security patches, protect it against hacking attacks and securely back up your data, to name just a few of the tasks. This is what an operations team does. And you need to rely on it because you are using services that are not under your own control. Means whatever software runs in a cloud or datacenter will need operations.

Software development vs. operation

Traditionally there has been a very clear separation between teams that develop software and the ones that operate it:

DevOps separation of development and operation — Photo by Raj Eiamworakul on Unsplash (stuff in red by Tom)

The developers would write their code and maybe even test it 🙂 but as soon as possible throw it over the wall to the operations team. Then it would be up to them to figure out how to install and run it. Okay, maybe there are companies with good collaboration between these groups, but still there may be some conflict of interest. Developers will want their new versions out on the productive system as soon and as frequently as possible to bring new features and bug fixes to their users. The Ops team will try to slow things down a bit and play it safe since every update is considered a risk and may required some system downtime.

Operation for cloud software

Now fast forward to cloud world with an agile team in Scrum mode. Software sitting idle waiting to be deployed is considered ‘waste’. The infrastructure does not consist of physical servers owned by administrators any more. Now infrastructure is code and the dev teams’ architectural design decisions have a huge impact on the required cloud building blocks and corresponding cost. Also the ongoing operations effort is to a large part determined by the architecture. System changes may require modification of the infrastructure code, too: adapted configuration of the used cloud provider services, extensions to the system monitoring etc. The entire separation between development and operations does not make sense any more. As they say: you can do it, but it’s not good.

DevOps to rescue

Instead let’s put everybody in one team. Ensure that operational concerns are considered during development and when designing the system architecture. The team should consider the overall lifecycle cost and minimize effort accordingly. This is what DevOps is supposed to mean. No wall any more, not even two distinct groups, hopefully.

development team — Photo by rawpixel on Unsplash

Check out these related articles:

January 2, 2019February 2, 2019

What is SCRUM – setting up highly productive development teams

What is the best team setup for efficient development? How the Scrum methodology and agile principles can help to increase software development team productivity.

The challenging way to version 1.0

Let’s assume that you have a good-enough-idea of what you want to build, at least for the first version of the new baby. How do you start off? Just let the team work – after all they are professionals and they should know… You can do that but it won’t work. Why? First of all, despite all the lengthy discussions you still don’t know exactly what you want to build. Most likely every member of your team has a different version of that clear goal. And the tricky thing is that you are not aware of these subtle but relevant differences. Between your idea and the first version of tangible software you’ll have to cross the land of realization. These are dangerous grounds where orks roam and you better not send out the team unprepared. As long as you are safe in Rivendell make a plan that everybody can follow even when walking alone for some time. Slice the mammoth. Break down the work in small pieces and write them down, each part at a time. Make sure all parts fit together and that the sum of the parts makes up what you have in mind for that next version of your software. Put the parts in an order, with most important ones first, and assign each part to one of your team members. Sounds a lot better than just heading out into the wilderness, doesn’t it?

No Master Plan required?

No. You won’t create that 200 page master plan for all that may need to be done within the next 3 years. Don’t waste your time, things will change and if we’re honest you don’t even know what’s going to happen 3 months down the road. But the next 2 weeks should be doable. That seems to be a time range that humans can overlook. 2 weeks are still soon enough to review the first achievements. And just long enough to give the team the time required to create results in the first place. Feedback and intermediate steps are what you are looking for when navigating the uncertain plains of software development. An incredibly high percentage of software projects fail and that should not happen to yours. Don’t be afraid but also don’t be careless. The orks are out there and your team needs a structured approach to succeed. Many methodologies exist and the best choice depends among other things on what you want to build. If you’re about to prepare the next mission to mars you may want to stick to rather old school waterfall style approaches. For all others: take a close look at

Scrum and agile principles

This may sound very abstract at first but is very straight forward once you have a basic understanding. Much of this about accepting that humans seem to have great difficulty to define upfront how exactly that final software should look, work and behave. Of course you can try to define it to the last detail before building it – but it won’t be good, at least not nearly as good as it may get with more degrees of freedom to change and adapt. Think about a cook inventing a new recipe. Will he do it all upfront at his desk? The basics, sure, but the difference between good and great happens during preparation in the kitchen. So with software. Learning usually happens over time and the teams growing understanding of the domain unlocks new possibilities. However if change hits too often chances are that nothing gets ever done. Some balance seems to be required and this is where Scrum comes in. It strikes a balance between not planning at all (no, not good) and overdoing the planning part. It allows for enough flexibility to factor in changes but provides the basic guide rails and structure to keep chaos at bay. And it provides some insight into work and team progress over time. Sounds great, doesn’t it? Let’s take a closer look. If you want the full story go to scrum.org. What you read here is a high level summary.

Epics, User Stories and the Backlog

Did we mention the 2 week time range and that a breakdown of the vision to smaller parts is required? In Scrum these small parts are called epics and user stories. A user story describes a functionality, a feature, some requirement from a specific stakeholders perspective. It should be a very tangible thing with clearly defined outcome, with just enough information so that the people in the team understand what is expected. The team will order the user stories by priority – they all go into a central list called the backlog. And this is what the team is working on within the 2 week period – which is by the way called a sprint. Before the sprint starts the team has agreed on the user stories to be tackled and their priority. The backlog for this sprint is then fixed and the team can focus on working on it undisturbed. No change of priorities, no sudden additional tasks. Let the team do its job. In time before the next sprint starts there’ll be another planning for this next sprint. Then the game starts over again.

Why 2 week sprint cycles?

This is some sort of balance between structure and flexibility: the current sprint is fix, but you can plan the next one as required, factoring in whatever changes that came up. So the backlog is your definition of the work to be done, and it will change over time. However the team will work on stable ground and produce results every 2 weeks. Note that there is no fixed deadline and no fixed overall goal any more – it just doesn’t make sense in this ever changing environment. Instead there is the backlog and each completed user story drives the team towards the next better version. Some forecast about what will be available when is still possible but with less binding guarantee. This approach may sound very challenging in an enterprise, customer, contract context, and it somehow is. By the way – how reliable was your binding guarantee in the past anyway?

Overcome lack of transparency

This is another major challenge in software development. Where does the team stand? How much time is still required to get that next version ready? When will it be done? If the answer is ‘one more week’ or ‘one more month’ you know you’re doomed. These answers mean in fact that we are optimistic and quite sure that we can get the job done – but have zero clue about the remaining effort. Just leave me/us alone and come back again in one week/month. Then by the way you’ll get the exact same answer again. Sounds familiar? This is not bad intention, this is how humans are structured. We tend to underestimate larger work packages. Back to Scrum and the user stories. A story should be small and have a clear, tangible outcome. Something you could see or ideally test once it is done. Like an updated version of that UI with the new feature xyz. Small means it should be feasible to complete it within 1…3 days.

Break things down into smaller parts

If this sounds impossible break your story down to several parts, think about a step wise approach. Sounds like a bit of extra work but the team will get many benefits in return. Each small part will either be not started at all, in progress or done. That’s it. Nothing in between. No it’s-almost-done-come-back-in-a-week any more. The team will see that progress is visible on a day by day basis. Check out this blog post [] for more details. Ah, how will the team see that by the way? The backlog will be visible to the team at any time. You could do it via post-it notes on a wall, one per user story, as very basic approach. Or probably much better via one of the many available Scrum tools with online access for the team and a virtual board for each sprint. You can find some recommendations in the tools article.

Streamline team communication

This is the next critical element. No lengthy status meetings any more, from now on there’s only short ‘stand-ups’ but these happen each and every day. The team meets standing up. Don’t get too comfortable, this should be a quick thing, just a few minutes. Everybody tells what his/her current topics are, any problems or help needed? Then up to the next one, no lengthy discussions. And over, back to work. This approach is much more efficient than weekly status meetings that nobody follows over the entire time anyway. For any detail clarifications people can meet one-one-one as required.

Learning and improvement

Good is never good enough. At the end of each sprint the team will change the perspective to the meta level and meet for a ‘retrospective’. The purpose of this meeting is to think about the team as such, reflect on what worked well within the sprint, what did not, what could be improved? The team should capture and follow up on proposals and ideas. Make team members address impediments together. Get stronger and more efficient over time. And by the way this is it. Stand ups, backlog grooming and planning, a retrospective. No other meetings. Leave the team alone, give them the time to do their work.

And how about roles?

Scrum articles often start with role descriptions as if this was the most important thing. I think the idea is the most important, the team working together, ever improving. Yes, somebody should own the definition of the software product to be build, collect feedback and bring it back into the team, in general act as interface for the team towards the outside world. The outside world will be first of all customers and users, but also most likely other internal stakeholders like upper management, sales, marketing, whatever – depending on company size. This is the Product Owner (PO). The team as such is responsible for contributing to and for delivering on the backlog, but the PO is the one who writes the user stories and prioritizes them. This puts him in the driver seat for steering the product and ensuring fits with the overall vision and market requirements. However a smart PO will always collect as much buy-in and feedback from his team as possible. And then somebody needs to look after the process as such. Invite to backlog planning, ensure stand-ups are short and remove impediments for the team as such. This is what the the ‘scrum master’ does. The team could determine one of the members, or rotate the assignment. For larger organisations there may be full-time scrum masters that looks after several teams. And that’s it.

Check out these related articles:

January 1, 2019March 17, 2019

Team productivity and cloud software development

This blog is for you if you want to set up your software development team for success. Produce steady, high quality outcomes based on best practice processes. Leverage the cloud to your advantage.

We’ll talk about team setup. How to organize work, track progress and avoid disasters. You’ll find some basic introductions, best practices, guidelines and checklists.

What I write about has been covered in books, articles, conferences. This blog condenses my personal views and learnings from what has worked and what hasn’t. You may look at it as sharing some practical experience. My views will not be true for everybody. Take whatever is helpful for you and leave the rest aside. This blog will not make you an expert in any of the topics covered – but it can provide an overview and guide you toward further reading.

This blog is for the software adventurers of our time. Join the ride.

Start on the path - towards team productivity, cloud software and other adventures of our time

Some recommendations to start reading:

Photo by Lawrence Walters on Unsplash