Category Archives: Agile

Agile, Project Management

Agile Cambridge 2019: piecing it together

Leave a comment

Last week was Agile Cambridge, the annual, three-day, conference on Agile at Churchill College in Cambridge. Despite the fact that my new job role doesn’t directly involve Agile any longer, I was speaking and I wanted to go for all three days anyway, because a lot of the sessions looked interesting.

For the last couple of years, I’ve been thinking of Agile Cambridge as getting a bit samey – the same topics, often by the same speakers – and I wasn’t getting a whole load of value from it. This year was different, however. The conference had a different feel and a fresh approach, which made it very useful once again. I summarised it after the event on Twitter, thus:

End of 3 days at the 10th #agilecam. Main takeaway: Agile is now mainstream. Few talks discussing process, how to release faster, how to get user feedback. Instead, lots of team formation, the need for diverse (inc. neurodiverse) people & safe communication. We’ve come a long way

https://twitter.com/dnas2/status/1180147651448315905

In this post, I wanted to cover some of the sessions that particularly resonated.

Agile Cambridge

It starts from the team

Previous Agile conferences have tended to lump together the team as a single, homogeneous unit. This time, the role of individuals within the team and their diversity was a key topic.

  • Zara Sheldrake from ARM gave a particularly thought-provoking presentation on the subject of Neurodiversity in Agile Teams, talking about some of the ways that Agile is both good for neuro-diverse individuals (the Prime Directive, the role of Scrum Master as a shield, and pairing) and terrible (the enormous PI planning events from SAFe, open plan offices, and the language of sprinting). She discussed some ways that ARM have tried to mitigate some of these factors, both through things like mental health awareness and making the environment better suit the team.
  • Through this work, which includes features like setting team norms and personal user guides, they are trying to create a psychologically safe environment for learning. Two other sessions explored how such safe environments have been adopted in other contexts to ensure lessons are effectively learnt in life-and-death situations. In One Super Dog and her servant leader, Darren Yates from Lowland Rescue talked about how techniques for search and rescue scenarios have evolved over the last 20 years. Similarly, Jeff Foster from Redgate explored Why planes don’t crash, again about how the aviation industry goes to very substantial lengths to ensure they fully understand root causes for any incidents, so that flying is made safer. It really made some of our retrospectives seem quite superficial.

It’s always communication…

  • Of course, all of this ultimately comes down to the most difficult problem in software development: communication. Nick Maidment, now at ARM, previously Metail, explored some of the challenges Metail had faced and overcome as they moved from a mainly onsite team to a mainly remote one. As you might expect, neither extreme proved particularly problematic, but The Space Inbetween, where only a couple people are remote, is where the difficulties lie.
  • One of the few talks to directly relate communication to processes was Your bugs aren’t special by Emily Tate. The challenge is the perennial one that ‘little things’ often get done on the sides of the prioritised work. In one case cited, the ‘little things’ were taking up to 30% of the iteration time. The thing I found most interesting was framing this as a communication breakdown, that different stakeholders aren’t adequately talking about their competing priorities.
  • Related to both the psychological safety and communication themes, Redgate’s Chris Smith ran a workshop on Crucial Conversations. He explored some of the theory from The Chimp Paradox to explain how we can help get our ‘chimp brains’ under control at work, and how to accept it when we, or other team members, have chimp responses.

How to get change

  • One of the bases of what Chris was talking about is that you can only change yourself, not others. You’re much more likely to be successful by opening your own mind than trying to convince other people of your opinion. This was also a message that came across in Linda Rising’s second keynote of the conference on Moral Foundation Theory. Her argument was that we should aim to listen people into agreeing with us, and we can do that by examining issues from the opposing moral standpoint. For liberals, we tend to argue about care and fairness, while conservatives tend to focus on loyalty, authority and sanctity, so listen for their arguments from their moral foundations. Linda’s other key message in this talk was that correcting people with data tends to lead to them holding their view more firmly. This is because we’re evolved to take quick decisions and justify them later. It’s much more effective if we tell stories around our ideas like a press secretary, rather than as a judge spurting facts.
  • The different points of view in a team, and the challenges arising from them, was also the topic of Linda Rising’s first keynote of the conference, Who do you trust? Beware of your brain. She used the example of the Robbers Cave Experiment to show how we are keen to divide ourselves into them/us tribes, even when there is no difference between the groups, and how the tribes were only unified when there was a significant, real-life problem, to solve through joint teamwork. This is a lesson that software team leaders can learn from.
  • Selena Delesie’s keynote The Soul of Agile, picked up on some of these points. Her fundamental message was that we get the best from our teams at the intersection of three things: An open mind (curiosity, to have power with an individual, not over them), an open heart (compassion, focusing on the individual) and an open will (reducing fear and increasing courage in the team by being willing to release the need to control individuals and releasing the need to have future plans). Notice the focus on the individual here – you may need to take different approaches for different people.

Diversity in background

  • If all of the above focus on individuals seems hard, it’s worth it because of what different skill sets bring to the team. In the keynote The Death of the Liberal Arts, Ashley Hunsberger argued that there is a need for liberal arts students in development teams, because understanding what to analyse, the context of data , writing (communication) and giving feedback are all transferable skills that are gained in essay writing in literary criticism. The ability to juxtapose having a big picture view while also focusing on details is (literally) a feature of art criticism. So why do we often insist on Computer Science degrees in our job advertisements? The other thing that liberal arts students can bring is the consideration of ethics. The need for ethical programming is becoming increasingly visible with the likes of Cambridge Analytica news stories, and so asking ‘How can what I’m building be used to hurt somebody?’ is a useful thing to consider.
  • The other regard in which we might want to consider ethics more is asking ‘How does your product end?’. In the final keynote of the conference, Ends – The lost narrative of consumerism, Joe Macleod promoted thinking of the off-boarding journey for your product as much as you think about the on-boarding journey. Considering what happens to a product at the end of life, whether it’s software and there’s a hard user experience to let go, or hardware that needs recycling in an easy way, off-boarding can make business sense in increasing loyalty. (Incidentally, in our field of academic publishing, the whole industry has mechanisms to ensure content lives on after a publisher has ceased to exist.)

So what am I taking back to work?

First off, I haven’t covered a couple of the useful sessions in this post so far, because they didn’t quite fit the narrative flow. Given my recent change in job title, however, I found Stefan Tilkov’s session on Agile Architecture and Innovation particularly useful, and I’m certainly talking to my colleagues about his ideas. I especially liked his proposed Architecture Manifesto on the basis of the Agile Manifesto as a way of explaining what we are trying to do:

– Conscious trade-offs over emerging architecture
– Documented rationale over quick ad-hoc decisions
– Sustained changeability over easy initial development
– Design for replacement over design for re-use
– Simplicity over fast delivery

Stefan Tilkov

Beyond this, the other things I want to do include:

  • Considering the needs of individuals more than thinking of Agile as a one-size-fits-all thing that the team agree and compromise on. I may well promote Zara’s talk a bit within the company.
  • Trying to change the P1 process we have at work for retrospectives so they are more deep and get to better root causes. Of course we aren’t the airline industry, but thinking about some of their detailed analyses is something we could do to improve.
  • I’m going to read The Chimp Paradox so I can look out for ‘chimp responses’ in individuals and account for them better.
  • Think more about working with remote colleagues as we are in that ‘space inbetween’ of having just a handful of remote workers.

Agile

Hard Reset: A workshop at Agile Cambridge 2019

Leave a comment

Back in November at work, we ran ‘Reset Week’. Basically, we had a problem: we had three products needing work and three teams that weren’t quite the right shape and had the wrong skill-sets in the wrong places. But by making the teams gel, the result would be that the resulting teams wouldn’t have worked on their products before. To make matters worse, two of the three products had no roadmaps.

Due to externally motivated deadlines, I was given a week to resolve this. By the end of the week, we would need fully-formed roadmaps (at least to give a few months of runway) and teams that bought into them and understood the products.

How did it go? I’ll be presenting a workshop at Agile Cambridge with my former colleague Roger Hart, where we’ll re-run a compressed version of the week in 90 minutes. Come along and you’ll create your own roadmap working with people you probably haven’t met before…

Wednesday 2nd October 2019 at 13:25 at Churchill College, Cambridge as part of Agile Cambridge…

More information…

Agile, AWS, Software

Moving on…

Leave a comment

It’s time for a bit of work-related, personal news… My career has been one where I’ve followed a slightly unusual path, jumping at opportunities that have presented themselves. From a languages degree to Technical Authoring, then to Agile as a Scrum Master, I’ve been keen to explore different areas in a way that’s hopefully also been useful to my employer at the time.

I’ve been fulfilling a Scrum Master role for more-or-less seven years now, ranging from a highly Agile software company, to an American multi-national, to most recently an academic publishing business, which can only be described as slower-paced. I’ve really enjoyed working with various teams over the years but I’ve struggled to settle into a purely Agile Coach role, often ending up more of a Project Manager (for better, or for worse).

Our current challenge at the Royal Society of Chemistry is a big one, but one that will be familiar to many companies. We’ve grown up with a huge, and largely poorly-understood, legacy in-house infrastructure which has now become unsustainable. Over twelve months ago, we took the decision to take a strangulation approach to it, and gradually rearchitect it in the Amazon Web Services cloud.

It quickly became evident, however, that even after this decision had been taken, there was little movement in actually spinning up AWS infrastructure and we didn’t have the in-house knowledge to take a best-practice approach to the architecture. As Scrum Master to the team most directly impacted by this – and therefore as the person who was supposed to remove blockers and impediments – this rapidly became a significant concern.

And so, I took it upon myself to throw myself behind the cloud adoption strategy. Last autumn, I passed the AWS Certified Cloud Practitioner exam and I started studying for the Solutions Architect Associate exam. More recently, working with an external consultancy, we’ve been looking at forming a Cloud Centre of Excellence (CCoE), as recommended by AWS. One of the features of a CCoE is to drive the organisational change needed when adopting cloud technologies, and in particular evangelising a DevOps operating model. I’ve also been showcasing the technologies to our stakeholders to help them understand the transformation we’re doing. Both of these aspects are very similar to what I had been doing promoting Agile.

Last week, all this became official. My job title is now a Solutions Architect, still at the RSC, but now in the Site Reliability Engineering team, with the aim of driving our cloud-related projects forward. I’ll still be working on the same cloud migration project with the same people but now in a much more technical capacity. I’m really looking forward to getting stuck into the new role properly as we have the opportunity to make a significant difference. Time will tell, but the team has already delivered some impressive work in the last few months, and I’m sure there’s a lot more to come.

Agile

Measuring Technical Debt

Leave a comment

Recently I saw a Twitter thread bemoaning the rise of ‘psychological safety’ as a phrase. Not because it isn’t important – it is – but because giving it a name has made it acceptable and risks giving a scapegoat to hide behind. There’s the possibility that just saying ‘My organisation / team doesn’t promote psychological safety’ makes it OK.

I have similar feelings about ‘Technical debt’. As a phrase in its original sense (the fact that not keeping code clean and deployable eventually incurs a real, financial debt that must be repaid), it is useful. But every non-startup I have heard of has struggled with ‘technical debt’ and it has started to be accepted as just something that happens.

Let’s not forgot that not all technical debt is bad. There can be good reasons for taking on debt, and in fact I’d actively encourage it in embryonic, experimental projects, provided it is monitored and quickly repaid.

But with excessive debt, a time comes when development teams become so crippled by the legacy code that progress slows to a snail’s-pace and further work becomes unacceptably risky. In these situations, where should you start with ‘debt reduction’? I’ve seen projects that have quickly spiralled out of control into a whole rewrite project, or worse the team can become embroiled in a long discussion about the best way to fix things.

One of the teams I work with was in such a state last year. In fact, it was worse than that. They actively did not know which products they had on live servers (these are all websites) and there were so many inter-dependencies between them that it was almost impossible to know where to start.

Fortunately, our boss understood the problem and was willing to give some time to sort it out. Together with the Product Manager and our Site Reliability Engineering team, I helped to set some goals for the team to start tackling the issue, without ending up rewriting everything. Our definition of what good looked like was:

  • No websites should be running on unsupported operating systems
  • All sites should be deployed through our CI pipeline
  • All code should be deployable at any time
  • Nothing should be deployed with a CVSS score greater than seven (still high, but a reasonable, measurable start)

(This may sound unambitious, but based on where we were starting from, it was a big challenge!)

The team then set about fixing up the most obvious candidate products.

We soon hit our biggest issue. With the team struggling to untangle the code – often written by contractors, and so completely unknown to the team – progress towards these goals was slow. It was also almost impossible to report on. Furthermore, because we didn’t even have a full list of products or which servers they were on, any measure of progress towards an end-point (such as burndown / burnup) would be meaningless. 

In SMART terms, we didn’t know whether our goals were achievable or realistic, and so we also couldn’t timebox them, which was unacceptable. It was clear we needed a way to measure and to report on the beast. This was not just for management. Part of our thinking was also that measurement would help morale in the team. Although at this early stage, they were pleased to have the opportunity to work on something that had been a pain for years, but it was clear that this would not last without regular celebration.

It was time for a creative solution. Inspired by both the term ‘technical debt’ and the product management prioritisation technique of asking stakeholders to vote on backlog items with money, I sat down with the team and we discussed which of the goals were ‘worth’ more, and put some values on them. We also took the opportunity to hold ourselves to account by putting a bit more rigour around the goals, for example, when we said ‘code must be deployable through our CI pipeline’ we specified ‘to live’, since deploying to UAT only might make the situation worse.  Similarly, to measure ‘deployable at any time’, we specified that there must always have been a live deployment within the last two months. If no development work had happened for the previous eight weeks, deploying an unchanged build would be fine, but at least demonstrates that rot hasn’t started to set in. It’s the little failures nobody notices while they have other priorities that tend to add up to debt later.

The result was our new list of problems and their debt value:

  • Websites running on unsupported operating systems = £25
  • Websites that couldn’t be deployed all the way to live via the CI pipeline = £50
  • Websites not deployed in the last two months = £20
  • Websites with a CVSS score that was 7+ or which was unknown = £40

This allowed us to take the list of products that we did know about and calculate our starting level of debt: £3535! This approach also meant that, as we uncovered new information (whether good or bad), we could show this by adjusting the debt amount. In fact, five weeks after starting, our ‘debt’ figure had worsened to £4015, as the team had learnt more.

But above all, it meant that we could show our progress graphically and set up a celebration point for the team whenever we had removed £1000 from the debt.

To get a bit more momentum around this idea, I also acquired a load of Monopoly money and displayed it in the office, so colleagues could see money moving between a ‘debt’ and ‘repaid’ pile whenever something was ticked off the list.

Technical Debt visualised as Monopoly money

This visualisation had a mixed response in the team. In all honesty, most developers were ambivalent about it, even if managers mainly liked it. It also had the slightly unforeseen difficulty in not being a great demonstration of continuous progress: The inter-dependencies tended to mean that the graph would go several weeks without many changes and then a big improvement as many products got deployed together. I’m not convinced that I ever managed to fully explain that that wasn’t a problem. If you try this in a similar context, I’d certainly recommend stressing that it shows repayment of whole pieces of debt, not progress through the work itself.

By the way, ‘repaying’ the debt wasn’t only a development job on this team. It was also a product management job as we managed to decommission a number of rarely- used websites. In those cases, we repaid all the debt on those products at the moment of decommissioning.

Recently, there has been an unexpected result of the graph. Remember that rule that everything should have been deployed in the last two months? In November and December, priorities changed, and the technical debt reduction work was paused (or rather, deprioritised) for a while. Due to this, and the fact regular deployments hadn’t been set up, the graph is now starting to head in the wrong direction, as deployments made in October start hitting the two month limit. This has served as a useful reminder that there’s still work to do if we aren’t going to have more problems in the future.

Debt graph

Are we aiming to repay the amount of debt all the way to zero? No, probably not. There wouldn’t be sufficient value in doing this. But monitoring this figure in the future should be a good mechanism to track where we are, and to ensure that the products are not allowed to become bad again, after all the investment we made during the latter half of 2018.

Agile, Project Management

My first six months in a new role

1 Comment
The spectacular view of the London skyline from my office roof

As those of you who know may will recall, I was made redundant from my job as a Scrum Master at Redgate back at the end of May 2016.

Luckily, I wasn’t out of work for too long and since July I’ve been working at a major multinational ecommerce player in the travel and tourism sector (very boring company rules on publicity mean I’m not allowed to name it in blog posts but that doesn’t matter)…

The team I’m in is a backend data team, looking after a number of the main company data sets. As such, our ‘customers’ are the business analysts and finance people who use that data in their reporting.

The wide variety of the sorts of data we handle means that the team is split into sub-teams, handling everything from the basic booking data to feeds of bookings with commission payable on them, and from the data relating to A/B testing on the website to the OLAP cube that’s used by the financial people for reporting up to the CEO.

New challenges

This role has brought a number of new challenges for me so far, which stem just from the nature of the job:

  • Our customers do queries in the databases we provide, so ‘user stories’ are often no more interesting than ‘provide this data in a new column in this table’. Sure, we can explain the business value but there’s no concern about usability and, with the stakeholders in the building, there’s often no interpretation required about what they are actually trying to achieve.
  • Being a publicly-listed company, there are significant overheads from security and auditing concerns. As much as I might like the team to fix their own problems, they don’t have access to a lots of the production systems they deal with in order to enforce separation.
  • Since I joined, it has become a global team, with people in both London and Bellevue (Seattle). In an attempt to avoid siloed knowledge, we have refrained from having some work done in the US and other work in the UK – the work is shared between the two locations, even though this creates a bigger need for communication.
  • Like any team handling backend systems, work is regularly interrupted by critical production issues that can’t wait. This is made more difficult by the fact that we’re handling very big data where processing times are measured in hours if not days. Any delay in fixing a problem can have significant impact on downstream systems and, ultimately, the ability of people to do their jobs.
  • The sub-teams work with a Technical Product Manager, rather than a Product Owner as defined in Scrum. The difference is that the TPM owns not only prioritisation but also provides a lot of detail (sometime too much) on the implementation side.

The first few weeks

As is always recommended, I spent my first few weeks watching how things were working and understanding what I should concentrate on trying to assist with first, resulting in a set of objectives for the first few months, such as:

  • Restarting retrospectives
  • Getting epic-based swimlanes on the Scrum board

    (Individuals had their own work in their own swimlane, which was clearly hindering the notion of working as a team when they each had diverse work to do, and would make migrating towards a Kanban approach in the future problematic)
  • Stopping the useless sprint demo sessions the team was having

    (Stakeholders never attended them and so it was just a ritual for the developers to show each other what they had done in deep technical detail)
  • Supporting the existing initiative to have dashboards showing what data was available and how up-to-date it is

    (The team was working in a very reactive way and often wasn’t aware of problems until our end-users had spotted them.)
  • Improving communication with stakeholders through introducing regular A3 Reporting on high-profile / larger projects

There were two much bigger things on my plate, though, which I’ve spent the bulk of the last few months on: getting the Bellevue team set up and developing the sub-team that’s responsible for the OLAP Cube.

Creating a remote team

This was actually a process that was underway around the time I started. My line manager had just moved to Washington State from London to found the team there, and moving the two Indian developers to the US was just the first step. Between October and December 2016, we had two more developers join in America too.

What fell to me was the practicalities of making this work, when the majority of the developers were still in London, as were all the stakeholders. When it had just been two developers in India, standups could still happen in the morning in London, which was early afternoon in India. The switch to the Seattle area now meant that it was the middle of the night there during the morning in the UK. I didn’t want to move the London standup to the late afternoon, as that would rather defeat the object in my opinion, so we introduced an additional catch-up call at 4pm UK (8am Pacific) to ensure work is kept in sync. To complete the cycle, the team in Washington State leave a message in HipChat at the end of their day, which we can pick up the following morning.

This sort of practicality, and the need for planning and retrospectives to be held at the end of the London day, is easy to resolve, while not ideal. It has certainly led to me having a very odd-shaped day, where I’m relatively free during the middle of the day, and very busy with successions of meetings at the end. The most unexpected issue we’ve faced is a perception among stakeholders that quality has reduced: If things broke overnight London time, they were used to it being resolved by the Indian developers before the stakeholders arrived at 9am UK. That no longer happens, of course.

The biggest issue however, predictably, is communication. As I’ve said, we’re trying to avoid silos by having work shared between people in London and Bellevue, rather than allocating different tasks to each place, but that means getting communication right is incredibly important and complex. Add into the mix the need for the TPM in London to answer the majority of questions, and you’ll understand the difficulties when there’s only 90 minutes of time overlapping between the two offices per day. We’ve already had a couple issues where the Bellevue team-members weren’t quite aligned on the project goals, and a near miss where work was almost duplicated between two people in different places.

Cube team

When I started, the Cube team was in a really bad place. Unlike the rest of the wider team, they’d been pretty much ignored and didn’t really have any process at all – neither Waterfall, nor a form of Agile. They had lots of disjointed, small tasks with no real stories tying them together, and a technical debt backlog kept separately from the product backlog. We’ve had a couple of stories that were aborted during development, because it became clear that the value wasn’t as expected. Above all, however, the people in the team didn’t get along. There was a lead developer, who had the technical skills but struggled with people skills. He was also the original (only) team member. Subsequently, another, more personable, team member had joined and had been made team lead before I started, which naturally caused some tension between him and the original developer. And, most recently, a new junior developer had come into the mix, and was naturally caught in the tensions between the two, which I think was partly a cause of him having little confidence.

What a dream scenario! I believe strongly that people problems can be fixed if caught early enough. I took them through a ‘5 Dysfunctions of a team’ training session over several weeks and we managed to resolve many of the interpersonal conflicts. Simultaneously, I put some gentle pressure on the TPM to improve the roadmap and started setting them up with a basic sprint cadence and teaching them Scrum. This doesn’t come overnight, of course, and they’ve struggled to complete their sprint commitments but by showing them data on what their velocity is, we’re getting nearer.

The plan from now on

The next few months are also going to be very busy. A couple of new major projects is almost certainly going to result in having to adjust the team structure slightly. But beyond that, there are a number of more fundamental things to fix:

  • I want to get the OLAP Cube sub-team working closer with the other sub-teams that they are dependent on. This isn’t helped by them having a different TPM and a different roadmap.
  • I’m introducing a more thorough kickoff process for larger pieces of work, moving from a model where one or two people do a spike and produce reams of spike documentation, to one where the whole team is involved hearing directly from the business what the requirements are.
  • Starting to experiment with introducing BDD as a way of having conversations about the requirements.
  • Set the expectation that Sprints should end with a tangible thing that can be demoed. At the moment, everything is still very waterfall. I see creating regular, tangible, value as the key to getting stakeholders back into demo sessions.
  • Starting to introduce Kanban in the sub-team that has most production issues to deal with.

Exciting times ahead… 

Agile, Project Management, Red Gate

Direction, momentum and oil: Why software development teams are like bikes

Leave a comment

I haven’t posted for a while on what I’ve been up to at work and there’s a good reason for that: the team I’ve been working with was pretty dysfunctional twelve months ago and I’ve been spending the last year trying to turn them around. In this post, I’ll describe that journey and why I’ve concluded that software development teams are like bicycles.

Context

I was first asked to spend a couple weeks in April 2015 working with Redgate’s SQL Monitor team as an external observer, finding out why their Project Manager was struggling and why the team was perceived as delivering slowly. In this environment, some of the team were even starting to leave the company…

A bit of background: the SQL Monitor project had been stopped for just over a year after the release of version 4 in January 2014. In the intervening time, no work had happened on the product, and the original team had disbanded. Many had gone off to do other things. In the meantime, revenue had continued to increase and so, in early 2015, the company decided to form a new team to continue that growth. The new ‘team’ (a mix of different people that had been cobbled together, most of whom had no experience on SQL Monitor) would work on one feature in order to get used to the codebase, while the company decided on what the further roadmap looked like. That didn’t sound like a team set up for success.

In my favour was the fact that the team knew they had a problem and so were prepared to work with me to address it. We didn’t have much time, though: the following week, two newly-hired developers were set to join them and we needed to make their introduction as good as possible. So, the usual advice that a Project Manager / Agile Coach should just start by observing the team went out the window. I had to act fast.

I started by organising a retrospective with the team to investigate their various problems. This was a fairly painful session, which I facilitated, using a fishbone diagram technique:

Fishbone diagram

As expected, there were a number of really fundamental issues here, mainly with the state of the code, but feedback was that it was quite cathartic for the team to see them written down in one place.

To summarize, this was a pretty bad situation: the team had no goals, no deadlines and were struggling to do valuable work. In other words, they had no direction or momentum and the bicycle chain wasn’t even allowing the different parts to work in concert.

Giving a direction

During the same week as that retrospective, I sat down with the Product Manager and got his opinion on commercial direction. It turned out he had a presentation, that had already been presented to other managers, with his idea of a roadmap. It was a well-thought through plan; it’s just that the dev team hadn’t yet seen it. That was another easy fix – I got the Product Manager to give the presentation to the developers and let them challenge the rationale. With this approved, he could then go away to create a 5-year roadmap – rather unusual at Redgate but vital for giving the developers a sense of confidence that the project wasn’t about to be stopped for another year again.

The outcome of the Product Manager’s work was that there were two obvious big features to be done soon – support for Active Directory and support for SQL Server Availability Groups. Both of these were features which competitors had implemented already, and were actively costing us sales. The work felt likely to take a few months, it was mid-April 2015 and there was a big tradeshow, SQL PASS 2015, coming up in October. The way forward seemed clear. I’d re-focus the whole team on those two features and temporarily shut down the other distractions the small team was simultaneously trying to work on (mainly support escalations and code quality improvements). Stopping distractions, and especially support escalations, isn’t easy, and can’t be a long-term solution, but in some cases I’ve found it to be a very useful tool. If you have the right conversations, even the support personnel in the firing line from customers can be brought into the idea.

Some people may question the idea of stopping the work to address code quality but the fact is that random pieces of improvement work were being picked up by developers with little planning or sense of what the most important work to do was. Instead, I asked the team to just write down the known technical debt on Post-Its as they were discovered, and stick them to the wall. Together with the actions from the fishbone retrospective, that became a visible reminder of how much debt needed paying back and, in due course, would form a backlog of improvement work to be addressed.

The tech debt wall

Of course, things never are straight-forward. Just as I was starting to get this running, a critical security vulnerability was found in SQL Monitor 4.1. The plan had to be put on hold while SQL Monitor 4.2 was released, and the fix was also backported to previous versions of SQL Monitor.

Explicitly showing what work is prioritised and what is to be done later on

 There – my job was done. The team had a road to follow and, a month or so after joining them, I could leave them in the hands of their normal Project Manager to continue the work…

Speeding up

At the end of July, I returned from a holiday to discover that the existing Project Manager had decided that he no longer felt confident in managing the project. It was a very brave decision to make, and meant that I was catapulted back into the SQL Monitor team again – this time as full-time Project Manager.

My job, though, was fairly obvious by now: keeping the team focused on delivering in time for SQL PASS 2015. This mainly consisted in keeping a fairly tight reign on the team’s approach to sprint planning and capacity planning, at least at first. This ensured also that there was no time available for some of the quality improvement work the team wanted to do, a deliberate choice by me to keep the sense of purpose. Over time, we found a way to handle support escalations once again, and I eventually released the brakes I’d artificially applied to guard against scope creep/changes.

To bring the team back on track, we enforced user stories, estimation and acceptance criteria.

We did it – SQL Monitor 5.0 was released just before the tradeshow, on time and without significant amounts of cutting things from the planned scope.

Through the work on version 5.0 we’d deliberately not released frequently: releasing was hard and required extensive manual testing because the automated tests were unreliable or covered the wrong things. Given the length of time since the last major release, and the team’s lack of knowledge, trying to release regularly could have caused too many derailments.

So, having reached SQL PASS 2015 intact, it was time to set the next challenge for the team: work out how to release frequently. For the major release, the testers had done an extensive risk-based analysis on the major areas of the product they’d need to test. They now used that as a first step to working out what minimal testing would be required. Meanwhile, the developers set to work getting rid of those unreliable automated tests, replacing them where possible, and changing our branching strategy so we’d always be in a position to release.

And that backlog of improvements to the code we needed to make? I got buy-in from the stakeholders so that we could keep our promise. We gave the team two months, without pressure to deliver features, to start working their way through the backlog.

By the start of 2016, we could release every sprint and some of the most egregious code problems were gone. The bike had gained its own momentum.

Providing the oil

Naturally, it hasn’t all been completely plain-sailing. The release notes from the start of 2016 show that there were several bugs and regressions that we introduced as we moved to the new model. For the customers who were affected, we’re sorry. But the fact that the SQL Monitor code, while still far from perfect, is improved, better understood by the team and able to be released quickly means that we’re in a much better position to do some great work in 2016 than we were last year.

By now it should be clear what my role in all this was. The team is pretty much unchanged. The way they work is only slightly different. The developers, testers and designers are still the gears, chain and wheels of the bike. Without them, we wouldn’t have a bike at all. My job was one of lubrication: providing the oil and a bit of direction and momentum to ensure they could go on their journey. It was done through adding brakes and stabilisers where necessary to prevent them from going too far off-course in the early days, then empowering the individuals by giving them the time and space to address the problems they wanted to fix and by protecting them from the mud of management that might have interfered with the way they were working.

Postscript

The team’s now working on some really exciting stuff ahead of SQL PASS 2016, including a very impressive new dashboard which is nearing completion.

As for myself, my mentor, Simon Cromarty once told me that the aim of any project manager should be to make themselves redundant: You succeed when the team doesn’t need you any more. So, in a rather perverse way, it should be good news that I stand to be made redundant from Redgate in the coming weeks. If you need someone to help oil your bicycle,  let me know…

Agile, Project Management

What is technical debt?

Leave a comment

This article is my own synthesis of ideas presented at Lean Agile Scotland in September 2014 by Dan North, Jabe Bloom and Torbjörn Gyllebring, with some of my own embellishments. Any misunderstanding of their points is mine! I’ve linked to their original talks at the foot of this post.

Conway’s Law says that the design of our system will be rooted in the design of our organisation.

So, we can never design a system from scratch: some of the design will have already started before we start designing.

This isn’t necessarily a problem, however: indeed, we should embrace it, because organisations suck if they don’t have the right APIs and other architectural borders in place between where teams are (that’s when teams start to get bureaucratic layers preventing their work from proceeding as their pace.) We should take time to get this right, and this is why having micro-services to do communication between teams is a good idea.

However much we work on this, however, we will face problems in the code we write, many of which will relate to the architecture. We can either chose to fix the symptom (via a quick fix) or to fix the underlying problem (which is generally more time-consuming). Quick fixes aren’t necessarily bad when applied in conjunction with a longer-term fix but remember they are akin to painkillers, and you can’t take painkillers indefinitely.

Quick fixes often take the form of a mandated structure: we might add too much or too little process, a particular build system, an architectural design ‘recommendation’, a decision on a particular source control system or IDE, or similar. As managers, we should be especially careful with such quick fixes. We are aiming to apply a ‘Goldilocks’ amount of constraint on our teams: too constrained and development will stop because the people can’t do their job; too little constraint and there will be too many options to do the right thing. The right amount is a strategy that’s clear enough to give us a task for today and a constrained set of options for the future. That strategy is actually a conversation – top-down constraints from management, and bottom-up information and designs from the development teams.

Partly because of the difficulty of identifying the ‘Goldilocks amount’ without information from the team (which presupposes time taken to do investigation), quick fixes generally tend to exhibit more unintended consequences than fixes that correct the underlying problem.

In a Cynefin sense, rather than taking time to analyse a problem as demanded by Complicated scenarios, our quick fix has made matters worse by creating a Complex relationship. The side effects can’t be foreseen, but become apparent in retrospect. Worse still, the quick fix may fail to address the root cause of the problem, and so we may get a positive feedback loop causing problems to increase as a result of our solutions.

This is technical debt, because the system starts constraining what we can do.

Cynefin tells us we should monitor for this positive feedback and, when we sense it, take time to respond to it, possibly by rolling back the change and trying a different approach.

To check for technical debt, therefore, we should periodically take time to ensure that our system isn’t constraining us. We can do this, for example, by ensuring our tools haven’t become our masters by checking that we can still build our products using a single command line (without complex CI systems) and that refactoring can be done in Notepad (without Resharper).

Original videos:

Agile, Project Management, Red Gate

Understanding a feature through 3-amigos

Leave a comment

The last project I ran here at Red Gate was on SQL Compare / Data Compare 11 project. SQL Compare is a superficially simple product designed, as it’s name suggests, to compare Microsoft SQL database schemas. Its cousin, SQL Data Compare compares the data in the tables.

Like the product itself, the Compare 11 project had a superficially simple goal: add support for the new features in SQL Server 2014. Easy, right? It turns out, not so much…

The largest new feature is Memory-Optimized Tables (MOTs): the ability to keep data in memory so you get significant performance improvements through reduced disk-access. But as you dig into what MOTs can and can’t do, you realise that you’ll need a lot of special-cases in the code to deal with the limitations. The details of those limitations is well outside the scope of this post, but one of my colleagues Michelle Taylor has written a useful post on our company blog for those who want to know more.

When the team came to start work on this story, which was before the release of SQL Server, some of the limitations weren’t immediately apparent. This led me (as project manager) to a false sense of security, and so I was far too lax on getting the user stories and acceptance criteria defined in detail. The lack of knowledge this created in the team caused the feature to become massively under-estimated: just 50 task-points (at their velocity, each task-point turned out to be just over an hour of real work).

The first tasks related to this story went on the Scrum board early in the project: ‘Add property to CreateOrAlterTable’, ‘Add property to DatabaseObjectsTable’, ‘Update DatabaseObjectsTable.Equals()’. You see what’s missing? We had dived straight into implementation with no form of planning first. That first sprint was estimated as 15 points. By the time we got to the end of it, we had uncovered quite how unexpectedly complex this feature was likely to become. We were nowhere near 30% (15 out of 50 points) complete on the feature. We carried some of the tasks forward to the following Sprint, however, trying to convince ourselves that we’d crack it soon. Before I knew it, we’d spent nearly a month on this feature, continually uncovering more and more complexity, and morale on the team was starting to become impacted.

Even then, my monthly project A3 report excessively optimistically promised that we’re complete both Inline Indexes (another 2014 feature) and Memory-Optimised Tables in the next four weeks.

It was time to pull the Andon cord.

Luckily, this also coincided with a slight change to business focus. The team was being asked to ship SQL Compare 10.7, giving support for connecting to SQL Server 2014 (but not the new features) to ship with the final release of SQL Server 2014.

Whilst this was a massive disruption to the team, it did give us some breathing space to find out more about MOTs before we had to work on them again.

By the time we were ready to start work on MOTs again, the Product Owner had defined the Acceptance Criteria in more detail. There were a series of tasks, themselves totalling 42 task-points in estimates, which centred on reviewing these ACs and making sure they made sense. This worked OK, and put us in a much better place, but with a large team, the meetings didn’t really work.

At this point, one of my colleagues pointed me to the idea of a 3-amigos meeting. I’d actually tried this once before on a different project, and I found myself wondering why I hadn’t tried this earlier on the MOTs work. The idea is to have three people (specifically the Product Owner, a Developer, and a Tester) in a room together, defining how the feature should work for users, in the code, and in test cases respectively. There’s a good write up of the meeting on the Scrum Alliance website; the only difference for us being that the Product Owner also doubled in the BA role for us. Although it felt slightly more waterfall than we were used to, the output from the meeting was a set of really well defined, clear Acceptance Criteria and a good common understanding of what MOTs involved, that was ready for sharing across the team.

The team re-estimated the work, and came up with an estimate of 250 points. In the end, it took 337 – still rather more than 250, but also far more than the original 50 points planned!

This process worked so well that we adopted the same meeting for other features in SQL Server 2014, rotating the role of the amigos through the dev team. As long as the communication about what was discussed is good immediately after the meeting, everyone became brought into the idea of 3-amigos, and ensured that the chaotic start to MOTs wasn’t repeated during the rest of the project.

Agile, Project Management

A year of whiteboard evolution

Leave a comment

My colleague, Simon (aka The Agile Pirate), has written a post on his blog about how our team’s Scrum Board has evolved over the past year. Until I Simon’s summary, I wasn’t quite concious of how much we have achieved since January.

It’s probably worth supplementing Simon’s post with something about the aim of the ‘bigger picture’ sections of the board, which I was primarily responsible for instigating. One of the problems we had with the standard current-sprint scrum board is that the team wasn’t always aware of where they were in the bigger scheme of things. This meant that there seemed to be a tendency to get bogged-down on a particular feature or story, and lose sight of how it fitted with the rest of the release. The was a particular problem on the ANTS Performance Profiler 8.0 project, which was a longer duration than many of our normal projects. Ensuring we had a visible record of what’s coming in future sprints meant that the team kept this focus better.

The Project Plan sheet that I had developed by the October photograph is an attempt to clarify our goals and objectives a lot better, helping the dev team to take informed decisions about the value of any particular task. I’ll blog about the rationale behind the design of that page another time.

Read A Year of Whiteboard Evolution on The Agile Pirate.

 

Agile, Project Management

Reflections from Agile Cambridge 2013: Day 3

1 Comment

[This post continues from Reflections from Agile Cambridge: Day 2]

Real Options in the real world (Chris Matts)

This was a fascinating keynote investigating how we can consider the options we face in project management as being Options in the financial sense: they expire, they have a value and we should avoid committing to them too early.

Being able to take options requires you to have slack time in which to take them: Chris used the analogy of the lion who is an effective predator because he spends most of his time asleep in a tree. Taking options also involves considering how we will roll back a decision. There was a warning against using mathematical probability to decide what is unlikely to happen, because human beings tend to be too optimistic when calculating risk. Instead, we should aim to have a plan for what we will do if the risk occurs.

The other strand in this talk was around staff liquidity. All too often, we allocate people to tasks because they are the expert in that field. This just creates silos and dependencies on individuals. Instead, we should aim to keep the experts for training other people or for responding quickly to unforeseen circumstances.

The art of systemic feedback (Marcin Floryan)

Marcin’s talk considered the best ways of collating feedback on what we do. The main thing I learnt was about the relationship between the Cynefin model and the appropriate feedback loop to use. Marcin demonstrated how Complex problems should be considered using the OODA loop, while PDCA is more suited to Complicated problems. One area in which I wasn’t so happy was the idea that Simple problems don’t require feedback because there are established practices. Surely those practices themselves have room for improvement?

It’s not your fault – why targets don’t work (Francis Fish)

This talk also had two parts. The first was a reminder about normal distributions and standard deviations. If your business asks for a 6% increase in sales in the next 6 months, this is meaningless if the 6% is within the natural variability of the measure. All too often, we forget this, because we just look at the output from a spreadsheet.Francis suggested that we should put the standard deviation on all the graphs we produce for management to highlight this issue.

The second part of the talk described the distinction of ‘efficient’ and ‘effective’: the former is best for a short-term bottom-line but is high-paced and so there is little room for innovation; the latter is more about the long-term, and allows for innovation and experimentation. The final slide contained an interesting note: that we need to be ‘response-able’ for improvements, i.e. that we need the slack time to improve.

 Unlimited resources deliver faster… right? (Alun Worrall)

I always appreciate conference talks where companies feel able to give a ‘warts-and-all’ presentation, owning up to simple mistakes they have made. This was one such talk as Alun described a big project at AutoTrader which involved releasing some software by a critical date. Having estimated that it wouldn’t be possible, the company threw a load of contractors at the problem, hoping that it would speed it up. But the product was still delivered when it was originally estimated, and all that they had achieved was a lot of stress. It also appears that some fundamental mistakes with prioritisation of the individual components of the system could have been done better, which might have helped some of the project to have been delivered earlier.

How I became a spaceship commander (Tomasz Borowski)

Tomasz had tried using the YouTrack system to create YouGame: a gamification of the story-points achieved. This was initially done in response to a particular set of issues, including tasks moving straight from ‘To Do’ to ‘Done’ on the board, and inaccurate estimation. Tomasz described the issues he had faced in various iterations of the game’s development, including the risk of the leaderboard being used for performance bonuses (not the intended use!), and the need to add a concept of ‘stamina’ to prevent people from marking excessive amounts of work as Done.

To me, it sounded very much like an excessive response, and it must have taken a lot of time to create the app, however impressive it is.

The Art of the Retrospective (Chris Smith)

I’ll admit to having seen a practice of this presentation at Red Gate a few weeks ago but this made it even more interesting for me to see the eventual, finished, talk. As someone who hasn’t traditionally done many Sprint retrospectives with my team, I found the number of practical examples in Chris’ talk very useful. I’m certainly looking forward to trying many of them out.

 

Overall, I thought Agile Camrbidge was excellent, and I hope to be back next year!

Agile Cambridge