Hard conversations – Burn the ships

I want to tell you a story.

Paul* started as a junior engineer at a sexy tech company shortly after completing his BSc CS Masters. He worked hard, wrote good code, and got promoted. He developed a reputation very quickly as a rising star and soon got put onto teams doing tough projects.

Let’s move to introducing John*. John moved to Paul’s company after learning the ropes at another very good development shop, and came with a good reputation. John and Paul are now two mid-level engineers with a great track record, and are given a big ‘prove-yourself’ project and a handful of young engineers to lead. Here things fall apart. The team is morbidly unhappy, sprint goals are missed week after week, the team starts slipping on delivery dates like a toddler in a skating rink. Internal conflicts start picking up and people area leaving not only the team, but the company.

I have encountered this team in different levels of disfunction in various environments, either as a team member or an observer, and it is a pattern I have started to recognise in our industry. Let’s unpack some symptoms I have observed in the past:

  • Code reviews are scathing and personal. “Are you THIS stupid?!”, “????!!!!!” or paragraphs of passive aggressive criticism are the norm.
  • A senior engineer holds the big model in his head and hands out little chunks of micro-managed work to other engineers. When questioned, he answers with a ‘well, why do you need to know everything, I know what we are working towards, you just do your bit’. No one knows how the project is tracking or how to even find out how the project is tracking.
  • When someone throws a white-board across the room, you can just as well disband the team and go on holiday.
  • Newer team members are not allowed to contribute to a conversation, and are shut down with the “well, once you have been here as long as we have, you will learn how we do things around here.”
  • The one who talks the loudest, wins.

Let’s head back to Paul and John and see how they are doing. Unfortunately, the constant praise they are used to have now de-generated into increasing pressure from senior management to get their team and project in order. In the past, Paul and John got where they are because they were smarter than everyone else AND they had the grit to push harder to get the work done. They have been rewarded for this repeatedly throughout their careers. It is just a case of doing this once again, right?

Unfortunately the strengths required from Paul and John right now are completely orthogonal to being the smartest and most hard-working engineer in the room. What no one told them was that they now suddenly have to empower other engineers to be the smartest and grittiest workers in the room. In order to do this, Paul and John suddenly have a steep learning curve ahead – they need to create a high conflict, high trust space in which younger engineers feel safe enough to deliver stretch work despite high ambiguity and self-doubt, probably while experiencing all of this themselves. This is a very steep learning curve and I have seen engineers fail in this space either by getting themselves fired (the white board incident was anectodal) or leaving out of frustration.

I have good news and bad news.

The bad news is that most engineers are a John or Paul at some point in their career, and for some this is a wall they can’t overcome.

The good news is that I believe that this is fixable.

A legitimate strategy can be to isolate technical experts and buffer a team from their emotional toxicity by separating a technical visionary role from a generalist team lead role. I am personally not a fan of this, because this means you have very senior brittle engineers and team success relies on inter-person collaboration between people known for EQ as a weakness.

Another strategy is to foster a culture in which Paul and John is introduced to this learning curve earlier in their career and are given mentors who are successful at facilitating team cultures like this. I can name people I regard as these role models I have had in my career – Malcolm Hall, Mylo Mannya, Anthony Robinson, James Greenfield and, most recently, Ian Davies.

I have, to date, chosen my career opportunities based on the presence of such people in a space – this is very important to me.

I don’t like throwing more and more critical feedback at John and Paul – this makes them more anxious about their inability to succeed and probably leads to worse leadership behaviour and more general angst. If we don’t empower John and Paul to succeed in this learning curve, we risk losing great engineers.

This brings me to burning the ships. Captain Hernán Cortés arrived at Veracruz in 1519, and had to capture new territory. He instructed his troops to burn the ships – he knew the only way to instill enough motivation to succeed was if there was no option to turn back. I have walked away from professional spaces with problems in the past, as many people do. We are fortunate (or unfortunate) enough to be in an industry where development skills are in high demand, so we always have the ‘ah, I have options open’ in the back of our minds. The accessible back door is a psychological safety tool that all people tend to leverage. The problem with this is that we don’t tend to take ownership for the problems in the space we are in now.

We need to burn the ships psychologically and commit wholly to taking ownership for the culture of the space we find ourselves in right now. The great working environment we all want won’t exist except if someone intentionally builds it.

How am I taking responsibility for building the kind of working environment I would like to work in? By opening up the tough conversations, while making sure people around me are emotionally safe. By rewarding the quiet rock stars vocally. By advocating and sponsorship of engineers who might find themselves in tough conflict situations. By talking about problematic behaviour I observe and building allies through conversation. Reward the good, don’t attack the bad. Ride out the tough times to stay long enough to see the good emerge.

Go burn your ships.


Keep your call path primed

I recall asking my Mom as a kid on Winter mornings why she always turns the key before starting the engine, and her response was “I need to prime the engine, because it is cold.”

My team is working on something that is common in a service-based environment – writing a replacement for an existing service that we intend to hot-swap without taking an outage. If something goes haywire (initially) we retain the ability to flip traffic back to the legacy call path as a backup, right?

My concern is this: As soon as you reroute traffic from the legacy call path, you will have to assume that that service’s ability to serve that call volume immediately starts to degrade. The team that owns it becomes disincentivised to maintain that service to accommodate the call volume capacity it is carrying today. Since they don’t serve production call volumes anymore, they won’t notice if their service degrades, and in our eco-system, that happens in a matter of days.

I recommend a new migration design pattern – keep your call path primed by routing traffic through ALL call paths until you are comfortable retiring a call path completely. You can do this by routing production traffic through shadow call paths, generate mock service calls that hits extraordinary paths frequently or rely on integration test/black box test coverage. This also includes validating backup restores and tasks you rely on in operational crises.

Hostage Work

I am fortunate enough to work on a team where we are able to address about 30% (thumbsuck) of the needs in our backlog. I know this sounds incongruous but it means that everything I work on is really, really important – important enough to trump other work not making it into sprint.

I am also fortunate to work on a team with people who are constantly driving for improvement. We need to improve the technical debt incurred by existing bugs and badly implemented service architecture. If we were to whip our code base into the shape we would want it in, we wouldn’t be delivering feature work for at least a year, but isn’t that the case for all developers?

In our environment we use a code review tool – all commits need to be reviewed and shipped by your peers before it is allowed to be committed. This is awesome. It serves both as a quality control measure and a mechanism for continuous knowledge sharing.

The problem, however, comes in when people start holding off on shipping submissions with a’I just spotted this bug/potential refactor in the peripheral code, could you just fix it?’.

In our last sprint I picked up a task to implement a set of integration tests to validate new filtering behaviour. I implemented them and found similar integration tests for a similar test case that had been written by a colleague of mine. In the process of writing my tests I found a bug in his integration tests, so I fixed the bug and submitted it as one code review. This took me a day. On that code review a senior engineer on my team saw that the other tests I had fixed was using an enum that was placed in a service layer package instead of a common shared package and asked me to move it ‘while I am at it’. Sure, sounds good, moving it. This started unravelling and the next thing I saw was four days of uncovering architectural pain points that didn’t behave incorrectly but needed some refactoring in the long term and had nothing to do with the original integration tests I was supposed to write. I asked whether I could get a ship it on the bug fix and my integration tests and create a new task to do this refactoring? Resounding silence and no ship its. Until I have a ship it on my code, my work is not done, so I can’t just walk away from it. This scenario brought the idea of ‘hostage work’ to mind.

Sure, I am a firm believer in the boy scout rule. I am also a firm believer in working on the most important stuff first. By holding out on a ship it, this senior engineer was forcing me to do internal improvements (on code he had written a few weeks prior, as commit history proved, by the way) that was not on the sprint board. This is a very effective way to introduce work by circumventing the democratic prioritisation process that makes SCRUM successful. What you are actually saying is that this refactoring work is more important than what was placed on the sprint board as well as what was left out of sprint due to limited capacity. This is not fair. There are mechanisms for introducing discovered work into sprint. If it is a thirty minute improvement the boy scout rule applies. For more substantial refactoring and improvements, even discovered bugs – create a separate fire story on the sprint and make the work visible to the team. This creates the opportunity for the importance of this work to be prioritised properly by the team and it also does not skew the perceived effort of the original work. The story I was working on had an actual effort of one day, including a boy scout bug fix. The effort reflected on the board leaned towards five days. In a team where we estimate effort by benchmarking it against known other work, this introduces additional risk. The team also runs the risk of not meeting the sprint commitment which places additional pressure in subsequent sprints because of additional work backlog, tighter deadlines and a negative reputation for delivery.

Another form of hostage work is a manager or senior engineer approaching individuals on sprint to ‘quickly just implement a fix’ or ‘provide an estimate’. The engineer can’t decline because of the perceived seniority of the asker, so they do it. This does not just take away the time spent on the effort from the sprint, but also bears the additional cost of introducing context switch thrashing, loss of knowledge sharing while the engineer is working rogue and implied prioritisation of asked task over prioritised sprint work.

A last form of hostage work is an engineer who go rogue on their own and defend it with ‘I know this is not on sprint, but this is a learning opportunity for me’ or ‘I have this pet project or agenda I want to push, so I am rather going to work on that than the priority established during sprint planning’. In effect you are leaving the (now compounded) responsibility of meeting sprint commitments to your colleagues. You lose the trust of your colleagues who now have to work harder to meet sprint commitments – effort eating into their capacity for learning and personal investments.

SCRUM only works when everyone on a team actually protects the sprint and commit to the process.

Setting up multiple versions of Java JDK on a Mac

A quickie but a goodie – setting up multiple versions of Java JDK on a Mac. 🙂

Install Homebrew and Cask

  • Install cask if necessary
    • ➜ ~ brew tap caskroom/cask
  • Ensure brew caskroom is up to date
    • ➜ ~ brew upgrade brew-cask

Install jenv

  • Install jenv
    • ➜ ~ echo 'eval "$(jenv init -)"' >> ~/.bash_profile
  • Always run jenv (choose one :-))
    • ➜ ~ echo 'eval "$(jenv init -)"' >> ~/.zshrc
    • ➜ ~ echo 'eval "$(jenv init -)"' >> ~/.bash_profile

Install Java JDK’s

  • Java JDK’s are installed on iOS in /Library/Java/JavaVirtualMachines/
    • ➜  ~  ls /Library/Java/JavaVirtualMachines/
  • If you are missing any, use brew to install JDK
    • ➜ ~ brew cask install caskroom/versions/java7

Add JDK’s to jenv

  • for each jdk:
    • ➜ ~ jenv add /Library/Java/JavaVirtualMachines/[…JDK Version..]/Contents/Home
  • Look for all known binaries and rebuild shims
    • ➜ ~ jenv rehash
  • ➜ ~ jenv versions

Women are better coders than men?

Social media was rife with references to the latest study (study by Cal Poly, San Luis Obispo, and North Carolina State University) indicating that women contributors on GitHub have a higher merge acceptance rate…if they don’t indicate their gender.

This triggered a lot of headlines like “Why women are probably the best coders on GitHub” and “Suck It Dudes. Science Proves Women Are Better Coders.

My first response to this was ‘sigh, are we attacking men again in the name of ‘war against STEM misogeny?’ – and started writing this post intending to criticize the original study. I then read the study, and found it well presented and factual.

In summary, they found:

  • Not taking into account their declared GitHub gender, female GitHub contributors had a 78.6% pull request acceptance rating against the corresponding 74.6% for male contributors
  • When taking into account only contributors who explicitly stated their gender, the female pull acceptance percentage drops to 62.5%

They raised several good questions to provide further context to the data:

  • Is there a small number of ‘insider’ female contributors who know their project really well, skewing the bias data?
  • Are women making more pull requests than needed?
  • Are women making smaller changes?
  • Does the average pull acceptance percentage increase over time for a female contributor, showing a small pool of female contributors who ‘made the initial cut’
  • Is a woman’s pull request accepted more often because she appears to be a

What I am not finding in this study is the intent being to determine which gender’s code contributions carry more merit. GitHub pull acceptance is much too naive to use as a metric to say ‘women has a higher pull acceptance rating, therefore their work has more merit’, and taking that away from this study subtracts from its value.

I am impressed with the study – not so much with what media did with it. I want to see less ‘hate the boys, the girls won the war, wha ha ha!’ and more ‘so gender does come into play in software development environments? How, why, and how can we leverage it to build better teams?’ I truly feel that a lot of my male colleagues are painted with a misogynist brush in a knee-jerk naive reaction to a more complex situation. Guys – you are still good developers, don’t worry!

Managing on-going learning

I am in a new role in a new environment, and I have to make a plan to organise all the niblets of things I learn. Learning is hard – especially when it is happening in practice instead of a formalised learning environment.

I think I currently rank in phase 2 of Noel Burch’s competence hierarchy – conscious incompetence in most of the work I currently do, which leads to a lot of anxiety and frustration. I keep going down rabbit holes of information which opens further tangents – every day you get home with severe mental fatigue – you even start dreaming about boxing, arranging and labeling things – I needed a system.

I am one of those people who needs to see the big picture to get a framework in which to ‘box’ new information, and I realised that making notes in a notebook is just too linear for the fragmented way I am currently gaining knowledge.

Typical of a software developer, I created version controlled projects in IntelliJ, organising the content using this mind map editor plugin. These mind map nodes allow you to create links to files, notes and images. This allows me to modify and add to my ‘big picture model’ as I go along. I can also fill in the gaps later, where I need to. On top of that, it is also searchable. Here is an example of my collapsed Ruby mind map:

ruby project.png

Now I just need a few more! Let’s see how this system works out 🙂

Scrum: Story Granularity

We have three ‘homes’ for stories in our Scrum process:

  1. Unprioritised backlog – all items go here initially
  2. Prioritised backlog – We drag the clearest and highest urgency items into this, and rank items in this backlog. This is like saying ‘the team will commit to this work’
  3. Sprint board – during sprint planning, the prioritised backlog is groomed and the top n (depending on your sprint velocity) story points are added to the next sprint.

Items created in the unprioritised backlog can be:

  1. Features
  2. Bugs
  3. Operational work
  4. Investigation work

Feel free to add a high level story on the unprioritised backlog, even if it is large and vague. The team might discuss it and realise that there are questions to be answered before a design decision or commitment can be made. A separate spike story can be created, added to the prioritised backlog and sprint. The outcome of this story might lead to the high level story being split into three more concrete ‘sub stories’ or minimal deliveries. The original placeholder story’s granularity was much higher than the resulting child stories, and it never makes it onto the sprint itself, but it was still valuable to keep in the unprioritised backlog.

It might be useful to think of stories as having generations and be open to ‘spawn children’ should a smaller viable minimum value be better suited.

‘Create the new underlying database tables with migrated data’ might be a story separate from ‘adding a report displaying XYZ’ for example. ‘Completing a client request, but with mocked out back end data’ could re-size a complex story without necessarily compromising on value added.

Remember to keep stories smaller rather than too big – I have seen a distorted velocity when a team didn’t define the minimum value added scope correctly and only 80% of the task points were completed. Since only completed stories are counted in the sprint velocity, you don’t get recognition for the 80% completed work. Initially it might feel more intuitive to cluster all related tasks under one story ‘because they belong together’. Lean towards chopping work up into absolutely minimum work required to deliver value and iterating on that. One ‘value’ parent story might necessitate various generations of child stories making it onto the sprint board.

Two special kinds of stories worth defining are:

  1. Fire stories: If a story/task is blocked by something unforeseen mid-sprint, it can be unblocked by a fire story. This is additional task points that are introduced to the sprint as a clearly defined ‘Fire Story’. This is not an excuse to introduce urgent unforeseen work into the sprint. It is only intended for use to unblock a specific story on the sprint.
  2. Spike stories: If insufficient information or understanding is available for a team to correctly define requirements, a spike story is created to answer the unknowns. The definition of done can be a requirement specification or a meeting to introduce the findings to the team.