Engineering excellence: strengthening Point’s execution

Over the past year, Point’s engineering team has undergone a transformation. We revamped nearly every aspect of our engineering workflow. I'd like to highlight our journey in one particular area: execution excellence. 

In search of execution excellence

Our main goal was to develop excellence within the engineering organization around how we execute our work. Point is at a critical juncture in its journey, and now, more than ever, we need to deliver new capabilities in a predictable and high-quality manner. 

‍

In partnership with our other engineering leaders and product management team, we set about providing more structure to ensure that we could consistently deliver maximum value to the business. 

A restructuring 

The journey started with the restructuring of our engineering team. We reorganized the team into three pillars, each targeting a specific user base: homeowners, internal operations, and investors. This meant that our teams were now set up with a new level of focus.

Enter Scrum

To manage each team’s work and establish a roadmap, we decided it was time to move to Scrum. Scrum is a framework for iterative product development, emphasizing collaboration, flexibility, and continuous improvement. For those unfamiliar with Scrum, it involves the following:  

• Identifying and prioritizing features in a product backlog
• Iterating on that backlog in a sprint lasting two weeks
• Daily stand-up meetings to track progress
• Reflecting on the sprint to identify improvements

‍

Before this change, our teams worked through a large backlog in a somewhat chaotic manner, resulting in delays and individualized work. With Scrum, we achieved more predictability by focusing on the work to be completed within the two-week sprint and reorienting teams toward accomplishing shared goals — instead of maximizing the amount of work in flight.

‍

Moving to Scrum was also a foundational part of our larger plan to build more predictability and accuracy within our engineering process and better serve the business. On top of measuring team velocity and estimating story points, we could move to a more organized system of quarterly and even yearly planning against our expected capacity.

‍

The transition to Scrum was not without its challenges. The team had to learn how to operate in a new way and get used to the rhythm of Scrum. It took us a little while, but it feels like second nature now across all the teams — and for just a little bit of growing pain, we get to share some pretty great metrics with everyone.

Establishing metrics

We focused our efforts on three execution metrics to start: capacity accuracy, sprint accuracy, and scope creep. I’ll explain each one in turn:

Capacity accuracy

This metric measures how good a team is at predicting its overall capacity for a sprint. It’s calculated by taking the number of points the team committed to at the start of the sprint and comparing it with how many points they completed at the end of the sprint – or the velocity for the sprint. 

‍

Over time, we expect teams to get very good at predicting their capacity accuracy and knowing how much they can complete within a sprint. Our target range at Point is 80-120% — to balance responsibility with a little bit of ambition.

‍

You may notice that we have an upper end of our target at 120%, so the obvious question is, why have a cap at all? Why not have teams go to 200%? Simply put, numbers like 200% can indicate a problem. In all likelihood, the team is underestimating their capacity, which is worth investigating if it becomes a pattern.

Sprint accuracy

The next metric, sprint accuracy, determines how good a team is at predicting the exact set of work they will deliver in a sprint. This is calculated by taking the points the team committed to and comparing them to the story points completed during that sprint. 

‍

Over time, we expect teams to build a solid track record around sprint accuracy as they develop a sense of focus and priority. Each team will be different in terms of their roadmap stability and what a realistic score here is, but our general target is 80-100%. I typically try to get teams to 80% specifically, as it’s a good balance between completing most of what you committed to and being ambitious.

Scope creep

Once you have calculated the previous two metrics, the scope creep is very easy to calculate and should give you a great idea of how much interrupting or urgent work is coming up for the team. We calculate scope creep as the points completed within a sprint that were not in the sprint when it started. Using this metric, you can see how much of your capacity is being taken up by other work — which can lead to interesting discussions that clarify the process. 

‍

For example, if a team has a high amount of scope creep within a sprint, it could be because extra work was discovered as a pre-requisite for a ticket in the sprint. That could lead to a productive conversation around project planning and thoroughness, or building a document that clarifies your team’s definition of ready. Another example may be manual work that the team has to do due to unprioritized tech debt, which could lead to some reprioritization now that the impact is seen.

Collective analysis

Between these metrics, you can get a lot of takeaways. For example, if a team has a good capacity accuracy score but low sprint accuracy and high scope creep, you can see that there's an issue with thrash on the team that should be discussed. 

‍

There will often be a significant swing between sprints, so focusing on the trends sprint-over-sprint is important. Typically, I have found these to be good metrics to be able to point to problems that occurred within a sprint and lead to especially productive sprint retrospective meetings. Over time, these conversations and the resulting changes or action items lead to their improvement.

A shift to project-based work

Another major shift we underwent was focusing more on larger projects rather than small ad-hoc pieces of value. It allowed us to make significant gains for the business by executing more complex work than was previously possible. 

‍

By building more execution rigor and organization, the teams can consider work they otherwise might have been too risk-averse to take on over clearly-defined small, quick wins. 

‍

The shift to project-based work also enabled us to do quarterly planning and build out a more robust view of a team's roadmap while having a reasonable data-informed rationale for what is and is not possible. This has been key for us to be able to meet even the largest needs of the business on a predictable schedule.

Positive results

Our efforts have yielded tangible results. We've gone from being overextended by our commitments to consistently delivering on our promises to the business. At the start of the year, we did not track how many of our commitments to the business we completed; then, we completed 27% of them, then 81%.

‍

During the last year, we've delivered automation for entire sections of our workflow, drastically improved customer engagement and education, increased responsiveness for our investors, and reduced the workload of some of our operations users by over 30%!

‍

This has not only strengthened our relationship with stakeholders but also boosted morale within the engineering team. Most importantly, it has given us a foundation upon which we can deliver user-impacting results sustainably.

Gratitude

I would be remiss if I didn’t give thanks to everyone who was involved in this effort; none of this would have been possible without the entire organization pulling in the same direction. I am grateful for the hard work and support of our talented engineers, product managers, designers, and quality engineers, who pushed through every bump in the road to see us through to the end of this transformation. Together, we can accomplish great things, and I feel profoundly fortunate to be the one who gets to write this post in celebration.

Looking ahead

While we've made significant progress, there's always room for improvement. We're continually refining our processes and aiming for greater efficiency. 

‍

Our next goal is to automate our metrics tracking and implement the DevOps Research and Assessment (DORA) metrics for further insights into our performance. If you aren't familiar with DORA, I would highly recommend the book Accelerate. The metrics track:

• Deployment frequency: How often code is deployed to production.
• Lead time for changes: The time it takes for a code change to go from commit to production.
• Mean time to recover (MTTR): The average time it takes to restore service after an incident.
• Change failure rate: The percentage of changes that result in a failure in production.

‍

We are excited because these metrics provide meaningful insights into the efficiency, reliability, and effectiveness of an organization's software delivery and operational processes.

‍

Thank you for taking the time to read about our journey. It's been an exciting year of growth, and I'm optimistic about what lies ahead in 2024.