Quality in the age of AI-accelerated delivery

At Point, every change Engineering ship lands somewhere in a homeowner's journey. 

That journey starts when they explore what their home equity could unlock and decide to apply. From there, an underwriter helps them move through review, an operations teammate guides them through closing and funding, and our servicing team supports them for years afterward. At every step, homeowners—and the people helping them—are counting on our software to do exactly what it says. That’s why quality has always been central to how we work.

What's changing is how we build it. AI now helps us write code at Point every day. It drafts changes, writes the tests, and runs them, and that part has become genuinely fast. But speed was never the goal. Soon the volume of code reaching production will outrun what careful, one-at-a-time review can keep up with, and at that point the bottleneck is trust itself: knowing which changes are safe to put in front of homeowners and the teams who serve them. That's a good problem to have. It just changes what our job is.

What we’ve already built

So start with what we actually built, because it is already how we work, every day, across our repositories. When a change comes in, AI now carries most of the test-strategy load.

1. It reads the change and its impact. It looks at what a pull request changes and works out how that ripples through the rest of the application — including where a change in one app quietly affects another.
2. It finds the gaps. The missing tests, the end-to-end holes, the cross-application seams — caught now, instead of discovered in production.
3. It ranks them by risk. The gaps that could actually hurt rise to the top, rather than sitting in one flat list.
4. It writes the tests to our standards. It generates Playwright tests in the conventions the whole team already agreed on — not a dozen personal styles. In the last six months, our end-to-end test suite has roughly doubled, from about 530 to nearly 1,100 end-to-end tests.
5. It runs them and works them to green. It runs each test in a self-contained environment and keeps working until it passes, so what lands has actually run.
6. The standards travel through one plugin. They live inside the skills and reach every repository through a single plugin, so we sharpen the standard once, and every repo picks it up.

The payoff is upstream. Before a change ever reaches the main branch, the whole team can already see its impact: the gaps it opens, the flows it touches, and the tests that now cover it. That early visibility is where the confidence to release comes from.

The numbers tell the story. Over the last six months, throughput climbed 92% — a near doubling of our shipping pace. Over the same period, incidents fell 12.7%. That combination — dramatically more output with measurably fewer failures — is the clearest signal we have that the changes we made to how we work are paying off.

The confidence to release comes from seeing a change’s impact before it lands, not after.

Knowing what we can automate

Finding the gaps is a genuine win, and we are proud of it. The harder, more useful call comes next: which of those gaps we can actually automate. A Playwright test only works if it can drive the application into the state the assertion needs to check. Some gaps cannot be reached that way, and some are blocked by a dependency that is not ready — setup that does not exist yet, or work another team still owes. 

Those the pipeline does not fake a test for. It calls the blocker out, by name, for a person to clear. The ones it can automate, it can carry all the way to green on its own — but only after a human has approved them. Nothing is written and merged on the machine’s say-so alone. That approval gate is where a person still owns the judgment, and it is what keeps the suite something the team actually trusts.

Who checks the AI?

There is a trap you fall into fast when AI writes the code, which is letting AI be the thing that checks it. A model reviewing its own work approves what it just produced, and even a second model shares the first one’s blind spots: the same missing business context, the same edge cases neither was told about. 

So we review AI-written changes with a panel of different model classes, not one, and we treat their agreement as a question rather than an answer. Where the risk is highest, a person still owns the call. AI can draft the code and draft the review. It cannot yet be the only thing standing between a change and the customer.

What we haven’t cracked yet

We are not finished, and we would rather say so. We do not have a real measure of trust yet. As throughput climbs, we watch bugs and hotfixes instead, which is blunt but honest: flat or trending down signals genuine velocity, whereas an uptick forces us to pause and investigate. 

To date, production issues continue to decline. Front-end testing is still partly manual. We cannot put a clean cost on each task. We do not let changes merge on their own. And a new kind of problem is arriving, because as more of what we ship is AI-powered, we have to judge probabilistic output, not just deterministic code, and that needs a different sort of check.

The gap we feel most is scale. As delivery speeds up, we cannot run every test every time, because it is too slow and too expensive. So the work ahead is about running less while trusting more: selecting the tests that matter for a given change, quarantining and root-causing the flaky ones, and clearing out duplicates and dead code as we go.

What this frees us to do

Clearing the routine work off our plates is the whole point of building it. When AI takes the mechanical load of writing and running tests, quality engineers get their time back for the work that actually needs a person.

• More exploratory testing. The curious, unscripted probing that turns up failures nobody thought to write a test for. Machines cover the cases we already know; people find the ones no one predicted.
• Time to build tools that make developers faster. Turning the workflows we fought to figure out into tooling the whole team can lean on.
• Real work on evals, alongside product. As more of what we ship is AI-powered, we help decide what “good enough” means for a probabilistic feature and build the data that checks it.

The machines take the repetitive load, and the people move up to the work that needs judgment.

Code got cheap. Trust is the work now. As our checks get stronger, we can hand the agents more, because we can show the work holds up before we ship it rather than hope it will. Earn trust, and the speed follows.