Becoming AI-native: What we tested, measured, and learned

Every company says they’re “doing AI” now. But far fewer can say what they actually learned. That gap—between activity and judgment—is the whole story.

We got caught up in the rush like everyone else. Tools arrived faster than we could test them. The pressure to move was real. But we stayed grounded with one shared understanding: we were learning and testing. We weren’t racing to a decision. Real results take time. They’re rarely visible overnight.

That patience kept us asking hard questions: Where’s the evidence? Are we measuring this? What problem does this actually solve?

We had no master plan. Just a few promises: test openly, build safety as we go, measure honestly, and only standardize once we had proof it works.

Engineering went first. The tools were mature. Feedback was fast. But our goal was never just an AI-native engineering team. It was an AI-native company.

For us, being AI-native means building the systems, judgment, and habits to evaluate AI continuously. It means putting AI into real workflows when the evidence supports it. It’s not about adopting every new model or automating every workflow. It’s about knowing what to test, measuring honestly, building safeguards, and having the discipline to walk away when the evidence isn’t there.

Start with learning, not standardization

In early 2025, we deliberately did not pick a winner. Engineers tested several leading tools in their actual work. They tested Cursor, Copilot, Claude Code, Codex, JetBrains AI, and Windsurf. The team measured: output quality, time saved, adoption rates, security, and their fitment into our existing systems.

The rule was simple: pilot for a set time window, then converge. Don’t let a dozen different tools sprawl across the company.

Govern alongside adoption

Enablement without guardrails turns pilots into problems. Safety was a first-class workstream. We did four things:

1. Built a secure sandbox — A separate AWS environment with SOC 2 controls.
2. Wrote security policy — Security and Compliance signed off. It covered secure AI, ML, and LLM use.
3. Set up an access workflow — Decisions happened within five business days. No delays.
4. Assessed every vendor — Third-party security reviews confirmed each vendor met our standards.

Guardrails didn’t slow us down. They made safe speed possible.

Weigh the threats, not just the capabilities

A capability that leaks data is a liability, not an asset.

Every AI vendor we use passed a formal security assessment. We checked for: SSO and MFA, RBAC (role-based access control), encryption, SOC 2, PCI, and ISO compliance, U.S. data residency, a guarantee that our inputs never train the vendor’s models, and proof that PII is redacted.

Some passed cleanly. Others passed “with caveats.” That’s real.

We also tracked emerging risks. We watched for: compromised third-party software packages, malware impersonating AI tools, and shadow AI (unauthorized AI use). Our security lead said it well: “If you aren’t paying for the product, you’re the product.”

What about secure agentic AI? This is still unfinished work. An agent that reads issues, PRs, and files, then acts on them can be turned into a confused deputy. A malicious comment could coerce it into acting with a developer’s full permissions. So we’re building toward: sandboxed, temporary agent environments; no direct push rights to protected branches; and a rule that no agent approves or merges its own work.

Invest in people

Technology doesn’t transform anything. People do.

We created a learning network: a #topic-exploring-ai channel where people shared experiments, AI Champions office hours for hands-on help, workshops and brown-bag discussions, and opportunities open to everyone, not just engineers.

This openness was key. When adoption could cross team boundaries, it spread fast.

Focus on outcomes, not activity

A widely cited MIT report said 95% of enterprise AI pilots fail. We didn’t dismiss this. We debated it. The lesson was clear: chasing demos that never reach real work is the real risk.

So we chose high-impact use cases over a wall of pilots. When we measured, the signals aligned:

• ~7% self-reported productivity gains (from anonymous engineer surveys)
• +24% commits per PR
• +43% lines per PR
• Cycle time down 40%+
• Release-candidate testing cut in half

We stayed honest about what we could and couldn’t claim. But the consistency across every signal is the point. All of them moved together.

Standardize deliberately

By late 2025, the evidence had piled up. The tools had also improved, unlocking new leverage: skills (reusable workflows), hooks (automation in the editor), and agents (autonomous workflows).

These moved the advantage from “pick your own tool” to “share what works.”

When we standardized on Claude Code in December 2025, adoption jumped. Usage rose 38% the next month. In a few months, adoption went from one-third of engineers to two-thirds to nearly universal.

Why did standardization help? Engineers could reuse configurations and workflows. They didn’t have to rebuild the same setup over and over. That shared learning is the real win.

A capability inflection

Sometimes a new model is a step change, not just an incremental upgrade.

Anthropic’s Opus release in February 2026 was one of these moments. It was the first model we found good enough to carry a whole ticket, not just assist. Work that a weaker model churned on all day, Opus finished in about an hour.

That speed is what made our agentic workflows viable.

Build, don’t just use

The clearest proof: Chaordia—our proprietary, homegrown repository of AI tooling.

What is Chaordia? A shared repository of approved AI workflows, reusable skills, hooks, and agents, governed like production code, and a place where teams can build on what’s already been tested.

Teams don’t start from scratch. They start from what works.

The name is deliberate. Chaordic blends order and chaos. That’s honest about where we are. AI stopped being something we used. It became something we built.

Beyond engineering

The pattern that worked in engineering now travels across the company:

• Post-Closing — AI reviews title policies
• Underwriting — AI verifies identity
• Customer service — A homeowner voice agent answers calls
• Security and compliance — Agent “virtual teammates” help the team
• Hiring — AI conducts baseline interviews to assess how teams work today and identify AI opportunities

It’s earlier and messier than the engineering story. But the pattern, not any single tool, is what makes an organization AI-native.

The journey ahead

We’ve come a long way—from scattered tools and low adoption toward a shared, homegrown toolchain and near-universal use. But the curve keeps climbing. Most of the work is still ahead of us.

Becoming AI-native was never a destination. It’s a muscle we keep building.

The companies that pull ahead won’t be the earliest adopters. They’ll be the ones that learned fastest. They’ll be the ones with the honesty to drop what didn’t work. That discipline started in engineering. It was never meant to stay there.