Are You Working on Something Necessary? (A Story About DDQs, Audits, and Getting It Wrong First)

Are You Working on Something Necessary? (A Story About DDQs, Audits, and Getting It Wrong First) 

By Trey Reeme, Trabian's Chief Operating Officer

Early in 2025, I made a huge mistake. 

I was buried in due diligence questionnaires (DDQs) from customers while also deep in SOC 2 fieldwork. It was unsustainable. 

So I made what felt like a rational decision: I told customers we would start charging hourly for DDQ completion. 

That didn’t go over very well, as you can imagine. 

And it forced me to rethink the problem entirely.

The Necessary Question

At the AFT Spring Summit in Nashville, Columbia professor and New York Times Chief Data Scientist Chris Wiggins delivered the second day's technology keynote. For those unfamiliar, AFT is the Association for Financial Technology, and it’s where us fintech vendors who serve the ecosystem come together to talk about what’s real and what’s next. Trabian’s been part of this community for years. It’s one of the most important gatherings in our industry, and the technology keynote is named for Brandon Dewitt, MX’s co-founder and CTO, who we lost in 2021. Wiggins honored Brandon’s legacy well.

Early in his talk, Wiggins laid out a framework that stuck with me. While explaining what makes AI effective versus what makes it unreliable, he stated in the simplest terms:

“If you’re asking AI to do something that is well represented in the training set, you’re gonna have a good time. If you are asking AI to do something that is not in the training set, you’re gonna have a bad time.“

In a room full of fintech leaders who are all, to varying degrees, trying to figure out where AI actually fits in their organizations, that line landed hard. Then Wiggins followed by relaying a quote from the man for whom the keynote was named:

“Are you working on something that’s necessary?“

I sat with that for a while.

A couple weeks ago, I wrote a post about building an app with AI in 30 minutes. It resonated far more than I expected. But sitting in that keynote, I realized I’d told the exciting story. Now I want to talk about what I’m working on that’s necessary for our business.

What Compliance Actually Looks Like

Today, Trabian serves roughly 200 banks, credit unions, and fintechs. As COO, one of my key responsibilities is compliance. That means I run point on our SOC 2 audits. It means I frequently personally complete DDQs from prospects and existing clients who need to understand our security posture before they’ll do business with us. It means vendor risk reviews, evidence gathering, policy documentation, and audit preparation.

This work is, well, tedious. It doesn’t make for a good demo. Talking about clean audits won’t impress anyone at a dinner party. But this work is what keeps our company qualified to serve regulated institutions. It's what keeps our clients’ auditors satisfied. It’s what keeps the lights on.

It’s necessary, and in this audit cycle, I brought in AI as a teammate to help me get it right.

The Problem With Being Human

Here’s the thing about compliance evidence gathering: the hard part isn’t knowing the answers. I know our policies. I know our infrastructure. I’ve written half the documentation myself.

The hard part is not missing anything, particularly when your organization just went through two major transactions in the same year: an ownership buyback to go private again, and a platform acquisition. Systems change. Policies update. Evidence that was valid six months ago may not reflect the company that exists today.

This is not a skills problem. This is a pattern-matching-at-scale problem. And as Wiggins put it in his keynote, pattern matching at scale is exactly where AI excels.

“Code is a very small set of problems,“ Wiggins said. “It turns out to be a really good problem for what AI is, which is pattern matching at scale.“

Audit evidence gathering is the same kind of problem. The frameworks are public. The control mappings are standardized. The evidence types are finite. It is, to use Wiggins’ framework, well represented in the training set.
 

What AI Actually Did

Our SOC 2 audit starts with a workbook from our auditor, listing our controls, each requiring specific evidence files. The evidence lives across dozens of systems, platforms, and tools. If you’ve been through a SOC 2 audit, you know. It’s a scavenger hunt across every system your company touches, and the stakes aren’t trivial.

AI is really good at scavenger hunts.

I used Claude Code — Anthropic’s command-line tool — as my operational backbone. Together we built command-line interfaces that let me search email, internal communications systems, and other SaaS tools without leaving the terminal. We organized our compliance artifacts library — including SOC 2 reports and evidences from previous audit periods. We built an evidence gathering skill that maps each control to the specific systems and search queries needed to find its evidence. We built gap analysis documents that track what’s complete, what’s pending, and what’s missing.

By the end of a couple focused sessions, we had evidence files covering all controls — organized, cross-referenced, and ready for the auditor to proceed with sampling.

What AI Didn’t Do

If you work in financial services and you’re reading this, I know what you’re thinking. You’re thinking about security. You’re thinking about access controls. You’re thinking about whether I just handed an AI system the keys and told it to be home by midnight.

I didn’t.

It was a copilot, not an autopilot.

This is the part that I think people miss when they talk about AI in regulated environments. The question isn’t whether AI should touch sensitive data. The question is how you design the workflows so that AI handles the orchestration and pattern matching, while a human retains control over every security boundary.

Why AI Is Actually Better at This Than I Am

AI didn’t just save me time on this. It was more accurate than I would have been.

It didn’t forget that C15 requires evidence from three different systems. It didn’t accidentally reference a 2024 board resolution when the audit period called for 2025. It didn’t lose track of which of the controls still had evidence gaps. And it flagged evidence I hadn’t gathered yet — that C17 still needed infrastructure management screenshots, that C32 was missing uptime monitoring evidence, etc.

This makes sense when you think about it through Wiggins’ framework. SOC 2 control mappings are standardized. The relationship between a control requirement and its evidence type is well-documented. The universe of possible artifacts is finite and well-known. This is exactly the kind of structured, pattern-dense, well-represented-in-the-training-set work where AI outperforms human recall.

I’m not saying AI understands our security posture better than I do. It doesn’t. It can’t explain to an auditor why we made a particular architectural decision, or how our incident response process actually works in practice. That’s judgment. That’s experience. That’s mine.

But for the mechanical act of mapping controls to evidence across systems, keeping track of what’s been gathered and what hasn’t, naming files consistently, and flagging gaps?

AI is more accurate than I am.

Wiggins said it plainly, “Just because you’ve used a technology does not mean that the responsibility has been delegated to Sam Altman.“ He’s right. The responsibility is still mine. My name is on the engagement. I review every artifact before it goes to the auditor. I answer every follow-up question.

The buck literally still stops with me.

The DDQ Pattern

Due diligence questionnaires are the same story, amplified.

If you’re a fintech vendor serving financial institutions, you fill out DDQs constantly. Prospects send them before signing a contract. Existing clients send updated versions annually. Security teams, procurement teams, risk committees: they all want to understand your controls, your infrastructure, your policies, your incident response procedures.

The questions are nearly identical from one DDQ to the next. “Describe your access control policies.” “How do you handle data encryption at rest and in transit?” “When was your business continuity plan last tested?” “Do you maintain SOC 2 certification?”

We have a compliance artifacts folder with our past completed questionnaires, policy documents, and infrastructure documentation. The answers exist. The work is matching the question to the right answer, adapting it for the specific format, and making sure nothing’s changed since the last time we answered it.

That’s pattern matching. The control principles are “in the training set” and the docs are in the context.

And yet, in most organizations, including ours until recently, the process was a human being (yours truly) opening a spreadsheet, reading each question, searching through a folder of prior responses. Then comes the copying and pasting, tweaking, checking to make sure they didn’t reference an outdated policy, and finally re-reading the question and answer to make sure I didn’t miss a nuance in how the question was worded differently this time.

Wiggins referenced a concept from data scientist Monica Rogatti: the AI hierarchy of needs. Her argument, from a 2017 blog post, was that you can’t just “sprinkle AI” on your company and expect results. You need data fundamentals first: collection, instrumentation, logging, observability. Then you need the craft of business intelligence. Then analytics. Then optimization. And only then are you ready for AI.

The insight for compliance work is that the pyramid is already built. The data is structured. The frameworks are standardized. The evidence types are documented. The questions have been asked and answered thousands of times across the industry. Compliance is one of the rare domains where the fundamentals are already in place, waiting for AI to work on top of them.

Wiggins talked about the Gartner hype cycle and how everyone in a room is at a different point on this curve at the same time with respect to AI. Some people are still at the peak of inflated expectations, convinced AI is going to transform everything overnight. Some are in the trough of disillusionment, having tried a chatbot that hallucinated case law or generated a customer email with fabricated account details.

Meanwhile, the plateau of productivity — the place where AI actually delivers measurable, reliable value — is sitting quietly in the back office. It’s in compliance, in audit preparation, in evidence gathering, and in questionnaire response.

It’s not just me noticing this. Dimitri Masin, CEO of Gradient Labs, told Julie Mann, Senior Research Analyst, at Finovate: “With traditional automation, you just can’t automate many of the things banks need to do — they require judgment and nuance. Now, with advances in AI, you can automate those messy processes that were only doable by humans before.” He’s talking about work that was too nuanced for rule-based automation but is structured enough for AI to handle the pattern-matching that surrounds it.

The boring stuff.

The necessary stuff.

The Regulated Environment Question

Wiggins made a comment during his keynote that I think deserves more attention. He was talking about vibe coding — the practice of describing what you want in plain English and having AI generate working code — and his general proposition was: “I’m not in a highly regulated industry with a bunch of information security concerns... which is why you may wonder, is vibe coding really going to take over things or not?“

He’s right to flag that. And the answer, I think, is nuanced.

Can you vibe code a customer-facing banking application and deploy it to production? No. Absolutely not. Not without extensive security review, penetration testing, compliance validation, and human oversight at every stage.

But can you use AI — responsibly, with proper boundaries — to orchestrate the operational work that keeps a regulated institution compliant?

Yes. I just did.

The key is workflow design. You don’t hand AI the keys and walk away. You design a process where AI handles what it’s good at — navigation, pattern matching, recall, organization, gap identification — and the human handles what requires judgment, accountability, and control over security boundaries.

Andrej Karpathy, via Wiggins, said, “the hottest new programming language is English.“ In compliance, the hottest new workflow language is also English. I can describe in plain language what evidence I need, from which system, for which control. AI can execute that workflow. And I can verify every output before it leaves my hands.

That’s not vibe coding. That’s operational discipline, augmented.

Earlier today a DDQ that took me eight hours across a couple weeks last year to turn around took less than one.
 

The Question for Your Institution

If you lead compliance, technology, or operations at a financial institution or fintech, consider this:

How much time does your team spend each audit or exam cycle gathering and organizing evidence?

When was the last time a human error in evidence mapping — a wrong date, a missing artifact, an outdated policy reference — caused an audit finding or delayed a deal?

How many systems get touched by your team to gather evidence for a single control or question? Is the mapping documented anywhere, or does it live in someone’s head?

AI can build the app in 30 minutes. It can also make audits and DDQs actually manageable.

One of those gets cool points with other vibe coders. The other one keeps your institution in business.

Early last year, I thought the answer was to charge for the work.

It turns out the answer was to rethink how the work gets done.

Are you working on something necessary?