Making an AI Explain Itself

The Problem

An AI tool that flagged problems but never explained them

Imagine you're a compliance professional preparing for an audit. You gather every piece of evidence your Auditor asked for, upload it, and run the check that's supposed to tell you whether you're ready or if something is missing.

The check says something is missing. However, it doesn't say what.

You can't submit and hope for the best. An incomplete submission can mean your company fails the audit. So, you email your Auditor but they can't figure it out from an email either, so now there's a call on the calendar and leadership is on it. One missing file has become three people's afternoon.

Every unclear First Pass result has sent a customer to their Auditor. Auditors absorbed that cost every time and often took the blame for the tool's lack of specificity.

The existing workflow included five steps: evidence upload, run First Pass, customer reviews First Pass results, customer submits evidence to audit, Auditor reviews.

The constraints I designed inside

• One Project Manager and six Engineers, no other Designers. Three months to understand and solve the problem.
• Leadership wanted First Pass fixed, because slow audits were cutting into profitability. The auditors wanted it deprecated. I needed Auditor buy in because they know the workflow better than anyone in the company and designing without them meant guessing.
• Previously I had been focused on the customer facing tool, so the Auditors didn't know me. Additionally, there was tension between Audit and Product that predated my role.
• This fix was urgent and required me to work within our design system's existing components as much as possible in order for Engineering to be able to build it in time.

My Approach

Finding the Real Problem

The initial problem leadership presented to me was to "fix the First Pass inaccuracy" with the diagnosis being that First Pass didn't know the customer's scope. The initial diagnosis only highlighted the first step to resolving the inaccuracy issue. Scope will tell the model which systems the audit covers, but it doesn't tell the model which systems already have evidence attached. Without that second piece, First Pass still can't name what's missing which was slowing down audits. The full solution must deliver a way to connect every file to the system it came from.

I proposed a second model designed to map the files to the source systems before handing off the mapping and audit's scope to First Pass. Our CEO named it "Scope Sort".

My Engineers were focused on usability and pushed back on this solution at first. If we automate the mapping would customers understand what was being mapped? What about when the mapping is wrong? Would customers know? Would they fix it? My Engineers pointed out potential issues of usability, potential for mistakes in the mappings, how to identify those mistakes, and how to fix them. I believe that a well-designed feature shouldn't need a person to explain it regardless of how complex the issue is.

How I Ran the Research

I interviewed six auditors, each working on various types of audits. In order to respect the Auditor's time, I presented my questions during an existing weekly meeting. I collaborated with Customer Success to run usability testing and analyzed how often First Pass was incorrect. Customer anecdotes revealed that First Pass was perceived as unhelpful every single time it was used.

Getting Auditors to work with me

Building trust with the Auditors was key to making meaningful progress towards a solution that would actually be adopted. Early on, showing a finished design read as presumptuous: a decision made without them, by a Product team they already distrusted. Once we built trust, I was able to receive feedback in the morning, iterate with Claude Design, and share a revised design that afternoon. By taking time to gain an understanding of their perspective I was able to improve our working relationship while gaining valuable insights.

Five findings during research that shaped what I built

1. Auditors are required to review every file by hand, so a detailed evidence view was noise to them rather than help.
2. Scope changes often and only auditors can edit it. Stale scope means wrong First Pass results regardless of mapping, so auditors needed help keeping it current.
3. Auditors needed an alert when evidence came from a system outside the audit, because only they can update scope.
4. Customers wanted to know which tags had been edited by a human and which were tagged by AI.
5. Multi-product customers couldn't use a single unified view, since they only knew their own product's requirements.

Mapping the flow before designing

I mapped out the full user flow and walked through it with the project's stakeholders before designing anything. This allowed us to confirm we were all aligned on what we were building. This became the reference point the team came back to when our priorities became unclear. I was also able to continue to update it as new problems or ideas surfaced.

Directions I Considered but Did Not Ship

• Customer tagging. The idea was to allow customers to tag files with the source system upon upload. Through customer interviews I discovered that often the user uploading the files was not aware of which system they came from. Additionally, users typically upload files in batches, so manually tagging each file would be cumbersome.
• Upload buckets. The idea was to give customers an upload bucket for each in-scope system. This also did not work for the same reason as stated above. Often the user uploading the files was not aware of which system they came from. This lead to an excessive amount of files being placed in a miscellaneous category that rendered the whole system as useless.
• Mapping folded into First Pass. This idea involved condensing two AI processes into one. The mapping itself would be shown inside the answer from First Pass. Usability testing and leadership both killed this idea. If the mapping was incorrect those errors would flow straight into the final result with no opportunity for the customer to catch it. When First Pass was already struggling to gain customers trust, one wrong tag became more reason to believe that First Pass had no idea what it was doing.
• Confidence Scores on Scope Sort Results. This was a design I had used on previous projects that had proved successful. Ultimately, we decided it wouldn't be useful for this project. When testing out the Scope Sort model on customers evidence we found that the model usually would either identify the system correctly or not be able to identify it at all. Either the file was legible or it wasn't.

The Solution

Mapping as its own reviewable step

With my revisions our customers can now see every in-scope system and where files are being mapped, including tags that they can edit before running First Pass. This direction highlighted all potential errors and allowed the people with the most contextual knowledge to come back and review the tagging.

The workflow now runs in seven steps: evidence upload, automatic system matching, customer reviews and/or corrects matching, run First Pass, customer reviews First Pass results, customer submits evidence to audit, Auditor reviews.

For customers: knowing what's missing

With the help of Scope Sort, First Pass now knows which systems are in scope and what evidence has already been uploaded to each. That context made First Pass accurate, and it made it specific. When something is missing, First Pass can name it.

• Automatic matching. Evidence is matched to the correlating system on upload.
• Itemized results. Customers see which in-scope systems still need evidence, and what's missing from each.
• Early mismatch alerts. If evidence maps outside the audit plan, customers hear about it before they submit.
• Editing of AI (Scope Sort) results. Evidence matching is editable to account for the times mapping may be wrong.

Customer emails to the Auditors and expensive calls for follow up are no longer the only way forward.

For Auditors: help keeping scope up to date

Auditors must review each file manually, therefore the customer-facing solution was not a fit. I did, however, find a way to utilize what we designed for the customers to also help Auditors keep scope up to date in order to keep First Pass results more accurate.

• Out-of-scope alerts. Auditors are now notified when a customer uploads evidence from a system that isn't in scope. They can update scope in just a few clicks
• Evidence counts by system. How many attachments each in-scope system has, and which have none. If an in scope system has no evidence uploaded this could be a sign to auditors that this system is no longer in scope.

Key Decisions

Key Decisions and Trade-offs

Shipped narrow instead of complete

Testing revealed that a unified view for multi-project customers lacked usability. My Project Manager and I held the feature back from multi-project customers in V1 rather than rush a fix or stall the release for everyone. What it cost: our multi-project customers, under 14% of audits, still don't have Scope Sort. They'll get a version built for them instead of one that technically included them.

The V2 view for multi-product customers.

Built less for Auditors than leadership wanted

Leadership wanted the customer view mirrored for the Auditors, but my interviews supported shipping a single mismatch alert. What it cost: a harder conversation with leadership, and I had to bring the Auditors' reasoning back rather than my own preference.

Settled for a temporary home for the alert

Auditors needed the alert as a notification, but our notification feature was in bad shape and they weren't using it. Another designer was building an Auditor dashboard at the time, so I worked with her to fit the alert into her design. It now appears on the dashboard as well as on the evidence review page. What it cost: it's a workaround. I documented that Auditor notifications need a real fix.

Refused to ship AI the user couldn't override

Some of the product team didn't think correction was necessary for V1. The model would be right most of the time, and the build was tight. I argued it wasn't a question of hit rate: a mapping the customer can't fix becomes a First Pass answer they can't trust, and untrustworthy answers were the entire problem. What it cost: less than I expected. It turned out cheap to build, but I'd have pushed for it either way.

Can they tell what evidence was tagged by Scope Sort vs what was tagged by a human.

We'd been marking AI output with a sparkle icon, but the icon slot on these pills was already carrying something more important: whether the system was in scope. Evidence from outside the audit is a red flag, and needs the icon more than AI attribution. So AI attribution moved to a purple-blue glow I'd already established elsewhere in the product that met accessibility standards, and the telescope, our existing scope icon, took the slot.

AI tag redesign

The Outcome

What this project demonstrates

That's the outcome I care about most. She isn't a security expert. She's the person who has to go ask for the right file, and now she can name it.

The numbers, with a caveat

First Pass adoption went from 20% to 38%. Audits in that window ran about 12% faster. No full audit cycle has completed since launch, so everything here is early. I can't attribute all of it to this feature, since other work shipped alongside it. The measure that matters more, how many customer questions reach Auditors, needs a full cycle before I'll trust it.

What hasn't shipped

V2 isn't built. Multi-product customers still don't have this feature, and the Auditor alert is still in a temporary spot. Both were deliberate calls and both are still open.

The trust deficit, still closing

Auditors had watched this tool fail every time for months and Product had a history with them before I arrived. That's not fixed. What did change is that several of the Auditors who wanted this project dead now work with me directly, which makes the next feature easier to build than this one was.

My Reflection

What I'd Do Differently

I was wrong about my idea for wrapping both layers of AI into one UI experience

Folding the mapping into First Pass was the direction I pushed hardest for. One AI moment, less to explain, fewer steps. What I was optimizing for was the experience when the model is right, and I'd stopped short of designing for the case where it isn't. Testing and leadership caught it before I did. It changed how I evaluate AI features now: I start from what the wrong answer costs the user, not from what the right answer feels like.

I ran my two stakeholder groups in parallel, and it cost me weeks

I'd take a position from leadership to the Auditors, get a different answer back, carry that to leadership, and repeat. Every loop was time I could have spent designing. I should have gotten both groups in the same room about the problem in week one. One conversation would have surfaced the disagreement immediately instead of one message at a time.

Next Steps

More Opportunities to Help Auditors Keep Scope Accurate

It's important to note that this entire feature hinges on scope being correct. We now flag potential scope issues/changes when a customer uploads evidence in order to keep scope up to date.

During research Auditors told me something I didn't expect: scope gets captured repeatedly over the course of an audit, through several different channels. Every time scope changes is a chance to catch a mismatch, and we're only using the last one.

The cost of catching a scope change late falls on the customer. They spend hours collecting and uploading evidence for a system that ended up not being part of their audit, or they miss the chance to automate evidence collection for a system that is part of their audit. Either way they find out after the work is done.

The next step is moving the check upstream, to catch scope changes earlier rather than only after evidence is uploaded.