When AI Gets It Wrong: The Risk to NGO Clients
Somewhere in a nonprofit office, between two meetings, a social worker opens an AI tool to write up a monthly donor report a little faster. She types in a client's name, a few details about the case and asks for a clean, tidy summary. The tool answers in seconds - fluent, confident, grammatically perfect. She copies the text, pastes it into the report, hits send. The whole thing takes less than two minutes.
Nobody in that office will suspect anything went wrong that day. The report looks professional. The tone is right. The sentences read well. The problem is that one detail in that report - a date, a number, someone else's name - doesn't exist anywhere in the original file. The tool made it up, quietly, with no warning, the same way it makes up thousands of similar details every day, all over the world. And right there, in the silence between the mistake and its discovery, is where this story begins.
Why This Matters More for Nonprofits Than for Businesses
When a bank gets a customer's data wrong, that customer has options. They can switch cards, close the account, move to a competitor, hire a lawyer if they can afford one. The market, for all its flaws, offers some kind of way out.
When a nonprofit gets it wrong with the data of a domestic violence survivor, a refugee, or a child in a support program, that kind of way out rarely exists. People who rely on NGO services often have no other provider in their city, no resources to fight for their rights, and in many cases don't even know a mistake was made until the fallout shows up somewhere else - in their safety, their legal status, their relationship with the authorities.
Trust, in this sector, is rarely something built over years and lost in an instant - except when it is. An organization that loses a community's trust rarely wins it back quickly, no matter how good its intentions were. That's why AI mistakes aren't some minor technical footnote for nonprofits. They go straight to the reason these organizations exist in the first place.
What makes this even more urgent is how fast things are moving. What was, just two or three years ago, a topic for conferences and theoretical debates is now part of daily routine at most organizations - the tools crept in through the back door, through individual staff members trying to save time, long before any policy caught up with them.
The Kinds of Mistakes AI Tools Actually Make
When people picture "AI mistakes," they usually picture something dramatic - a major hack, headlines, millions of leaked records. The reality of day-to-day nonprofit work looks a lot quieter, and precisely because of that, more dangerous. The mistakes that happen most often are small, almost boring, and that's exactly why they go unnoticed.
The first type is making things up. When the tool hits a gap in its information, it would rather fill that gap with a plausible guess than admit it doesn't know. The second is mixing up cases - when several different clients come up in one session, the tool can accidentally attach one person's details to another, blending two stories into one. The third is oversharing context - the tool answers a seemingly harmless question by revealing information nobody asked for. The fourth is misjudging urgency - the tool rates a client's situation as less serious than it actually is, or the other way around, and that judgment, if nobody double-checks it, ends up shaping how the team responds.
None of these mistakes require bad intent, a hacker, or a sophisticated attack. They happen during ordinary, routine work, precisely because the tool gets used every day, quickly, without much thought - the same way people use a calculator or a search engine.
Hallucinations: When AI Invents a Fact About a Client or Case
In language that's become part of everyday conversation in just a few years, this is called a hallucination - the moment an AI model states something untrue with exactly the same confident tone it uses for facts it actually got right. The model isn't lying, not in the human sense. It has no intention to deceive. It's statistically predicting the most likely next piece of text, and sometimes the most likely continuation of a sentence just happens to be false.
In client work, this often shows up when summarizing a long case history. Someone asks the tool for a quick overview of months of work with a family, and the tool, trying to make the summary sound complete and coherent, adds in a date that doesn't match any real meeting, or quotes something the client never actually said. If that summary gets passed along without a second look - in a report to a donor, in an email to a partner organization, even shown directly to the client - the mistake stops being a technical quirk and becomes part of the official record of someone's life.
The risk is highest exactly when time pressure is highest - in the last hours before a grant report is due, in rushed communication with several institutions at once, when the temptation to accept an AI answer without checking it is strongest precisely because there's the least time to check.
Data Leaks: How Client Information Ends Up Somewhere It Shouldn't
When people imagine a "data leak," they usually picture a movie scene - a hacker in a dark room, lines of code scrolling down a screen, a system break-in. The reality, in most nonprofit cases, is a lot more mundane and a lot less dramatic.
The most common scenario looks like this: a staff member, with the best intentions of saving time, types a client's name, address, or health details into a free, publicly available AI tool, just looking for help with a translation or a report. That tool, under its default, rarely-read terms of service, stores whatever gets typed in and sometimes uses it to "improve" the model further down the line. At that moment, the data leaves the organization's control - not because anyone meant harm, but because nobody stopped to read the fine print before hitting enter.
That's exactly why, for any work involving client data, paid business-tier models should be treated as mandatory - the kind that come with a contract explicitly ruling out the use of submitted data for training, and that guarantee information stays inside the organization instead of ending up in the training set for the next version of the model.
The problem gets worse in organizations that don't have a clear, written rule about which tools are allowed for which kind of content. Without that rule, every staff member makes the call on their own, in the moment, based on their own judgment - and different people, quite reasonably, judge risk differently.
Algorithmic Bias and Vulnerable Client Groups
AI models don't come from nowhere. They're trained on huge amounts of text written by people, and along with grammar and style, they absorb the biases baked into that text - often invisible, often unconscious, but real. For organizations working with refugees, survivors of violence, children, or minority communities, this isn't some abstract academic footnote. It's a concrete operational risk.
In practice, that means a tool can, without any awareness of its own error, apply stereotypical assumptions when automatically categorizing a case. It might interpret how serious a situation is differently depending on where a name comes from or what language the original text was written in. It might suggest wording that sounds neutral to a developer reading it in Silicon Valley but comes across as stigmatizing to the community the text is actually about.
This kind of mistake rarely gets caught right away, because bias almost never shows up as an obvious, glaring error. It shows up as a subtle shift in tone, in what gets prioritized, in the nuance of a recommendation - exactly the kind of difference that's hardest to spot without someone deliberately watching for it.
Protecting Client Data: What to Check Before Adopting a Tool
Before any AI tool becomes part of an organization's daily work, there are a few questions worth getting clear answers to - ideally in writing, before signing any contract or even before the first free sign-up:
- Does the vendor store the data users enter, and if so, for how long?
- Is that data used for further training or improving the model?
- Where are the servers physically located, and where is data processed and stored?
- Is there a formal contract that clearly defines confidentiality - not just generic terms of service written for an individual user?
- Can the organization request full deletion of data on demand, and how fast is that request actually honored?
If a vendor's representative can't answer these five questions clearly, concretely, and in writing, that vagueness is itself a warning sign - maybe reason enough to keep looking elsewhere.
GDPR and Local Data Protection Law: What Nonprofits Need to Know
For organizations operating in Europe or working with European donors, GDPR remains the core legal framework, but AI tools have opened up a gray area that legislation, written before generative AI took off, didn't fully anticipate. The central question is simple: does entering a client's data into an AI tool count as "processing personal data" in the legal sense? In practice, the answer is almost always yes - which means the same rules apply as for any other kind of data processing: a clear legal basis, a defined purpose, minimizing how much data gets shared, and a set retention period.
Things get more complicated for organizations funded from multiple sources at once - the EU, American foundations, local governments. Each of these donors often comes with its own additional data protection requirements, sometimes stricter than the legal minimum. The takeaway: simply "following the law" isn't always enough. Organizations also need to honor the contractual obligations tied to each individual donor, keeping in mind those requirements can differ from project to project.
Who's Responsible When the AI Gets It Wrong
The question of responsibility rarely has a simple, single answer, but a few principles hold up almost everywhere, regardless of jurisdiction or the specific situation.
Responsibility to the client and under the law stays, in practice, with the organization - regardless of which tool actually made the mistake. A contract with a software vendor might limit the vendor's own financial liability, but it rarely lets the organization completely off the hook toward its own clients. Individual staff members, on the other hand, rarely bear personal responsibility - except in cases of clear misuse or knowingly breaking a clearly defined, written policy.
The practical takeaway from this legal landscape is simple, if uncomfortable: an organization can't hide behind the claim that a tool made the mistake, not a person. From the client's point of view, the law's point of view, and even the public's point of view, the tool is treated as an extension of the organization - nothing more, nothing less.
What an Incident Actually Looks Like - A Scenario and the Response
Let's go back, for a moment, to the scene at the start of this piece. The social worker sent her report. A few days later, a colleague preparing the next report notices something that doesn't add up - a detail that doesn't match any meeting notes. She checks, compares, and realizes: somewhere in the summarizing process, the AI tool merged two separate stories into one, and that merged version is now sitting in an official document already sent to a donor.
A calm, level-headed response from an organization in this kind of moment usually follows a few clear steps. First, figure out exactly what was sent, to whom, and in what form. Then contact the recipient with a clear, professional request to delete or replace the incorrect document. At the same time, document the incident internally - without rushing to find someone to blame before actually understanding what caused it. Check whether there's a formal obligation to report this kind of incident under the law or under a specific donor contract. Finally, and most importantly, make a concrete, measurable change to the process so the same mistake can't happen the same way again.
Organizations that have a plan ready for moments like this - even a short, simple one - respond faster, stay calmer, and do far less damage to community trust than organizations facing this question for the first time in the middle of a crisis.
Free AI Tools vs. Tools With a Data Confidentiality Agreement
At first glance, the free, publicly available version of a popular AI tool and its paid business version look almost identical - same interface, same text box, same friendly tone in the answers. The difference hiding underneath that surface, though, can be decisive.
Business, or "enterprise," versions of these tools usually come with a formal contract that explicitly rules out using submitted data to train the model further, clearly defines how long information is kept, and gives the organization the right to review and verify how its data is handled. The free version of that same tool, in most cases, simply doesn't offer any of those guarantees.
For an organization handling sensitive client data every day, the difference between these two versions isn't about convenience or extra features - it's the line between acceptable and unacceptable risk. The problem is that the free version, in the moment, works "well enough" - the text sounds the same, the answer arrives just as fast - so many teams never actually get around to the serious conversation about whether they should switch to a paid, contractually protected alternative.
The Minimum Safeguards Every Organization Should Have
Regardless of budget size or team size, there's a set of steps available to almost any organization, and none of them require major spending - just a decision to take the issue seriously.
The first step is a written policy that clearly states which tools are allowed for which kind of data - short, easy to understand, available to everyone, not buried in some document nobody reads. The second is a clear rule that identifying client information - names, addresses, contact details - never goes into free, public AI tools, no exceptions. The third is naming a specific person or small team responsible for approving any new tool before it goes into regular use, instead of letting tools sneak in from the bottom up. The fourth is regular, short training - not a one-time session, but an ongoing habit, since both the tools and the risks around them change faster than almost any other technology the organization has dealt with. The fifth, and maybe the most important, is having a simple internal channel for reporting a suspected mistake - one that doesn't trigger fear of punishment, but encourages people to speak up quickly and openly when something goes wrong.
None of these five steps require a big budget. Together, they require something rarer and more valuable than money: the decision to deal with this before an incident happens, not after.
Training Your Team to Notice When Something's Off
In the end, most of the real protection here doesn't come from technology, licenses, or contracts - it comes from people who know what to look for and when to be suspicious. A well-trained team learns to ask itself a few simple questions, almost automatically, before accepting any AI answer as final and correct.
Can I check this against the original document, meeting notes, or conversation? Does this answer sound "too smooth" and tidy for information that, by its nature, should be messy and full of nuance? Did I just share, in a rush, something I shouldn't have?
A culture where it's completely fine to say "I'm not sure, let me check first" - and to say it out loud, in front of colleagues, without worrying it'll be read as weakness - is worth more, in the end, than any piece of software money can buy. An AI tool should stay exactly what it is by nature: an assistant that speeds up the work, not an authority whose answers get taken at face value. That one small shift in attitude protects both the clients whose lives the organization touches and the organization itself, whose standing in the community depends on how carefully it handles other people's stories.
Data sources used in this article: Identity Theft Resource Center (H1 2026 Data Breach Report), IBM Cost of a Data Breach Report 2026, the Virtuous / Fundraising.AI report on AI adoption in the nonprofit sector (2026).
