The AI You Can See Isn't the Only One You Should Worry About
A few months ago, Laura Carter, PhD, a researcher and fellow WAIE+ fellow, put out a call in our community asking if anyone wanted to help draft a submission to the United Nations. The UN’s Committee on the Elimination of Discrimination Against Women, known as CEDAW, was finalizing a recommendation on gender stereotyping, and their draft included a section on AI. Laura is a researcher whose PhD focused on exactly this intersection, gender stereotyping and data-driven systems in public services, and she wanted people who could speak to how AI actually works in practice.
I said yes immediately. I’m not a human rights lawyer or a UN policy expert, but I recognized the problem she was describing from the inside. I spent several years as a clinical annotator at a healthcare AI company. Patients had consented to having their conversations analyzed as part of the work, but what they couldn’t see were the specific decisions happening downstream: which labels to apply, which responses to flag, which edge cases to escalate. Those decisions trained the model, the model shaped what the system learned to recognize, and the system went on to inform clinical decisions about real people, with my read of their words somewhere in the foundation of it.
I thought a lot about who was being simplified in that process. Whose complexity wasn’t surviving the translation into data. That experience is why, when Laura described the gap she was seeing in the CEDAW draft, it landed immediately. Laura and I submitted formal comments to the Committee on behalf of WAIE+. This is what we found, and what we asked them to fix.
The AI making headlines isn’t the only AI making consequential decisions about women’s lives.
Image Credit: UN Women
What CEDAW is, and Why it Matters Here
If you’re not familiar with the UN human rights system, here’s the short version. There are a lot of human rights treaties, and there are UN bodies whose job is to monitor how well countries are actually living up to them. The Committee on the Elimination of Discrimination Against Women oversees the main treaty on discrimination against women, and periodically it issues what are called General Recommendations, short documents that interpret specific provisions of the treaty and give countries guidance on what they’re actually obligated to do.
General Recommendation No. 41 is focused on gender stereotyping. The draft includes a section on AI and digitalization, which is genuinely significant. A major international human rights body naming AI as a site of gender-based harm is not nothing. The draft acknowledges that AI systems reproduce and reinforce gender stereotypes. It calls on governments to regulate. It’s a meaningful step.
But Laura and I both saw gaps that needed to be named, and named specifically, or the recommendation would be too vague to actually protect anyone.
The AI People Can See Isn’t the Only Problem
When most people think about AI and gender bias, they think about things like image generators that hypersexualize women, or chatbots that give different answers depending on whether you present as male or female. Those are real problems. But the CEDAW draft, like a lot of AI policy conversations, risks focusing its attention on the visible stuff while the more structurally embedded systems stay out of frame.
Think about the AI that decides whether your benefits application gets flagged for review. The algorithm that scores your likelihood of repaying a loan. The predictive system that child welfare services use to assess family risk. Most organizations maintain that a human remains ultimately responsible for the final decision. But the extent to which that human can meaningfully interpret an algorithmic recommendation varies enormously, and when automation bias takes hold, the distinction between recommendation and decision stops mattering in practice.
And here’s what makes this a gender issue specifically: these systems are trained on historical data. Data collected from interactions between women and institutions that were already discriminating against them. When you train a model on biased data, the model learns the bias. When you deploy that model in the same institutions, the bias gets encoded into the infrastructure, and it becomes harder to see, harder to challenge, and much harder to fix.
The AI Now Institute traced this problem directly in their 2019 report, Discriminating Systems: Gender, Race, and Power in AI. The report documents how the lack of diversity in AI workforces and the bias in AI systems are two versions of the same problem, showing how AI used in hiring, healthcare, and criminal justice reflects and reinforces existing patterns of gender and racial discrimination. When the people building these systems are predominantly white and male, and the training data reflects historical inequities, the systems that result tend to replicate and sometimes amplify those inequities, often in ways that are invisible to the people most affected by them. That report is now seven years old, but the systems it described are still running.
Three Gaps in the CEDAW Draft Worth Naming
Laura and I submitted formal comments pushing the Committee toward three specific improvements.
The first was about language. The draft uses the term “artificial intelligence” as if it refers to one thing. It doesn’t. Predictive analytics, algorithmic decision-making systems, and generative AI work differently, get built differently, and cause different kinds of harm. An image generator that produces non-consensual intimate imagery is not the same kind of tool as a welfare scoring algorithm. Lumping them together lets policymakers believe they’ve addressed both when they’ve really only thought about one. We pushed the Committee to name specific types of AI systems and require that governments be responsive to the specific forms of harm each one causes.
The second was about the public sector. The draft’s AI section risks focusing primarily on consumer-facing tools and non-state actors. But governments themselves are deploying AI in the delivery of public services, social services, healthcare, housing, and child welfare. The data these systems are trained on comes from communities that have historically been over-surveilled and under-served. The simplification that happens when a complex family situation gets reduced to a risk score is not neutral. It reflects and reinforces the same gender stereotypes the recommendation is trying to dismantle. We pushed the Committee to name this explicitly and to require pre-deployment gender impact assessments for any AI tool used in public service delivery.
The third was about specificity and accountability. The recommendations in paragraph 60 of the draft, the AI and digitalization section, call for regulatory frameworks, dialogue mechanisms, and participation initiatives. That sounds like a lot. But when you compare it to how specific the Committee got in the sections on education, health, and media, the AI section falls short. The education recommendations name specific obligations. The health recommendations name specific obligations, the AI section calls for conversation. Dialogue without accountability doesn’t protect anyone so we pushed for transparency requirements, mandatory gender impact assessments before deployment, gender-disaggregated outcome data published regularly, and accessible complaints procedures with real obligations to investigate and provide remedies.
Non-Consensual Synthetic Imagery Needs to be Named Directly
Generative AI has made one specific form of gender-based harm dramatically easier to produce at scale and that is non-consensual synthetic intimate imagery. Images and videos of real women, fabricated without their knowledge or consent, created and distributed to humiliate, silence, and harm.
Research published in 2024 across ten countries found this is a documented and growing problem, and the people most targeted are women and girls. The technology to create it has become cheap and accessible. The legal frameworks to address it are still catching up almost everywhere.
The CEDAW draft mentions gender-based violence in the digital space. But it doesn’t name this specific harm directly. We pushed the Committee to name non-consensual synthetic intimate imagery explicitly, and to require that governments put in place legislative and regulatory measures to prohibit its creation and distribution, with real reporting and removal mechanisms on platforms and defined timelines for response.
Naming matters. If you don’t say it, you can’t require governments to address it.
What Actual Accountability Would Look Like
Here’s what we asked for (translated out of treaty language into plain English):
Before any AI tool gets deployed in a public service, a government should have to assess whether it will discriminate against women. Not after the harm shows up in the data. Before. That’s a pre-deployment gender impact assessment, and it should be mandatory, not optional.
After deployment, governments should have to collect and publish data showing how their AI tools are actually affecting women. Not aggregated data that hides disparities, but gender-disaggregated data, broken down in ways that make it possible to see where the system is producing discriminatory outcomes.
When a woman is harmed by a discriminatory AI output in a public service, she should have somewhere to go. A real complaints mechanism, with a real obligation on the government to investigate and provide a remedy, including the possibility of suspending or revising the tool that caused the harm.
And the people who build, procure, audit, and regulate these systems should have mandatory, ongoing training on gender stereotypes and how they surface in AI systems. Not a one-time module. Regular, recurrent training with accountability for what they do with it.
None of this is radical. It’s the same standard of specificity the Committee already applies to education, health, and media, we’re just asking them to apply it to AI too.
Image Credit: Christina Animashaun/Vox
Why this Matters Right Now
General Recommendation No. 41 is still in draft and the Committee accepted public comments through May 2026. Laura and I submitted our formal comments on behalf of WAIE+ on May 11th. When the final General Recommendation comes out, it will be interesting to see if any of our recommendations get adopted.
I want to be honest about what a UN General Recommendation can and can’t do. It doesn’t create binding law on its own. But it shapes what governments understand themselves to be obligated to do under a treaty they’ve already signed. It sets the standard that advocates and civil society organizations use to hold governments accountable. And it puts language into international human rights law that can be cited, built on, and used.
The gap between the AI frameworks that exist on paper and the AI systems actually running in people’s lives is real and it is widening. The people most affected by that gap are disproportionately women, and disproportionately women facing intersecting forms of discrimination, on the basis of race, disability, immigration status, and more.
The full WAIE+ submission, co-authored with Laura Carter, is available here and here.
What You Can Do From Where You Are
You don’t have to be a human rights lawyer or a policy expert to engage with this.
Public comment processes exist at every level — international bodies, national governments, state legislatures, local agencies. Most people don’t know they can participate, or assume their input won’t count. It does. Formal submissions from individuals and civil society organizations shape how recommendations get written, how regulations get scoped, and what obligations end up on paper.
Start by looking up what’s open for comment in your own backyard. Is your state developing AI guidelines? Is your city procuring a new algorithmic tool for public services? Is a national agency seeking input on a proposed regulation? These processes are often public, often underpopulated with voices from affected communities, and almost always open to individual contributors as well as organizations.
Research what AI systems your local or national government is actually using. Many procurement decisions may be public record. Benefits assessment tools, child welfare algorithms, hiring platforms used by public employers, these systems often exist in plain sight, just outside the frame of public attention.
If you encounter a discriminatory output, document it. Name it. Report it wherever a mechanism exists, even an inadequate one. The paper trail matters.
None of this requires a credential. It requires paying attention to the systems already running in your community and being willing to name what you see.
References
Carter, L., & Kroot, K. (2026). Submission to the Committee on the Elimination of Discrimination against Women Draft (CEDAW) General Recommendation No. 41: Dismantling Gender Stereotypes and the Unequal Power Relations that Sustain Them. https://doi.org/10.5281/zenodo.20129821
Cook, R. J., & Cusack, S. (2011). Gender Stereotyping: T r ansnational Legal P erspectives BOOK REVIEW. Human Rights Quaterly, 243. https://doi.org/10.1163/ej.9789004175600.i-350
Draft general recommendation No. 41 on dismantling gender stereotypes and the unequal power relations that sustain them | OHCHR. (n.d.). Retrieved June 16, 2026, from https://www.ohchr.org/en/documents/general-comments-and-recommendations/draft-general-recommendation-no-41-dismantling
ICO. (2024, November 19). Task 5: Prepare implementers to deploy your AI system. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/explaining-decisions-made-with-artificial-intelligence/part-2-explaining-ai-in-practice/task-5-prepare/
Umbach, R., Henry, N., Beard, G. F., & Berryessa, C. M. (2024). Non-Consensual Synthetic Intimate Imagery: Prevalence, Attitudes, and Knowledge in 10 Countries. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, 1–20. https://doi.org/10.1145/3613904.3642382