An LLM-native psychometric instrument does not predict LLM behavior: Evidence across 25 models
A psychometric instrument designed for LLMs yields coherent, reliable self-reports that nonetheless fail to predict model behavior across 25 models.
Juan Manuel Contreras, Ph.D.
AI product leader and researcher.
I help organizations build AI that works in high-stakes, real-world environments—from enterprise safety evaluation to billion-dollar regulatory decisions.
About
I co-founded and serve as CEO of Aymara, an AI safety evaluation company that grew to serve Fortune 50 enterprise customers and leading AI labs—without any funding. Before that, I led legal applied science at Uber, where my team's work was cited in court decisions and influenced billion-dollar regulatory outcomes.
I have a PhD in cognitive neuroscience from Harvard and an AB from Princeton.
Born in New York City and raised in Bolivia and the U.S., I run marathons, practice yoga, and am starting a band with AI musicians.
Experience
Aymara
Co-Founder & CEO
2023–Present
Uber
Data Science Manager → Applied Science Manager I → Applied Science Manager II
2019–2023
Even (acquired by Walmart)
Head of Data Science
2018–2019
Capital One
Principal Data Scientist
2016–2018
White House Social & Behavioral Sciences Team
Associate Fellow
2015–2016
Democratic National Committee
Senior Data Scientist
2014–2015
Research & Writing
Research, essays, and creative work on AI, cognition, and culture.
Research
Automated evaluation of gender bias across 13 large multimodal models
Automated evaluation of gender bias in image generation across 13 leading multimodal models.
Automated safety evaluations across 20 large language models: The Aymara LLM Risk & Responsibility Matrix
Framework and tooling for evaluating risk and responsibility of large language models across 10 real-world safety dimensions.
Writing
Speaking
Podcast interviews and conference talks on AI safety, research, and technology.
Contact
Email me at jm.contreras.phd [at] gmail [dot] com or submit a message through my contact form.