An LLM-native psychometric instrument does not predict LLM behavior: Evidence across 25 models
A psychometric instrument designed for LLMs yields coherent, reliable self-reports that nonetheless fail to predict model behavior across 25 models.
Juan Manuel Contreras, Ph.D.
AI evaluation and data science leader, hands-on with model behavior, safety, and measurement.
Co-Founder & CEO of Aymara·Ph.D., Harvard·ex-Uber Applied Science
About
I build and lead teams that measure how AI systems behave, and I design the methodology, benchmarks, and infrastructure that determine when a system is ready to deploy in high-stakes settings.
I'm Co-Founder and CEO of Aymara, a bootstrapped AI safety evaluation company serving Fortune 50 enterprises and leading AI labs. Before this, I led applied science for Uber's legal and regulatory function, where my team's analyses were cited in court decisions and shaped regulatory outcomes across jurisdictions.
PhD in cognitive neuroscience from Harvard and AB in social psychology from Princeton.
Born in New York City and raised in Bolivia and the U.S., I run marathons, write fiction, and am starting a band with AI musicians.
Experience
Aymara
Co-Founder & CEO
2023–Present
Uber
Data Science Manager → Applied Science Manager I → Applied Science Manager II
2019–2023
Even (acquired by Walmart)
Head of Data Science
2018–2019
Capital One
Principal Data Scientist
2016–2018
White House Social & Behavioral Sciences Team
Associate Fellow
2015–2016
Democratic National Committee
Senior Data Scientist
2014–2015
Research
Selected publications on AI safety evaluation and model behavior.
Automated evaluation of gender bias across 13 large multimodal models
Automated evaluation of gender bias in image generation across 13 leading multimodal models.
Policy-grounded safety evaluation of 20 large language models
Framework and tooling for evaluating risk and responsibility of large language models across 10 real-world safety dimensions.
Writing
Essays and creative work on AI, cognition, and culture.
Why AI Won't Make Us Dumber: The Next Flynn Effect
The Flynn Effect — rising IQ scores across generations — suggests that cognitive tools make us smarter, not dumber. AI may be the next one.
Speaking
Podcast interviews and conference talks on AI safety, research, and technology.
Contact
Email me at jm.contreras.phd [at] gmail [dot] com or send me a message below.