First day of my research visit at Oxford! Many thanks to @zparolin.bsky.social for the generous invitation.

I’ll be presenting on simulating human behavior with LLMs on Jan 28 at the @nuffieldcollege.bsky.social Sociology Seminar.

If you’ll be around Oxford, feel free to reach out!

4/ Great to learn from your work @tsrauf.bsky.social, Jan Voelkel, Jamie Druckman, and @jeremyfreese.bsky.social !

3/ One aspect of this study I particularly liked was the data quality of the sample: selection did not depend on research outcomes, the samples were nationally representative, and careful vetting of experimental designs suggests that insignificant results are not attributable to quality issues

2/ Beyond sample size, researchers could reduce residual variance (and thus increase power) by controlling for pretreatment variables that are strongly correlated with the outcome (e.g., government trust and policy attitudes). Future research could assess the efficacy of this strategy systematically

Effect sizes in social science survey experiments are typically small, requiring large samples and budgets for sufficient statistical power. It gets even trickier because increasing n/$ yields only diminishing returns to statistical power. This study points to an important barrier to credibility👇 1/

We learned from and built on the terrific work of scholars who thought deeply about how to leverage LLMs for the social and behavioral sciences. Feedback welcome!

To simulate-then-validate, to validate-then-simulate, or to calibrate: that is the question. We discuss ways to simulate human responses to behavioral science experiments using LLMs and strategies to address their limitations. Check out our preprint!

I'm super happy & proud to be able to work with this rock star group of a team (and academic guests)! 🧠🧠🧠💫🙏 @goetheuni.bsky.social @infer-frankfurt.bsky.social

- and it's really nice to check out parts of Frankfurt that I hadn't been to as part of our team event

Thank you for linking us under this thread, Shira! I enjoyed reading the discussion. It's worth considering in more detail how Prediction-Powered Inference borrows from older ideas, such as the GREG estimate--and how PPI differs from, for instance, Bootstrap sampling that uses just one data source.

Check out this article on leveraging AI for conducting social science research!

Excited to continue learning about the latest #CSS at @ic2s2.bsky.social! I’ll be at the Social Prediction Session, presenting the mixed subjects design on combining human and LLM data in experiments. Paper with Michael Howes and @austin-van-loon.bsky.social. Come join us! doi.org/10.1177/0049...

New in Sociological Methods & Research: Soc PhD candidate @davidbroska.bsky.social, @austin-van-loon.bsky.social, & Michael Howes show how combining human subjects and large language models can yield precise estimates at low cost, with implications for scientific productivity
doi.org/10.1177/0049...

How can we leverage generative AI to advance social science methods and research? Daniel Karell and Thomas Davidson led a special issue in Sociological Methods & Research to find out. Special kudos to them! journals.sagepub.com/doi/10.1177/...

Mixed feelings about silicon subjects (LLM predictions of human behavior) as replacements for human subjects? Consider the mixed subjects design.
🚨Now published at Sociological Methods and Research🚨
doi.org/10.1177/0049...

Congratulations, Chagai! Wishing you the best—though we’ll miss the causal inference powerhouse at @pascl-stanford.bsky.social. Good luck!

An image of the title and abstract.

Happy to share my new paper with Cat Dang Ton and @eollion.bsky.social on how to use generative LLMs for extracting information from textual data (conditionally accepted at Sociological Methods & Research)

Here's a rundown..

osf.io/preprints/so...

🚨 ACCEPTED AT SMR 🚨
Confused by colleagues who seem to want to study LLMs instead of humans? Frustrated by skeptics (e.g., myself 8 months ago) who dismiss LLMs as a potential source of data on human behavior? Check out our paper for a new way forward: osf.io/j3bnt_v3/