The remarkable advances in AI have given rise to a growing interest in AI-assisted decision support in domains ranging from medicine and drug-discovery, to criminal justice and education. The ultimate goal in AI-assisted decision support is to optimally combine the complementary strengths of humans and AI models to achieve greater outcomes than either can achieve on their own, in short human-AI complementarity. Achieving this goal though, has shown to be a major challenge as it typically requires humans to understand when they can rely on the AI model for their decision, which has shown to be highly non-trivial.

My research shows that it is possible to circumvent this challenge and achieve human-AI complementarity by proposing an alternative design of decision support systems. The key principle underpinning this design lies on adaptively controlling the level of human agency by using an AI model to narrow down the decisions a human can take to a subset. In this talk, I introduce this design in the context of independent and sequential decision-making tasks, where I present provably data-efficient algorithmic methods to identify the level of human agency under which humans maximize their performance in the decision-making task. Under this optimal level of human agency, my proposed design shows to achieve human-AI complementarity in practice, based on evaluation through two large-scale human studies with a total of more than 4,000 participants. I conclude the talk by discussing challenges and opportunities in human-AI complementarity that open up by AI models generating natural language, highlighting emerging avenues for future research.