Abstract
This paper explores the moral limits of relationships between users and advanced AI assistants, specifically which features of such relationships render them appropriate or inappropriate. We first consider a series of values including benefit, flourishing, autonomy and care that are characteristic of appropriate human interpersonal relationships. We use these values to guide an analysis of which features of user–AI assistant relationships are liable to give rise to harms, and then we discuss a series of risks and mitigations for such relationships. The risks that we explore are - (1) causing direct emotional and physical harm to users; (2) limiting opportunities for user personal development; (3) exploiting emotional dependence; and (4) generating material dependencies.
Lize Alberts
Doctoral Candidate | Research Fellow
DPhil candidate in Computer Science at the University of Oxford | Research Assistant at the Leverhulme Centre for the Future of Intelligence | Research Fellow at Stellenbosch University’s Unit for the Ethics of Technology