The aim of this paper is to mount a philosophical challenge to the currently highly visible research and discourse on empathy. The notion of empathetic perspective-shifting-a conceptually demanding, high-level construal of empathy in humans that arguably captures the core meaning of the term-is criticized from the standpoint of a philosophy of normatively accountable agency. Empathy in this demanding sense fails to achieve a true understanding of the other and instead risks to impose the empathizer's self-constitutive agency upon the person empathized with. Attempts to 'simulate' human agency, or attempts to emulate its cognitive or emotional basis, will likely distort their target phenomena in profound ways. Thus, agency turns out to be empathy's blind spot. Elements of an alternative understanding of interpersonal relatedness are also discussed, focusing on aspects of 'interaction theory'. These might do some of the work that high-level constructs of empathy had been supposed to do without running into similar conceptual difficulties.