I discovered this in OpenBSD. There, by default, at the beginning of an X session, it asks for the SSH key passphraseif there is a key. It does this with x11-ssh-askpass. I installed it on both of my boxes (the SlackBuild of x11-ssh-askpass intended for Slackware 12.1 still works as of Slackware 14.0), and borrowed this from OpenBSD’s default ~/.xinitrc:
…and then it worked.