I discovered this in OpenBSD. There, by default, at the beginning of an X session, it asks for the SSH key passphraseif there is a key. It does this with x11-ssh-askpass. I installed it on both of my boxes (the SlackBuild of x11-ssh-askpass intended for Slackware 12.1 still works as of Slackware 14.0), and borrowed this from OpenBSD’s default ~/.xinitrc:

# ssh-agent (Inspired by and stolen from OpenBSD)
id1=$HOME/.ssh/identity
id2=$HOME/.ssh/id_dsa
id3=$HOME/.ssh/id_rsa
id4=$HOME/.ssh/id_ecdsa
if [ -x /usr/bin/ssh-agent ] && [ -f $id1 -o -f $id2 -o -f $id3 -o -f $id4 ];
then
	eval $(ssh-agent -s)
	ssh-add < /dev/null
fi

# Insert window-manager line here...

# Kill ssh-agent upon exiting
if [ "$SSH_AGENT_PID" ]; then
	ssh-add -D < /dev/null
	eval $(ssh-agent -s -k)
fi

…and then it worked.