See http://www.karan.org/blog/index.php/2009/08/25/multiple-ssh-private-keys

chown my_user_name ~/.ssh/config

And see http://help.github.com/multiple-ssh-keys/