Are you relying on OS X's Keychain to protect your SSH key passphrases? You shouldn't. (The "plain" ssh-agent is fine, though.)

To be continued!