For the ssh key, are you using your yubikey on multiple computers or just one? I just started looking at this but it seems like there is a bit of setup needed for each computer. I guess it might be worth it but would be interested to hear about others experiences.
It's very easy to setup on linux, but does require a fair bit of setup to get it to work with windows. I haven't set it up on OS X but I expect it to be about as easy as linux.
I've been SSH w/ yubikey key only for about 6 months now and haven't had any issues with it. I regularly move between multiple computers. Once I set it up on one computer I've never had it take more than 15 minutes to get up and running on any computer I've need it on (window or linux).
The only thing I'm really missing is the ability to log into my server from my phone. There was some talk of getting ConnectBot and Open-Keychain talking to each other to get this working but it appears to be stalled.
Yeah, it does require some setup on the ssh client machine. So far, I only set up my home laptop with it. It's probably not hard on the work machine, but different OS so I haven't tried.