If the private key is protected with a strong(!) passphrase that cannot realistically be recovered with a brute-force attack, it's fine to store it on a USB stick. Adding another factor like time-based one-time passwords makes this a fairly solid scheme. An attacker would have to get your USB stick and your passphrase and your phone.
As a more secure alternative to general-purpose USB sticks, consider a specialized hardware token like a YubiKey. The benefit is that the private key is kept in tamper-resistant storage and never revealed to the client PC. Instead, all cryptographic operations happen directly on the device. So even if the client PC is compromised, it cannot read the key itself. If you lose the token, then there also isn't a simple way for an attacker to obtain the key.
Smartcards have the same purpose and might be a bit more convenient to carry around, since they fit into the wallet. However, then of course each client PC needs to have a card reader, ideally with a pinpad -- or you have to bring your own.