Yes, it is something "baddies" would know to try... but most likely wouldn't.
Your passwords are at risk from:
- Using an easy password like "1234"
- Reusing the same password from a site where it got leaked
- Capturing your password manager's master password
- Somehow else accessing the decrypted password database
- Directly capturing whatever you type in the password field
- Drugging and torturing you until you tell the password
Your strategy only enhances your security in case 4, where someone would somehow manage to get your unique site password without compromising your system.
Otherwise, you're compromised with or without it: in cases 1 and 2, it's your fault; cases 3 and 5 are basically the same case, you've got your system compromised and the attacker can capture whatever you type; in case 6, you're in bigger trouble already.
The likelihood of case 4, is relatively low, since the other cases are easier and/or more effective.