Sharp, even though I can relate to what you are saying since I do not run an AV scanner on my gaming computer. I do not advice or even suggest that others should do this as well. Why? Even though I know what I am doing and very careful, there are still ways to get infected without much if any user interaction.
That said, I do have various types of scanners installed even on my gaming system where performance is #1 for me, so I do not have the AV scanners run in the background at all times but time-to-time I update the definitions and have it scan the entire system when it's not in use.
After all, how can you say or think the system is clean? Not every malware causes a noticeable system performance hit which alerts most users! Most personal info transfering baddies are almost transparent as far as the system/application errors and lag goes!

I have multiples systems as well so even if my gaming PC gets hit despite my best efforts (and I always consider this possibility), I know I am prepared:
~ I have no critical, confidential info on that PC.
~ I make regular backups to an external drive and also sync the data with a data storage PC.
~ I do not use System Restore, however, I placed a shortcut for Erunt in a startup folder so with every bootup it does a full registry backup (I generally keep 3-4 backups, delete older ones as each backup could consume quite a bit of space).
~ I also have created a fully functional image of the system after Windows XP all sps + security patches installed and also have my needed apps installed and configured. So I can be up and running in as little as 45mins if a fully re-image is needed.

Remember, always consider giving advice that is applicable and proper for general, common PC user with the least amount of PC knowledge. You might know what you are doing and be virus-free but there are many users who could not or do not use the PC like you do, not to mention their lack of computer skill. Also most people use only 1 PC to do all their work!

~TL