There's no real problem with going
at your own pace, indeed some days it's nice to just dawdle and take in the scenery and smells, but it is worth looking to work on getting more comfortable with passing slower moving vehicles if only to put their possibly unpredictable driving behind you where it is of less hazard to you (as in they fiddle with the radio/sweets/shavers/satnavs and so on, brake unexpectedly when they see a suitable shop/picnic stop and all those kind of things.
Obviously if you are in fairly busy traffic it may not be worth passing, but if you gain a clear road ahead for a while to cruise along in your own manner then it's worth doing, for me I prefer to be able to roll on and off the throttle and join the corners up in a flowing, smooth manner rather than harsh braking and acceleration, with practice and a good view it's possible to flow past slower traffic with minimum disruption just by preparing to overtake, being in the right gear and at a suitable speed well before getting near the other vehicle and passing in a nice flowing path back to your side of the road ahead of them.
As you mention cars can often be doing 40/45 mph in 60 limits (and unfortunately they often keep to that speed in the 30 limits too! ) indeed I find it a bind to be stuck behind them in car and commercial vehicles with less chance to pass than on a bike.
Your bike is of a reasonable size to overtake safely with minimum disruption, you don't say what year/model of Fazer, I briefly rode a 2003 model and found it hard work, needing to be revved to get the best and I understand the newer ones were even more so, I found the Suzuki Bandit to have better spread of power and easier, but I was used to largish twin cylinder Yamaha TDM850 which has fair bit of grunt in each gear so less gearchanging again
Anyway, ride your own ride, but keep yourself open to new ideas and information on riding and decide for yourself hope that's some use.