As a drummer i would say you can go either way with sizes. In my opinion it's about tuning, skins and (no)dampening. Big sizes don't always say it all. I've have 18" bassdrum that sounds huge when it has to and punchy or boomy when that's required. Considering toms just take care you have a 12" and a floortom.
I would say put your money in the snare, that's a lot of the identity of the kit and in skins.