ryno11,
Do what you like. It's your car.
There are good gains to be had from just exhaust work; cat back, high flow cat or test pipe...
Switching out the downpipe for me was where it really took the cork out.
There's nothing wrong with starting out slow and doing one thing at a time.