What downpipe are you using? Did you buy it new or used? Are you using a test pipe?
If, and a I do say if, it's hitting the oil pan, it should be leaving some kind of marking on the part it's hitting. However, such a thing is far from usual unless this is something you fabbed yourself, bought a knockoff part, or had modified/built by someone that didn't measure correctly. It's more likely that it's hitting the lower brace. WHen you inspect it again, look there for evidence of repeated contact.