I admit that being a non-partying, non-pot smoking virgin makes me feel incredibly inadequate when I'm with my camp friends, who are pretty much everything I'm not.
Take that, only change camp friends to roommate, and you have me. Only I don't feel inadequate, because I know there is nothing wrong with me, just like there is nothing wrong with you. Alex is right, you don't need any of that to have fun. Find some friends that have similar lifestyles and hang with them. I'm not saying you should dump the ones you have now, just have some others you can feel comfortable with.
My only problem is that I only have one of those. The rest are in different colleges.