I am still friends with all the weed enjoyers I know and have met. Those who have fallen to the harder stuff are still where they were 10 years ago. I feel bad, but as a previous drug abuser myself, it was best to move on. I'm glad that marijuana legislation in the US has lessened the negative connotations associated with use, but it's still going to be a while before it's as accepted as drinking responsibly.
It's your preference to distance or not. It sucks when you have to, but if drugs are not your thing, then as an adult, you have that right to make that decision. No matter what, it still sucks to lose people.