My GP doctor tried to convince me that cannabis was going to give me testicular and brain tumors.
Instead of living in that fear, I decided to live life life happily and and enjoy every day :D
One thought is, couldn't you get tested for testosterone levels? If it was indeed low, then it might...