Firstly, ask your Endocrinologist what this little known fact has to do with you specifically?
Secondly, request links to the relevant studies.
....& lastly inform him of the well known fact that medical practitioners have traditionally pulled "facts" out of their ass when it comes to Marijuana.
A good question for your Endocrinologist. Whatever the answer, be sure to ask what evidence(specific to your case) is supporting his conclusion?
This is a great response. In fact, more specifically, ask the endocrinologist if you have a testosterone deficiency/if this has even been tested to be established?
It does sound like you do though, or no doctor would recommend you start taking T. I have a large number of lady friends who became male friends (transgender male) and T is very well known to have all kinds of side effects. A doctor would be completely remiss to prescribe T without due testing that this was deficient in the first place, or being used for gender transition surgery.
I am familiar with gynomastia, in the context of my abnormal psychology education (learning to diagnose mental illnesses as well as the onset and etiology of such things). The significance of this was that gynomastia can be a condition which itself causes depression, anxiety and poor body image.
Please be aware, my colleagues and friends are psychologists/psychiatrists/medical doctors/sociologists of medicine. Those who practice with patients are very quick to admit that in the context of medical/mental illness, the second a patient mentions cannabis use, the practitioner instantly starts thinking of it as a possible cause of symptoms. Many even fail to realise that this may be self medication, or even recognise that cannabis can be used as a medication at all!
Remember, mainstream medicine education has not yet caught up with the literature on this topic, a body of which is still burgeoning. We should not be surprised if doctors often have old-fashioned ideas in relation to cannabis in the medical context.