Let me elaborate a little since I'm not on my phone now.
I'm a psychologist, so I know a thing or 2 about depression. While it may be the case that low testosterone is correlated with depression, it's only a correlation. More likely than anything, however, is that it's a spurious one. That is, it seems related but only because of a third factor that's associated with low testosterone and depression (another example: ice cream trucks are spuriously correlated with aggression).
My theory is that low t is related to lack of motivation or drive. Low drive is related to lack of reinforcement opportunities such as social activity, exercise, etc. That's what's related to depression. I am willing to bet that if one were to do a controlled study, lack of reinforcement opportunities would mediate the relationship between low t and depression. That is, low t isn't related to depression at all except by association.
And that's all a logical leap on my part.
TL;DR: your doctor is a moron.