nothin...once you start vaporizing you won't want to use your vapes for anything else. I've got some nice glass pipes that will never get smoked in because I attach them to my vaporizers.
No it won't cause damage, but have fun getting it clean again. Even vaporizing tobacco will gum up and stink up a vape with nicotine and tars. If you want to smoke I'd use a pipe. A pipe makes for a much smaller lump in your pocket anyway.