I'm having a mild cannabis allergy as well. But only on skin contact (especially on the forearm). Fortunately, the allergens seem to vanish when I vape or burn the material.
Have to be careful not to touch my eyes after fiddling around with the ground herb, though.
Seems to be a cross-reaction...