Good point, it will lessen over time. But when it becomes OK is subjective, if the OP is sensitive to it (and apparently is) it could take a long time.
Also, remember, Linseed oil is only an oil until it dries. Unlike say mineral oils or waxes. It then becomes very hard, stable and long...