Damage may be a strong word to describe how non-sugar sweeteners affect you. What I mean by damage here is not something that can be measured biologically on a test, but the set point of your ability to appreciate sweetness.
How Your Palate Works
Your palate is like a thermostat and each time you eat overly sweetened foods, the thermostat to appreciate sweetness gets set higher and higher. As this happens, your ability to experience the pleasure of sweetness moves further and further away from being able to appreciate the sweetness of natural foods like berries.
Sugar is highly addictive. It has been found that “highly processed, hyperpalatable foods hijack the reward centers in the brain.” It impairs your ability to make good decisions, similar to drugs of abuse. 
When you load your diet up with stevia, monkfruit, allulose, erythitol, and other biologically safe sweeteners, you correspondingly decrease your ability to receive pleasure from Mother Nature’s bounty of healthy low-sugar fruits. Desserts and pastries, even with non-sugar sweeteners, should only be consumed on rare occasions when you want a treat. Using non-sugar or artificial sweeteners doesn’t help for you to learn to extract the greatest amount of pleasure from natural foods.
The same thing goes for salt. Like sugar, salt is also addictive. When your palate becomes used to high levels of salt in your diet, you start craving for more. Despite what a new wave of health gurus are saying about salt, higher than recommended levels of daily salt intake are not good for your blood pressure and it is related to a gene variant in a gene named AGE.
If you want to check how the foods you eat and nutrition in general affect your genes, I can order these tests for you. If you are interested, you can sign up for a free consultation with me. Some people depending on their genes can increase their sodium possibly with some benefits.
Gain Pleasure from Natural Foods
To retrain your ability to appreciate natural sweetness or saltiness, gain as much pleasure from natural foods as possible. This should be one of your goals to achieve optimal health. While non-sugar sweeteners might not be as bad as sugar or flour, they ruin your ability to achieve balance with natural foods.