bj is right when released in small amounts it is not harmful, when vacuuming you end up removing so many good things that it turns out to be detrimental for your aquarium, like your pods that eat waste, algae, and stir up your sand.
If you have an area (a dead spot) that doesn't get much flow and there is detritus build up it is okay and best to do a very light vacuuming in that area. But good flow in your tank should help you keep area's clean and things moving to your filtration.