It may not come as a surprise, but worms in your garden will actually help your soil. In a process called vermicomposting, organic matter is ingested, broken down in the worms digestive track, and excreted as matter teeming with bacteria and nutrients that help plants grow.
This illustration was done for an NPR on vermicomposting, published on NPR.org in August 2016.