I think hornets (and bumblebees) are the ones that build their nests underground. Spider killer and cicada killer wasps will also come out of the ground, but they just bury their prey and lay their eggs in the hole, they don’t build colony nests there.Are mud daubers the ones the come from under the ground? I hate those things.
It is kind of fun watching the spider killer wasps drag those paralyzed spiders across the shop floor on the way to the burial site. The spiders are usually much larger than the wasps, so they have to work at getting them over obstacles.