Sure there is--the Fifth Amendment allows for the deprivation of liberty if due process is given. Liberty is more than just your freedom to move about; it includes all liberties. There are lots of things that can--and often do--get restricted as part of a criminal punishment even if you never serve time.It doesn't matter that the Constitution doesn't prohibit it. There is no grant of power to do it.
Woody