you'd have to be really careless for that to happen. tritium vials are sealed to provide shock isolation and survive rigors of a firearm.
take little hits with a jewelers brass hammer but that's really over thinking it .. any small hammer will work fine .. easy does it ..
Oh, not careless, some gun companies use a 20 ton press to push them on, granted, not many, but the viles will break when trying a punch and a proper hammer. Small jewelry hammer wont budge them.