Your 7-figure friends are right. Getting money at under 3% is ridiculous. Put your house on a 15 year note at 2.75%. Then take your cash and buy as much rental property as you can at 3.5%. Let your renters pay off those houses while you pay off yours. You'll beat the spread on real estate appreciation alone, not to mention the equity you'll build as you go along.
I know more than one person that WERE doing very well at this in the 80s. One of them had 35-40 houses and a small apartment complex. All of them lost everything including their personal residence.