I sometimes where jean shorts and low rise work boots out in my shop an the wife makes fun of me. I'm not going for a job interview, I'm in my shop turning wrenches!
Yuhp do it all the time, especially on bike trips. Throw a pair of shorts on at the end of the day and the only thing I have with me are my riding boots.