We used to remove hats because they were a tool. A piece of protective equipment. Ladies weren't expected to remove theirs because they were for decoration.
If I am working on a project with goggles and latex gloves, I'll remove them before going back into my home. If I have a hat on that I am wearing to protect my face from the sun when I mow the lawn, I'll take it off when I go inside. On the same line of reasoning, if I wear a hat for style (which I almost never do), then I likely won't bother to take it off inside, anymore than a lady is likely to remove jewelry or decorative accessories.
If I am working on a project with goggles and latex gloves, I'll remove them before going back into my home. If I have a hat on that I am wearing to protect my face from the sun when I mow the lawn, I'll take it off when I go inside. On the same line of reasoning, if I wear a hat for style (which I almost never do), then I likely won't bother to take it off inside, anymore than a lady is likely to remove jewelry or decorative accessories.