Nope, just saying what a person does in his free time does not determine his political affiliation.
You are right it doesn't determine it - but there are many correlations. It is plainly obvious if you have ever been to let's say a Friends of the NRA Dinner versus a Public Interest Research Group (PIRG) teach-in..not all of the former are conservative and not all of the latter are liberal but the correlation is very high. Your one example is merely the exception that highlights the rule.