I think there's a distinct line in that red hat group between the religious fanatics and the "government is nazis and want to kill us" groups. The latter is all in on pot, the former, not so much. The former still wants to ban dancing, stores being open on Sundays, booze, interracial marriage, and of course pot. The latter were the ones growing it.
Of course a couple of well placed Supreme Court picks won't care about that. So you might be right.