The negative connotation also goes back to the idea of cats as familiars – and, witches (women, of course) as evil.