Yes, we all have bad times when we're short or mean with those close to us. That's not what we mean by a toxic person. No one is talking about someone having a bad period of time in their lives and being difficult toward their friends and loved ones. They're talking about lifetime patterns, about someone who is always usurious, delterious and disloyal to those close to them – someone whose entire way of being is narcissistic or sociopathic.
Overwhelmingly, when someone is identified as "toxic" it's because of a pernicious pattern of toxic treatment of those closest to them. It's almost never just because, "somebody doesn't 'get' me."