hold against

phrasal verb

held against; holding against; holds against

: to use (something) as a reason to have a bad opinion of (someone)

He lied to her once, and she still holds it against him.

Nobody is going to hold it against you if you don't come.