Right now you feel guilty, but I hope that you can see it as a moment to be proud of. You stood up for yourself, canceled their visit, and the world didn't come to an end. Standing up to them will get easier the more you do it. Set your boundaries and decide what you are willing to tolerate from them. In other words, YOU take control away from them. They get nasty on the phone? Hang up. They keep texting you? Block them. You decide when and how you are willing to engage with them.

It feels complicated, but it actually is very simple. If they want to interact with you and spend time with your child, they will treat you better. That's all. If they don't agree to that, then they can sit at home and talk about how awful you are, but you won't have to hear it. You have the power here! Use it wisely!