His right to express his political opinions. But unfortunately this move makes him look very petty and very small. An invitation to the White House is a symbolic goodwill gesture. Turning down an invitation because you disagree with the President's politics is pretty weak and poor manners. I would think the same of a conservative who turned down an invite from Obama or Clinton. I am a KD fan but he sure makes it difficult at times.
This is how I feel.
President Obama was JUST AS divisive as President Trump. Every bit.
However, if a politically conservative athlete refused to go to the White House and meet the President because the man was divisive, I would call that petty and immature.
It is, of course, his exclusive right not to go meet anyone he doesn't want to for any reason he sees fit. It's my right to have an opinion and yours to have an opinion about mine, etc. I mainly hope for consistency (intellectual honesty) and general decorum. (Even when others do not display such.)