When 'I miss you too' means 'I miss who you were'.
When you say it and you remember who that person used to be. The situation of how you met. The personality that just did it for you. The time spent learning to like or to love someone else as they are and not how you want them to be.
How all the things that have happened while getting to know that person managed to then render you estranged from that person you knew once.. and the person that they are now just isn't the same and there's no point trying to get along with that person. Or maybe the situation makes it so that there's too much mental strain involved in continuing to deal with that situation.
It's heartbreaking when not all of that is true and when the problem is you, and you have a problem with people around that person you once knew. They love that person too and there's just no place for you. You could probably make it work anyway but decide to make do with only having a memory of the person you once knew.