One thing always confused me - Matthew's arrival

In the scene when Matthew and Isobel arrive st downton, they are greeted inside the house by the family and staff. Isobel makes a comment about what a reception or something and everyone smiles... but Matthew says "what a welcoming committee" quite pleasantly, and everyone acts as if he just said an off color awkward dad joke.

Now, I get he's the new heir and there are some strained feelings, so I can see maybe Cora, Violet, and Mary sort of being cold, but Robert gives a look of "good god", when Robert is the one who insists he come and learn the place and he's settled that Matthew will be the new heir... so why does Robert, Carson, and everyone for that matter, react so ... weirdly and cold to his warm, pleasant greeting???