The standard read more link, as the comments in NodeViewBuilder say, is always displayed on a teaser by default because there is no (easy) way to know how the teaser content differs from the full view. Your content type might have several other text fields apart from the body which are only shown on the full page view.
However, it should be possible to implement a custom "read more" link that has the behaviour you seem to be asking for. I don't think anyone has implemented one yet, but the most fruitful approach would be a modest extension of the "summary or trimmed" field formatter, although you could probably also do something in a preprocess hook in your theme.
Edit: A demonstration module is up on github now. It's not perfect and I would like to refactor it and possibly make it more general-purpose but it works for the simplest case.