That may be part of it, but another large part of the difference is that, as previously stated, turning milk into cheese destroys lactose. You get fewer calories from the cheese because microbes already used up some of the calories that were originally in the milk in order to turn it into cheese, and thus those calories are no longer available for you.