Very good question. Faces seem to change alot over time, i mean the underlying bone structure. Just look at christian bale, he went from average kid to insanely good looking (with zygos to die for) during american psycho, then back to being almost repulsive in for example the machinist, back to being very good looking in batman, to looking like a boneless blob. Now hes getting back on track again.
Muscles tissue and fat play a role here. But i think the main driver for the change in appearance is bone remodelling. Never found an answer on how big this change really is and what influences its extent.