The previous Emperor was described as an almost feminine great beauty during his youth, that faded into ugliness with his aging. He was but a political figure, with the country being ruled by his mother - the Empress Regnant - and later prime-minister Shishou, her ally.
With the exception of his mom, whom he very much admired, he was afraid of adult women and was known to prefer his consorts young. His first son was the result of forcing himself on a 10 year old attendant to a consort. He died about 5 years before the start of the story.