Yahya related to me that Malik said, "The sunna concerning thetime of prayer on the ids of Fitr and Adha - and there is nodisagreement amongst us about it - is that the imam leaves his houseand as soon as he has reached the place of prayer the prayer fallsdue."Yahya said that Malik was asked whether a man whoprayed with the imam could leave before the khutba, and he said, "Heshould not leave until the imam leaves."
USC-MSA web (English) reference: Book 10, Hadith 14