Malik related to me that he had heard that when Umar ibn Abd al-Aziz left Madina, he turned towards it and wept. Then he said, "OMuzahim! Do you fear that we might be among those that Madina castsoff?" (ID)
USC-MSA web (English) reference: Book 45, Hadith 9 (ID)
Arabic reference (ID): Book 45, Hadith 1609 (ID)