Narrated 'Aishah (RA) regarding the saying of Allah the Most High:
"Allah will not punish you for what is unintentional in your oaths (5:89)." She said: It (was sent down about such phrases as) a person's sayying: 'No, by Allah' and 'Yes, by Allah.'[al-Bukhari reported it (Mauquf, i.e. as a saying of 'Aishah). Abu Dawud reported it as Marfu' (attributed to the Prophet).]
Reference: Bulugh al-Maram 1381
In-book reference: Book 13, Hadith 8
English translation: Book 13, Hadith 1381