Ahmad and Abu Dawud transmitted a similar hadith on the authority of Abu Sa'id, and they added the extra statement, "Go back and ask for their permission. If they permit you to go, then go for Jihad otherwise, be good to them (look after them). (FR)
English reference (FR): Book 11, Hadith 1302 (FR)
Arabic reference (FR): Book 11, Hadith 1275 (FR)