The Prophet (Peace Be upon Him) and the Righteous Caliphs did not call to prayer because of their tight schedule, they were busy with running the state's affairs and taking care of the citizens' needs, i.e. their main duties as rulers. This is why they were not devoted to adhan and its set times.
Al-Mawwaq said: The Prophet, Peace Be upon Him, did not call to prayer because if he says "Hay ala al-Salah" (Hurry to prayer), and they did not hurry, they would have deserved a punishment as Allah says: "…and those who go against his order should beware lest a trial afflict them or they receive a painful punishment." (An-Nur: 63) Omar ibn al-Khattab said: If it was not for (the responsibilities of) the Caliphate, I would have called for prayer (recited the adhan). Encyclopedia of Islamic Jurisprudence (32/158)
One of the common questions is whether the Prophet called for prayer. Al-Suhaily said that the Prophet, Peace Be upon Him, called for prayer when he was travelling along with his companions. They prayed riding, the sky above them and the wet land below them. (Narrated by at-Thirmidhi)
An-Nawawi also confirmed that the Prophet recited the adhan once while travelling. An-Nawawi linked this to what at-Thirmidhi said, but it was found in Musnad Ahmad as it was narrated by at-Thirmidhi. It was said "He ordered Bilal to call for prayer, and he did…" So, it was concluded that at-Tirmidhi narration was summarized. As for the meaning of "He recited the adhan commanding Bilal to do it," it is like when we say the Caliph gave this scholar a sum of money, but it was given literally by someone else and the Caliph gave him the order to do it.
Fath al-Bari by Ibn Hajar (2/79).