Ribbi Tarfon tells us in Pirqei Aboth (Ethics of the Fathers): "Wehassakhar Harbeh" (and the reward is great).
In the Midrash it says that when Moshe Rabbenu, 'a"h, spent the forty days and nights on Har Sinai (Mount Sinai), he did not sleep at all. Can we imagine if a king would tell his servant, "Count all these gold coins, and whatever you count from now until tomorrow is yours". Would the servant go to sleep at all during that time? Of course not. Moshe Rabbenu likewise said, "If I go to sleep, how many Pearls of Torah will I lose?"
This is why it says "the reward is great". In the few hours a day that you spend in prayer and study of the Torah, you get a reward of the equivalent of countless precious stones. This is out of all proportion to the time invested, especially when you compare this to what you make when you work at your business for the same amount of time.
That is why we should not be concerned about the few hours a day spent on prayer and Torah study. But the reality is that we see that people begrudge the time when the Hazzan takes a few minutes longer, or if they have to spend fifteen minutes after Tefillah (prayer) to study Hoq LeYisrael [Chok Leyisrael].
(See Hasdei Aboth, Pereq Beh)