Why is the abbreviation of G-d's Holy Ineffable Name the letter Heh? In view of the fact that His Name actually begins with the letter Yod, we should logically use a Yod as an abbreviation. In fact, the Heh is actually the last letter of His Name.
Heh is the letter with which the world was created, as it says: "Behibbar-am" (when they [the contents of the Heavens and the Earth] were created). Rashi says that by pronouncing it slightly differently we read it: "BeHeh Bera-am" (with the letter "Heh" He created them). This is one reason for its importance.
Another explanation in Torah Lishmah is that the letter Heh is different from all the other letters of the Hebrew alphabet. When you pronounce any other letter, you feel the letter on your lips or on your tongue, but the letter Heh doesn't touch anything and is completely pure (because it is essentially only air).
This shows us the importance and uniqueness of the letter Heh and why it is the most fitting to be used as an abbreviation for G-d's Ineffable Name.
(See Torah Lishmah, She-elah Taw Kaf Geemal)
