Does a room that is round require a Mezuzah? The answer is, as long as it has the minimum square footage, which is four 'Ammoth (cubits) by four 'Ammoth, then it requires a Mezuzah. So too, a room which has an unusual shape, such as five walls, requires a Mezuzah if it has the minimum footage.
This concept of a minimum of four 'Ammoth by four 'Ammoth, refers to the total square footage of the floor space. So for instance if the length of the room is long, substantially more than four 'Ammoth, but the width is very narrow, (less than four 'Ammoth), as long as the overall floor space is equal to or greater than sixteen 'Ammoth, it requires a Mezuzah.
(See Ben Ish Hai, Shana Beh, Parashath Ki Thabo, Oth Yod Teth)