Overselling is when a host sells more space and/or transfer than they have available. For instance, a host may be leasing a server with an 80gb hard drive and 700gb of transfer. If the hosting plans offered by the host is 20gb of disk space and 100gb transfer that would mean the host can only sell 7 hosting plans without overselling.
Without overselling they could only sell THREE
of the hosting plans before they had run out of disk space - the operating system, general files etc taking up some of the 80Gb, leaving under 75 for the 3*20Gb hosting accounts.