Greenmail is a sum of money paid by a company to someone who holds a large share of the company's stock, with the goal of preventing a hostile takeover. The term is a portmanteau of “greenback,” a slang term for American currency, and “blackmail.” Some businesses also refer to greenmail as a goodbye kiss or bon voyage bonus, referencing the fact that the company essentially says goodbye to a substantial sum of money when it pays greenmail.
This term originated in 1983, during the height of the hostile takeover boom in the United States. In the 1980s, numerous companies swallowed each other up, taking advantage of increased liquidity to snag a variety of new assets. Some companies were naturally displeased with this state of affairs, and they attempted to fight such takeovers. Greenmailers saw a potential for making substantial amounts of money, and seized upon it.
A greenmail attempt requires a lot of accessible funds, because the greenmailer must be able to purchase enough shares in a company to legitimately threaten a takeover. Once the shares are acquired, the greenmailer can alert the company to the situation, if the company hasn't already realized, and the company is invited to make an offer to buy the shares back. The company will typically pay a premium above the actual value of the stock to regain control of the shares, allowing the greenmailer to profit, sometimes substantially.
When a company pays greenmail, it may also attempt to negotiate an agreement with the greenmailer, usually to the effect that the attempt will not be repeated. Many companies also try to keep greenmail attempts a secret, to discourage any further attempts, and because capitulating to what is really just blackmail can be a bad public relations move. However, greenmailers may go public in the hopes of forcing a high settlement, an attempt which sometimes backfires.
This practice is perfectly legal: there's nothing to stop someone from buying a controlling share in a company, and nothing to stop a company from buying that share back. However, some people believe that it is a rather slimy business practice, and these individuals would prefer to see regulations which could prevent greenmail. At the very least, greenmail attempts could potentially be prosecuted by savvy attorneys if the greenmailer slipped up and crossed the line into blackmail, because blackmail is most certainly illegal.