Today, vanilla is so well-known that its very name means “common.” But for centuries, vanilla was a rare, New World flavor enjoyed mainly by European elites. That changed in 1841 thanks to a 12-year-old boy wielding a tiny stick.

Edmond Albius was an enslaved worker in the French colony of Réunion who, after close inspection of the vanilla orchid Vanilla planifolia, figured out how to hand-pollinate its flower to produce vanilla beans.

Albius used the stick to push up a flap in the orchid flower called the rostellum and press the pollen-coated anther against the female part, or stigma. Until Albius’s discovery, vanilla had been successfully cultivated only in its native southeastern Mexico, home of its pollinator, the Melipona bee. It was there that the Spanish explorer Hernán Cortés famously witnessed the Aztec Emperor Montezuma drinking a chocolate beverage flavored with vanilla.

In Réunion, output of vanilla soared thanks to the Albius method, and orchid cultivation expanded to nearby Madagascar. Today, about 80% of the world’s natural vanilla comes from smallholder farms in Madagascar. There, locals continue to pollinate orchids by hand and cure the beans in the traditional fashion.

It didn’t take long for vanilla demand to exceed supply from the farms of Madagascar. In the 1800s and 1900s, chemists took over from botanists to expand supply of the flavor. Vanillin, the main flavor component of cured vanilla beans, was synthesized variously from pine bark, clove oil, rice bran, and lignin.

Rhône-Poulenc, now Solvay, commercialized a pure petrochemical route in the 1970s. In recent years, of the roughly 18,000 metric tons of vanilla flavor produced annually, about 85% is vanillin synthesized from the petrochemical precursor guaiacol. Most of the rest is from lignin.

But the traditional vanilla bean is starting to enjoy a renaissance, thanks to consumer demand for all-natural foods and beverages. Last year, a string of giant food companies, including General Mills, Hershey’s, Kellogg’s, and Nestlé, vowed to eliminate artificial flavors and other additives from many foods sold in the U.S.

There is a problem, however: World production of natural vanilla is tiny and has been falling in recent years. Less than 1% of vanilla flavor comes from actual vanilla orchids. With demand on the upswing, trade in the coveted flavor is out of balance.

Flavor companies are working feverishly to find additional sources of natural vanillin and launch initiatives to boost the quality and quantity of bean-derived vanilla. Suppliers such as Symrise, International Flavors & Fragrances (IFF), Solvay, and Borregaard are using their expertise along the full spectrum of natural to synthetic to help food makers arrive at the best vanilla flavor for each product.

Food makers, meanwhile, are confronting skyrocketing costs for natural vanilla, reformulation challenges, complicated labeling laws, and difficult questions about what is “natural.”

Although consumer disdain for artificial ingredients has been building for years, credit—or blame—for last year’s wave of “all natural” announcements goes to Nestlé, which in February 2015 was the first major brand to announce plans to eliminate artificial additives from chocolate candy sold in the U.S. The announcement upended the mass-market chocolate industry practice of adding synthetic vanillin to counter the bitterness of cocoa.

“When Nestlé announced it would go all-natural, that just opened the floodgates,” observes John Leffingwell, head of flavor and fragrance market research firm Leffingwell & Associates. “Other brands weren’t going to do anything until a major player pushed it. With Nestlé being the biggest in the world, everyone had to follow.”

Food and flavor companies have been quietly reformulating products and sourcing ingredients that can carry a natural claim for at least a decade, Leffingwell points out. Indeed, Nestlé first cut out artificial additives from its U.K. confections in 2012. At the time, the company credited “a seven-year research and development program that has led to more than 80 artificial ingredients being replaced with alternatives.”

For a big food firm, however, switching to natural vanilla is akin to squeezing an elephant into a Volkswagen. While the ink was drying on those all-natural announcements last year, output of Madagascar vanilla beans had plummeted to 1,100 metric tons, about half the normal harvest. That, along with rising demand, caused prices to more than double to roughly $225 per kg by the middle of last year, according to Mintec, a raw material price-tracking firm.

Cured vanilla beans contain only 2% of extractable vanilla flavor, meaning prices for pure vanilla reached an eye-popping $11,000 per kg. The industry is closely watching this year’s harvest, hoping to see vanilla costs eventually return to pre-2012 levels of about $25 per kg for beans or $1,250 for vanilla.

Even that is a hefty price tag for one of the world’s most popular flavors. In the U.S., 2% of food products and 3% of beverages launched in the past 12 months were advertised as vanilla-flavored, according to the market trends tracker Mintel. Symrise, which sources natural vanilla and supplies synthetic vanillin, says 18,000 global products contain vanilla flavor.

“The amount of all the vanilla beans in the world is not sufficient to flavor everything that everyone wants to flavor with vanilla,” says Carol McBride, U.S. vanilla category manager at Symrise.

Dairy products such as ice cream and yogurt have relied on natural vanilla for many years, McBride points out. In the U.S., Food & Drug Administration rules state that vanilla ice cream must get its flavor from natural vanilla. If the flavor comes partially or fully from another source, the company must stamp “vanilla flavored” or “artificial vanilla” on the front of the package, a likely turnoff to consumers.

Other foods don’t get that level of government oversight. For makers of confectionary items, baked goods, and breakfast cereals, it’s consumer hankering for all-natural ingredients that is now having an impact, according to McBride. For nondairy products, “the labeling and brand message of the product usually determines the type of vanilla used,” she says.

For example, Nabisco’s Nilla Wafers cookies, no one’s idea of an all-natural product, needn’t make a claim on the front of the box. As the ingredients list on the back reveals, the “Nilla” flavor comes from “natural and artificial flavor.” In contrast, Ultimate Vanilla Wafers from grocery retailer Trader Joe’s boast “flecks of Madagascar vanilla beans” on the label.

Vanilla is also an important component of flavors such as chocolate, strawberry, caramel, and coconut. It rounds out the taste by adding creaminess, balancing sweetness, and toning or masking bitterness and acidity.

When vanilla flavor is used in chocolate, it becomes part of a proprietary recipe that makes a Nestlé bar taste different from one made by Hershey’s. “It’s significantly more challenging to convert an existing product to natural ingredients than to create a new product with natural ingredients,” says Liz Caselli-Mechael, manager of corporate communications for Nestlé. “Vanilla is among the most challenging flavors to match because the vanilla wheel has dozens of flavors.”

A flavor wheel is how the food community tracks the specific attributes of an ingredient, food, or beverage. The vanilla wheel used by the flavor company Fona International measures fully 29 distinct flavor characteristics. They are grouped into 10 main categories: smoky, spicy, botanical, sulfury, sweet, creamy, medicinal, cooked, fatty, and floral.

Like wine, natural vanilla grown in different places—Madagascar, Mexico, or Tahiti—has different taste and potency profiles. Madagascar vanilla, typically called Bourbon vanilla, is highly sought for its rummy taste and sweet aroma.

Food and flavor firms rely on highly trained tasters to help with the transition to natural. “We’ll perform multiple tests to get as close a sensory match to the existing product as we can. That includes taste, aroma, and texture,” Caselli-Mechael says. Potential matches are then tested with consumers. The process can take months or even years.

Food companies that abandon synthetic vanillin can turn to natural vanillin from sources other than vanilla beans. For example, Solvay makes Rhovanil Natural vanillin by fermenting ferulic acid, a by-product of rice bran oil, using a proprietary strain of yeast. The French flavor company Mane uses a different raw material, eugenol from clove oil, to make natural vanillin.

In a review of vanillin bioconversion processes, Nethaji Gallage and Birger Møller, plant scientists at the University of Copenhagen, found that both ferulic acid and eugenol are pricey raw materials (Mol. Plant 2015, DOI: 10.1016/j.molp.2014.11.008).

Production of eugenol is industrialized because the molecule is used to produce various flavors and fragrances, but its price tag is still around $50 per kg, the researchers say. Naturally extracted ferulic acid costs even more, around $180 per kg.

Compared with synthesizing vanillin from petroleum or extracting it from vanilla beans, bioconversion via yeast and bacteria presents limitations, Gallage and Møller note. High concentrations of both ferulic acid and eugenol are toxic to most microbes, as is vanillin. Indeed, all three compounds are made by plants as antimicrobials.

What’s more, yields are generally low because the microbes also produce undesirable vanillin alcohol and vanillic acid. Obtaining a vanillin yield of more than a few grams per liter of fermentation broth requires specialized or mutated strains and often lengthy incubation periods.

The biotechnology firm Evolva developed a process to get around high feedstock costs and toxicity problems by feeding glucose to a genetically modified microbe that produces vanillin glucoside. The sugar group makes vanillin much less toxic to the production organism but must be removed to get vanillin.

It is not yet clear whether vanillin made from genetically modified organisms will be adopted or marketed. Because the microbe that expresses Evolva’s vanillin is considered a processing aid, a product made with the flavor would not fall under U.S. GMO labeling requirements and could lend itself to no-artificial-ingredient claims. On the other hand, the Non-GMO Project says foods containing ingredients made with synthetic biology will not be allowed to carry its voluntary label.

Evolva licensed its technology to IFF in 2014 for commercialization and distribution, but flavor industry insiders haven’t heard much about it since. Leffingwell notes that IFF could be concerned about consumer backlash to the use of GMOs.

Neil Goldsmith, Evolva's chief executive officer, says Evolva supports the advent of mandatory GMO labeling in the U.S., adding that the Department of Agriculture has a few years to define the exact approach. Goldsmith emphasizes that Evolva's vanillin contains no GMO content. "When consumers get the facts, they prefer us over the alternative, which comes from petrochemicals," he says.

The allure of nonpetrochemical vanillin is that it can be used in foods that claim “no artificial ingredients.” For now, the alternatives’ big drawback is their high price—generally several hundred dollars per kilogram. That’s cheaper than vanilla but a lot more pricey than synthetic vanillin at around $10 per kg.

Food makers that have disavowed petroleum-derived ingredients but aren’t looking for an all-natural label can turn to Norway’s Borregaard, which offers a vanillin that is derived from plants but is much less pricey than either real vanilla or other alternatives.

Borregaard operates a large “biorefinery” where it mainly converts spruce trees into specialty chemicals and cellulose. But it finds itself with lots of softwood lignin, a large biopolymer made up of close to 90% coniferyl alcohol. Heat and pressure in the refinery convert the lignin to vanillin.

As recently as the 1990s, much of the world’s vanillin came from lignin in the waste stream of pulp and paper firms. But the process fell out of favor as papermakers changed their methods to reduce waste. That left Borregaard the only major firm to make vanillin from wood.

Although Borregaard’s vanillin is considered artificial by regulators, many food makers prefer it to vanillin made from petrochemical feedstocks because it is considered sustainable, says Amie Byholt, the company’s U.S. director for sales and marketing. It is made from a renewable raw material in a facility that uses 90% of its biomass in products and the rest for energy.

There is also a taste difference, Byholt says, thanks to tiny amounts of other aromatic components that come from lignin. “They add to the intensity and the roundness of the aroma profile, in effect giving it a stronger and creamier flavor compared to the equivalent amount of petroleum-derived vanillin.”

For companies that want to use natural or nonpetroleum flavors but can’t stomach high prices for vanilla beans, vanillin from alternative sources can be a practical choice, says McBride of Symrise. But to truly participate in the natural trend, sport an organic label, or sell in specialty retailers, food makers have to commit to the roller coaster of natural vanilla.

“People are trying to live through the current crisis so they can deliver that taste that consumers love and not have to always reformulate,” McBride observes.

To help prevent future vanilla crises, Symrise, Givaudan, Mane, and IFF have set up grower programs in Madagascar. The country’s northeastern, or Sava, region has the world’s best combination of climate and farm labor economy for vanilla production. McBride says the programs aim to help smallholder farmers maintain consistent quality and a sustainable supply.

For example, cash-pressed farmers often pick beans early. “But they may not realize that immature beans have lower quality and vanillin content,” McBride says. “The Symrise teams work to promote security for the farmers so they know we will pay a fair price to diminish the fears that lead to early picking.”

Symrise set up its Madagascar operations in 2006 and now reaches 7,000 farmers. Sustainability programs help farmers plant crops such as cocoa, cloves, and cinnamon along with vanilla vines on their land. The programs train farmers to preserve soil fertility and sponsor education, health care, and food programs.

Vanilla is a very labor-intensive crop. It takes 600 hand-pollinated blossoms to produce 1 kg of cured beans. Beans are picked while still green and sold to fermentation plants where workers sort, blanch, steam, and dry the beans in the sun. They are then sorted again, dried in the shade, and fermented while workers continually evaluate their aroma and inspect each bean for quality.

Farmers are increasing their income from vanilla by attaining valuable certifications such as organic, fair-trade, and Rainforest Alliance Certified. But it is difficult for them to plant more orchids because their farms are often quite small. In a best-case scenario, “everyone just plants five more vines,” McBride says. Even then, it takes four years for those vines to reach maturity.

To meet growing demand, the vanilla industry may need an innovation akin to Albius’s hand-pollinating technique. Again, the secret may be in the vanilla plant itself.

In 2014, Møller discovered that specific plant cells in the waxy interior of green vanilla beans enzymatically transform free ferulic acid into vanillin glucoside (Nat. Commun., DOI: 10.1038/ncomms5037). The cells express a gene that codes for the active enzyme—vanillin synthase. Møller was able to use variations of the gene to produce vanillin in vitro, as well as in vivo in a modified strain of yeast and in modified tobacco and barley plants.

The insight could be used by plant geneticists to find markers that help select vanilla orchid varieties that produce more vanillin or to develop transgenic plants with high vanillin synthase activity. Another option would be to engineer yeast to make vanillin from raw materials such as molasses, which contains ferulic acid, Møller suggests.

Regardless of what technologies may arise to produce more natural vanillin, McBride says the core challenge remains. “We need to grow more vanilla beans. We need to safeguard the sustainability of vanilla so that we can enjoy vanilla for generations to come.” 

This story was updated on Sept. 20 with a comment from the biotech vanillin developer Evolva that was received after C&EN's deadline.