Biofuels have long been touted as a carbon-neutral alternative to fossil fuels, doing for the world’s planes, ships and automobiles what windfarms and solar panels are doing for its electricity grids. With the transport sector accounting for almost one fifth of Australia’s total carbon emissions, green biofuels could be an important ingredient of the zero emissions future envisioned by the Paris climate agreement.

On paper, biofuels seem the ideal replacement for fossil fuels, which drive global warming by spewing tons of carbon dioxide into the atmosphere that would otherwise be locked away in geological deposits. With biofuels, the plants and algae used to produce the raw material inhale carbon as they grow, offsetting the carbon released when they are burned.

But the past decade has seen the biofuel industry face tough economic conditions and niggling questions over its green credentials. The fledgling industry is now turning to a raft of innovative crop and processing technologies to overcome its challenges.

One of the biggest criticisms of the early generation of bioethanol crops, such as corn and sugarcane, was their propensity to mess with food markets and alter land use. Direct impacts – felling forests to make way for a biofuel crop, say – are usually obvious, says Prof Bill Laurance, director of the Centre for Tropical Environmental and Sustainability Science at James Cook University. But indirect impacts can be no less devastating for the environment and are far harder to predict. “The devil is really in the details,” he says.

As an example, when farmers in the US opted out of soy in favour of corn for bioethanol, soy prices soared, suddenly making it an attractive crop for Brazilian farmers, which in turn drove demand for freshly deforested cropland. One analysis of these knock-on effects estimated that instead of cutting emissions, corn-based bioethanol would double emissions over a 30-year period.

This is without even considering loss to biodiversity, pollution from pesticide and fertiliser use, changes to water catchments and decreased food security for marginalised populations. The latest IPCC assessment report, released in 2014, acknowledged some of these risks and trade-offs.

Prof Rachel Burton, leader of the ARC Centre of Excellence for Plant Cell Walls at University of Adelaide, thinks that there is a smarter way forward for biofuels and it starts with selecting the right crop. Instead of growing food crops such as corn and sugarcane on prime agricultural land, Burton and others are looking to more hardy plants that grow on land too dry or saline for conventional crops. Australia could turn to crops like agave (of tequila fame), hemp, or the native saltbush and wild-growing sorghum for biofuels of the future, she says.

The once popular idea of generating biofuels from microscopic algae grown in ponds or tanks – which avoids land use altogether – has largely been abandoned due to the high production costs compared to fossil fuels. But economic considerations are also a factor for crop-based biofuels. Plant oils can be extracted and turned into biodiesel for vehicles and machinery, and aviation fuel that has already been used in commercial airline flights.

However, food oils from palm and soy hover at roughly twice the price of crude oil. “It really is fundamentally an economic problem rather than a technological problem,” says Dr Allan Green, innovation leader for biobased products at CSIRO Agriculture and Food. His solution is to make plants oilier. With more oil being produced on a given parcel of land, harvesting and production costs will inevitably fall. He and his colleagues have patented a way of tinkering with the genetic levers that control oil production in plants, so that a plant produces oil in its leaves, not just its fruit or seeds.

The technology, which has so far only been tested in tobacco, shows that oil production can be boosted to a third or more of the leaf’s weight, more than occurs naturally in any plant. If used in a crop that already produces oil in its seeds or fruit, the hope is that oil output could be doubled, though that theory is yet to be put to the test.

Changes to processing technologies are also influencing the direction that the biofuel industry is taking. Traditional approaches use plant sugars for fermentation to bioethanol, or oils that can be chemically transesterified for biodiesel production. A great deal of attention has gone into finding the perfect crops for these applications: plants dense in sugar-laden cellulose, minus lignins that make extraction more difficult; or crops that pump out high oil volumes. But the industry is also turning to methods that are less finicky about what plants are used.

Hydrothermal liquefaction uses heat and pressure to rip apart the long-chain molecules in whole plants into bio-crude oil, essentially compressing eons of geological time into a matter of hours. This can then be refined as you would petroleum-based crude oil, producing a catalogue of different fuels as well as plastics and other products.

“I think it has a huge future,” Burton says.

Similarly, torrefaction, a process adapted from coffee roasting, can turn essentially any plant matter into easily transportable bio-coal pellets. While both of these processes are energy intensive, combining them with renewables – say, solar panels or wind turbines – or co-locating them with power stations to harvest excess heat, could make the operations more environmentally sustainable, Burton says.

The advantage of a “crop agnostic” approach is that producers won’t be limited to crops designed to be biofuel-only crops but can instead choose species that deliver added benefits or income streams. Agave could be used to produce a high-value tipple, for instance, or hemp farmers could harvest seed for food or fibre for lightweight soundproofing like that used in BMW cars. Meanwhile, work by Kirsten Heimann, associate professor at the College of Science and Engineering at James Cook University, has shown that microalgae can simultaneously be used to produce biofuels and scrub mining tailings of contaminants.

“It’s much more sophisticated thinking,” Burton says, and could change the calculus for biofuels. “Biofuels maybe don’t need to be as cheap as we think they do, because you can make money out of the other things.”

The biofuel industry could well shape up to be a very diverse one, with no one crop or process surging ahead to claim the market, according to Green. “The amount of fuel we need to move away from petroleum is massive, so there’s plenty of space for all technologies,” he says.