Humanoid robots for the fruit growing of the future – automation, digitalization and AI

How humanoid robots learn for horticulture to understand and evaluate plants in their environment and derive actions from this

Commercial fruit growing demands the highest levels of precision, experience and finesse. Tasks such as thinning, pruning, rating or harvesting rely on both technical and implicit knowledge concerning plants, growth conditions, diseases and pests as well as environmental conditions.

At the Fraunhofer IFAM training and competence center for humanoid robotics in horticulture located in Stade, Germany, researchers are investigating how humanoid robots can acquire this empirical knowledge, combine it with existing artificial intelligence (AI) models for object recognition and use it to derive appropriate actions for specific situations.

This is carried out in close collaboration with fruit-growing farms and advisory as well as research institutions such as the ESTEBURG Fruit Growing Centre Jork along with The Fruit Research Association Altes Land (OVR).

Fruit growing as a key application area for humanoid robots

Fruit growing – particularly the cultivation of table apples – is characterized by a high volume of manual labor, seasonal peaks in workload and a growing shortage of skilled workers. At the same time, many tasks can only be automated to a limited extent, as each plant is individual and decisions must be made based on the specific context.

This combination is precisely what makes fruit growing an especially suitable field of application for humanoid robotics: on the one hand, work environments for human tasks and human-like use of tools and movements, on the other hand, variable, complex and knowledge-based tasks.
 

Implicit knowledge of plants as a key challenge

Experienced fruit growers rarely make decisions based on individual characteristics; rather, they evaluate plants holistically: growth habit, leaf arrangement, fruit set, vitality, the previous growth season, site conditions and weather all unconsciously influence every decision they make.

Until now, this implicit knowledge has hardly been formalized. For humanoid robots, this means that they must not only learn individual hand movements but also understand the plant as a biological and ecological system that interacts with its respective environment.

The training and competence center for humanoid robotics in Stade creates the conditions necessary to systematically record and analyze this experiential knowledge and translate it into capability models.

Combination of object recognition, evaluation and action

n fruit growing, there are already powerful AI models for object recognition – such as those used to identify flowers, fruits, shoots or planting stakes (SAMSON project). However, pure recognition alone is not sufficient for the effective use of humanoid robots.

Crucial factors for the subsequent assessment and further action are:

• How is the identified condition evaluated?
• What action is appropriate for the situation?
• Which interventions are appropriate – and which are not?

Humanoid robots must learn to link visual information, sensory feedback (force, position, resistance) and contextual knowledge. This is the only way to establish a reliable basis for decisions regarding actions such as grasping, cutting, removing or leaving something in place.

Learning from real-world experience: humans as trainers for humanoid robots

At the training and competence center, the training of humanoid robots is not abstract but hands-on. Experienced fruit-growing specialists demonstrate typical tasks directly on real plants or test facilities.

These demonstrations serve as a starting point for:

• learning fine motor skills
• interpreting plant characteristics
• making decisions based on complex situations

Similar to a traditional apprenticeship, the robots learn step by step – from simple tasks to complex, situation-specific ones. The data collected is incorporated into AI models which can be reused and transferred to various vendor-neutral systems.
 

Collaboration with fruit-growing farms, advisory services and research facilities

A key factor in success is close collaboration with fruit-growing farms and fruit-growing advisory and research institutions, such as the OVR. They not only contribute practical, hands-on knowledge but also help formulate research questions in a practically relevant manner.

Through close collaboration, the following is being investigated:

• which tasks are suitable for humanoid robots,
• where are the technical or biological limitations,
• how humans and robots can work together usefully and
• how new technologies can be integrated into existing operational processes.

This results in a continuous exchange between research and practice, with the goal of developing reliable, reality-based solutions.
 

Perspective: Transferable knowledge for other permanent horticultural crops

The methods developed for fruit growing are not limited to individual crops, e.g., apple cultivation. Many principles – such as evaluating plant condition, working with variable geometries or learning from demonstrations – can be applied to other permanent crops in fruit growing and horticulture in general, such as vegetable growing, ornamental plant cultivation and tree nurseries.

Thus, fruit growing serves as a reference application for further developing humanoid robots for complex biological systems and, in the long term, making them usable also for other fields.

By combining humanoid robotics, AI-supported perception and practical training of humanoid robots, Fraunhofer IFAM in Stade is creating a research and development environment in which experiential knowledge from agriculture and horticulture is systematically translated into digital capabilities. The goal is to develop new tools – e.g., for fruit growing – that reduce the workload on skilled workers, ensure quality and support the sustainable management of permanent crops with less labor.