InteLiPlan: Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy

¹ K. T. Ly and I. Havoutis are with the Oxford Robotics Institute, University of Oxford, Oxford, UK.
² K. Lu is with the Department of Computer Science, University of Oxford, Oxford, UK.

            Paper Link               Code (Coming Soon)

We introduce a lightweight LLM-based framework designed to enhance the autonomy and robustness of domestic robots, targeting onboard embodied intelligence. By addressing challenges such as kinematic constraints and dynamic environments, our approach reduces reliance on large-scale data and incorporates a robot-agnostic pipeline. Our framework, InteLiPlan, ensures that the LLM model's decision-making capabilities are effectively aligned with robotic functions, enhancing operational robustness and adaptability, while our human-in-the-loop mechanism allows for real-time human intervention in the case where the system fails. We evaluate our method in both simulation and on the real Toyota HSR robot. The results show that our method achieves a 93% success rate in fetch me task completion with system failure recovery, outperforming the baseline method in domestic environment. Our lightweight LLM-based planner achieves comparable performance to the state-of-the-art while guaranteeing real-time onboard computing with embodied intelligence.

The fine-tuning data is structured as text-only data, allowing deployment on different robotic platforms without adjustments. Below is a demonstration of the vision and feasibility module as chatbots.

We initially tested the 'fetch me' task in simulation with a Toyota Human Support robot. The results shows that InterPlan can plan the 'fetch me' action sequence, and replan upon failures with human intervention. The vision and feasibility modules in the form of multimodal perception allow LLM-based generated plans to be executable on the robot.

In the real robot experiments, we demonstrate the InteLiPlan capabilities of providing interactive on-board planner with failure reasoning and failure recovery.

This set of experiments showcases the ability of the fine-tuned model to plan the correct sequence of actions for different domestic tasks with no-failure cases.