InteLiPlan: An Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy

¹ K. T. Ly and I. Havoutis are with the Oxford Robotics Institute, University of Oxford, Oxford, UK.
² K. Lu is with the Department of Computer Science, University of Oxford, Oxford, UK.

            Paper Link               Code

We introduce an interactive LLM-based framework designed to enhance the autonomy and robustness of domestic robots, targeting embodied intelligence. Our approach reduces reliance on large-scale data and incorporates a robot-agnostic pipeline that embodies an LLM. Our framework, InteLiPlan, ensures that the LLM's decision-making capabilities are effectively aligned with robotic functions, enhancing operational robustness and adaptability, while our human-in-the-loop mechanism allows for real-time human intervention when user instruction is required. We evaluate our method in both simulation and on the real Toyota Human Support Robot (HSR). Our method achieves a 93% success rate in the `fetch me' task completion with failure recovery, highlighting its capability in both failure reasoning and task planning. InteLiPlan achieves comparable performance to state-of-the-art large-scale LLM-based robotics planners, while using only real-time onboard computing.

The fine-tuning data is structured as text-only data, allowing deployment on different robotic platforms without adjustments. Below is a demonstration of the vision and feasibility module as chatbots.

We initially tested the 'fetch me' task in simulation with a Toyota Human Support robot. The results shows that InterPlan can plan the 'fetch me' action sequence, and replan upon failures with human intervention. The vision and feasibility modules in the form of multimodal perception allow LLM-based generated plans to be executable on the robot.

In the real robot experiments, we demonstrate the InteLiPlan capabilities of providing interactive on-board planner with failure reasoning and failure recovery.

This set of experiments showcases the ability of the fine-tuned model to plan the correct sequence of actions for different domestic tasks with no-failure cases.

@misc{inteliplan,
    title = {InteLiPlan: An Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy},
    author = {Kim Tien Ly, Kai Lu, Ioannis Havoutis},
    year = {2025},
    eprint = {2409.14506},
    archivePrefix = {arXiv},
    primaryClass = {cs.RO},
    url = {https://arxiv.org/abs/2409.14506}, 
}