Human 0, MLLM 1: Unlocking New Layers of Automation in Language-Conditioned Robotics with Multimodal LLMs

Language-conditioned robotics aims to enable robots to perform tasks based on natural language instructions. Recent advancements leverage NLP and computer vision, but most existing frameworks heavily rely on human intervention. Humans are crucial for assessing task feasibility, intervening when robots deviate from goals, and determining task completion. This HITL approach severely limits scalability and practical deployment. This paper introduces a novel framework, MLLM-IL, to automate these human functions using Multimodal Large Language Models (MLLMs) like OpenAI's GPT-4 and Google's Gemini. These models possess the capability to understand and generate language and interpret visual inputs, which are essential for automating the human roles in the robotic control loop. The MLLM-IL framework aims to eliminate the need for human intervention in task feasibility analysis, progress assessment, and success detection, significantly increasing the autonomy and scalability of language-conditioned robotic systems. The research utilizes the CALVIN framework and a new dataset to evaluate the effectiveness of the proposed framework.

Existing research in language-conditioned robotics demonstrates significant progress in enabling robots to understand and execute tasks based on natural language instructions. Frameworks like CALVIN provide a robust foundation for developing and evaluating such systems. However, a common limitation is the heavy reliance on human intervention for tasks such as feasibility checks, error correction, and success verification. Many studies utilize HITL approaches, highlighting the need for more scalable solutions. The emergence of MLLMs offers a promising avenue to address this limitation. While previous work has explored the application of LLMs and VLMs in robotics, often involving fine-tuning for specific tasks, this paper focuses on the zero-shot capabilities of MLLMs for automating critical human functions without any domain-specific fine-tuning. Existing research on MLLM vision capabilities has focused on benchmarking and evaluating robustness across various domains, while this research specifically explores their application within the context of language-conditioned robotics.

The proposed MLLM-IL framework integrates an MLLM into the robotic control loop to automate three key human functions: feasibility analysis, progress assessment, and success detection. Each robotic task is defined by a natural language instruction (I) and visual observations (o) from a camera. A language-conditioned agent (A) performs actions, controlled by a controller (c). The MLLM (M) receives the instruction (I) and image observations (o) and performs the following: * **Feasibility Analysis:** Determines if the task (I) is achievable in the initial environment state (o0) before execution. * **Progress Assessment:** Monitors the agent's execution by analyzing a subset of past observations to determine if the agent is progressing correctly. If deviation occurs, it provides real-time feedback for correction or resetting. * **Success Detection:** Evaluates whether the agent successfully completed the task by analyzing initial and final observations, comparing the observed outcome with the expected outcome described in I. The MLLM's judgments guide the controller's decisions. Experiments use a dataset created based on the CALVIN framework, excluding tasks involving LEDs, lightbulbs, and sliders due to their complexity. GPT-4 and Google Gemini are evaluated for their performance in these three tasks. The study also analyzes the effects of image resolution (200, 500, 768, and 3072 pixels), prompt structure (CoT, non-CoT, CoT&CI), and the number of input frames on the MLLMs’ performance. Each experiment is repeated three times with 200 samples per run.

The experiments addressed several key questions: the vision reasoning capabilities of MLLMs for automating the three functions, MLLM deficiencies in these tasks, comparison of GPT-4 variants (GPT-4-Turbo, GPT-4-0) and Gemini 1.5 Pro, and the impact of image resolution, prompt structure, and frame input structure. **Feasibility Analysis:** Higher image resolution did not significantly improve performance; however, lower resolution (200 pixels) showed relatively worse results. GPT-4-0 and Gemini 1.5 Pro exceeded 90% accuracy at 768 pixels. GPT-4-Turbo performed poorly. Common errors included unexpected assumptions, scene misunderstanding, and disregard for scene details. GPT-4-0 demonstrated robustness to prompt changes, unlike Gemini 1.5 Pro. **Progress Assessment:** Chain-of-Thought (CoT) prompts generally improved accuracy and reliability for GPT-4-0 compared to simpler prompts. GPT-4-0 consistently outperformed Gemini. Increased levels of "thinking" and "criticism" led to more stringent evaluations. **Success Detection:** Using multiple sequential images for input generally yielded better accuracy than using a single grid image for GPT-4-0. The optimal number of frames was found to be two (first and last frame). While Gemini showed high balanced accuracy it also produced a large number of "Unsure" labels, unlike GPT-40. This highlights a trade-off between accuracy and the number of uncertain responses generated by the LLM.

The findings demonstrate the considerable potential of MLLMs to automate crucial human functions in language-conditioned robotics. The high accuracies achieved in feasibility analysis, progress assessment, and success detection suggest that MLLM-IL can significantly reduce the need for human intervention. This is crucial for scaling language-conditioned robotic systems to real-world applications. The impact of image resolution, prompt engineering, and frame frequency highlights important considerations for optimizing the framework. The differences in performance between GPT-4 variants and Gemini indicate the importance of model selection and potential future improvements in multimodal model development.

This research introduces the MLLM-IL framework, demonstrating the significant potential of MLLMs for automating human functions in language-conditioned robotics. The high accuracy achieved in feasibility analysis, progress assessment, and success detection underscores the framework's effectiveness in improving scalability and efficiency. Future work should focus on optimizing prompt engineering techniques, integrating additional sensory inputs, expanding the range of tasks and environments, and exploring different MLLM architectures.

The study focused on a specific set of tasks from the CALVIN framework, excluding certain tasks due to their complexity. The generalization of the findings to a wider range of tasks and environments requires further investigation. The reliance on existing MLLMs might limit the performance given the ongoing advancements in multimodal model development. Finally, the cost of using large language models could be a limiting factor for wider adoption.