Abstract

This paper introduces text-to-shape-display, a novel approach to generating dynamic shape changes in pin-based shape displays through natural language commands. By leveraging large language models (LLMs) and AI-chaining, our approach allows users to author shape-changing behaviors on demand through text prompts without programming. Key generative elements (primitive, animation, and interaction) are identified, and design requirements for enhancing user interaction are explored based on formative and iterative design processes.

We present SHAPE-IT, an LLM-based authoring tool for a 24 x 24 shape display, which translates textual commands into executable code. The tool allows rapid exploration through a web-based control interface. Evaluations, including performance metrics and a user study with 10 participants, reveal SHAPE-IT’s promise in enabling rapid ideation of diverse shape-changing behaviors. However, the findings highlight accuracy challenges and limitations, paving the way for further refinement of the framework to better suit shape-changing systems.

Contribution

• Introduced text-to-shape-display, integrating LLMs and AI-chaining for authoring shape-changing behaviors in pin-based shape displays through natural language commands.
• Defined foundational generative elements (primitive, animation, and interaction) to support system design and user interaction.
• Developed SHAPE-IT, a web-based authoring tool that translates natural language inputs into executable code for a 24 x 24 pin-based shape display.
• Demonstrated the potential for rapid, on-demand ideation of dynamic shapes and animations without requiring user programming.

Implementation

• Designed a software system integrating natural language processing with shape-changing hardware.
• SHAPE-IT supports the creation of dynamic shape-changing behaviors by translating natural language commands into control scripts.
• Utilized a web-based interface for real-time interaction and control of the 24 x 24 pin-based shape display.

Evaluation

• Performance Evaluation: Assessed SHAPE-IT’s ability to generate accurate and dynamic shape-changing behaviors from varying text complexities.
• User Evaluation: Conducted a study with 10 participants, exploring SHAPE-IT’s usability, versatility, and the effectiveness of natural language-based interaction.
  • Results showed the system’s promise in enabling rapid ideation but identified accuracy-related challenges requiring future improvements.

Abstract

We envision that AI-enabled shape display that renders dynamic 3D shapes leveraging Large Language Models. Unlike conventional shape displays requiring pre-programmed behaviors for shape and animation creation, SHAPE-IT integrates Generative AI (GPT4) with pin-based shape displays, facilitating on-demand and on-the-fly authoring of dynamic shapes. This innovative approach empowers users to dictate shape, motion, and interaction through real-time natural language inputs, bypassing the need for coding. The implementation showcases a software bridge between custom prompts for GPT4 and a Unity-based shape rendering software, generating scripts to control shape display based on user instructions. Applications extend to gaming, entertainment, on-demand teaching aids, adaptive furniture, and controllers, demonstrating a rich potential for this technology. A ten-participant user study sheds light on the promising concept and unveils insights for addressing future research challenges.

Contribution

• The significant contribution of this paper is the integration of Generative AI (specifically GPT4) with pin-based shape displays. This integration allows users to generate dynamic shapes on-demand.
• Users can interact with the system using real-time natural language inputs, which eliminates the need for coding.
• Implemented a software bridge between custom prompts for GPT4 and Unity-based shape rendering software. This bridge facilitates the generation of scripts that control the shape display based on user instructions.

Implementation

• Implemented a software system that processes language instructions and translates them into dynamic and interactive shape-changing behavior.
• The system is a proof-of-concept prototype that integrates: Hardware-based multi-modal interaction using inFORCE(5x10) and GUI-based interaction on Unity.

Evaluation

• A user study with 12 participants was conducted to evaluate the concept of SHAPE-IT. The study revealed the promise of the concept and also provided insights into potential research challenges.
• Performance of the system is evaluated based on text complexity as well as demonstrated the system’s versatility in creating shapes, motions, and interactions.