Language-Guided Multimodal Texture Authoring
Type “gritty but cushioned” and feel it: one prompt generates sliding vibration, tapping, and a matching visual.
Wanli Qian, Aiden Chang, Shihan Lu, Michael Gu, Heather Culbertson
Overview
Designing a realistic haptic texture usually means tuning low-level signal parameters by trial and error. This project replaces that with language: type what a surface should feel like, such as “gritty but cushioned” or “smooth hard metal”, and immediately see and feel the result through a 3D haptic stylus.
A single prompt generates three coordinated outputs: sliding vibrations that respond to how hard and fast you move, tapping transients that convey hardness, and a matching visual preview. A shared, language-aligned latent space keeps what you feel consistent with what you typed and what you see.
Highlights
- One prompt drives both haptic channels and a visual preview.
- A shared, language-aligned latent space keeps the haptic and visual output semantically consistent.
- User studies show that prompts give perceptually meaningful control over roughness, hardness, and slipperiness.
How It Works
- Text to haptic latent. A frozen CLIP text encoder and a lightweight projection head map the prompt to a 64-dimensional haptic latent.
- Two haptic decoders. One decoder outputs a tap bank (acceleration transients at 13 impact speeds) for hardness cues. The other outputs an AR matrix: autoregressive vibration models over an 18-point force–speed grid for sliding texture.
- Training. The model is trained on the 100 measured materials of the Penn Haptic Texture Toolkit, augmented to 2,000 training pairs, with five LLM-written captions per material. The loss combines reconstruction, KL regularization, and a contrastive (InfoNCE) term that aligns the latent with the text embedding.
- Visual preview. An off-the-shelf Stable Diffusion 2 pipeline turns the same prompt into a texture image. It is decoupled from the haptic latent, so the two channels are linked through the prompt rather than through shared weights.
- Rendering. A 3D Systems Touch device runs at 1 kHz. Tap responses are interpolated linearly over impact speed, and AR models are interpolated barycentrically over the user’s current force and speed.
Three prompts, three modalities. Top: diffusion-generated image. Middle: 13 synthesized tapping responses (brighter = faster impact). Bottom: pole distributions of the generated sliding-vibration AR models.
Results
- An organized material space. The learned latent space groups the 100 training materials cleanly, in close agreement with their labels.
- Blends that make sense. In a study with 17 people, participants rated textures generated between two real materials. Their ratings followed consistent trends: roughness tracked surface bumps, hardness tracked compliance, and slipperiness tracked surface films.
- Low-effort authoring. People found prompt-based iteration easy, felt that the haptic and visual outputs fit together, and agreed that the textures matched their descriptions.
This model is also the text-to-haptics engine behind TouchTwin, which applies it to entire 3D-scanned scenes.
