Uncovering the Token Splitting Effect in Soft Prompts for Multi-Model LLM Training
Published in SKILL24, 2024
Prompt tuning in natural language processing enables efficient utilization of Large Language Models (LLMs), but soft prompts often struggle with interpretability. This study introduces a novel multi-model training methodology for soft prompts, validated across the MultiBERTs collection using IMDb, Emotion, and MNLI datasets. We uncover the token splitting effect in soft prompts, a phenomenon where individual prompt tokens align with specific models within their embedding spaces, significantly impacting performance. Our findings reveal that post-training prompt compression enhances efficiency with minimal performance loss. We thereby advance the understanding of soft prompt behavior in multi-model settings, offering pathways for resource-efficient optimization and strategic compression in Large Language Models.