Uncovering the Token Splitting Effect in Soft Prompts for Multi-Model LLM Training
Published:
Prompt tuning in natural language processing enables efficient utilization of Large Language Models (LLMs), but soft prompts often struggle with interpretability. This study introduces a novel multi-model training methodology for soft prompts, validated across the MultiBERTs collection using IMDb, Emotion, and MNLI datasets. We uncover the token splitting effect in soft prompts, a phenomenon where individual prompt tokens align with specific models within their embedding spaces, significantly impacting performance. Our findings reveal that post-training prompt compression enhances efficiency with minimal performance loss. We thereby advance the understanding of soft prompt behavior in multi-model settings, offering pathways for resource-efficient optimization and strategic compression in Large Language Models.