Overview
This study investigates the language processing mechanisms of generative AI through an experiment using the Japanese colloquial expression “なんてったって (nante-ttatte)”. By analyzing how different AI models process and respond to this culturally embedded phrase, we gained insights into their internal processing mechanisms and potential implications for AI literacy education.
Experimental Design and Results
Experimental Setup
We conducted an automated experiment using ChatGPT (GPT-4o) and Google Gemini (gemini-1.5-pro). Each model was tasked with answering “なんてったって・・?” in two tokens with 1,000 sampling iterations.
Key Results
- ChatGPT consistently output “アイドル (idol)” with 100% probability
- Gemini showed different patterns:
- “最高 (best)” related expressions: 97.7%
- “アイドル (idol)” related expressions: 2.1%
Key Findings
Cultural Processing Patterns
The stark difference in responses between the two models revealed interesting patterns in how AI processes cultural context. ChatGPT’s consistent output reflects strong cultural learning from its training data, specifically the famous 1985 song “なんてったってアイドル” by Kyoko Koizumi.
Language Processing Mechanism
Our analysis suggests that Gemini might be processing Japanese through English as an intermediate language:
- Converting “なんてったって” to “After all”
- Generating “After all, it’s the best!”
- Translating back to “なんてったって最高”
This finding provides valuable insights into how multilingual AI models might process language internally.
Educational Implications
For AI Literacy Education
- Understanding how AI models process cultural context
- Recognizing potential intermediate language processing
- Appreciating the importance of prompt design
For Language Education
- Insights into AI’s language processing mechanisms
- Understanding cultural context in AI responses
- Practical applications in language teaching
