Building a Conversational Chatbot for Urdu: A Transformer Implementation from Scratch

 

Introduction

I built a conversational chatbot for Urdu using custom Transformer architecture in PyTorch. This article covers the key challenges, solutions, and lessons learned.

Tech Stack:

• PyTorch 2.0
• Custom Transformer (no pre-trained models)
• 15,000 Urdu conversation pairs
• Trained on Kaggle GPU (3.5 hours)

Links:

• 📂 GitHub: https://github.com/asheeradnan/urdu-chatbot-transformer
• 📓 Kaggle: https://www.kaggle.com/code/muhammadasheeradnan/nlp-urdu-urdu-conversational-chatbot-assign-2

The Challenge

Problem: Build an Urdu chatbot without pre-trained models.

Dataset Issue: Available data had 20,000 individual sentences, not conversation pairs.

Solution: Created 15,000 pairs using:

• 1,000 rule-based conversations (6.7%) - greetings, thanks, etc.
• 3,000 question-statement pairs (20%)
• 11,000 sequential pairs (73.3%)

---

Architecture

Code

Transformer Model:
├── Vocabulary: ~10,000 Urdu words
├── Embedding: 256 dimensions
├── Attention Heads: 2
├── Encoder Layers: 2
├── Decoder Layers: 2
├── Parameters: ~12M
└── Dropout: 0.2

Key Features:

• Multi-head attention
• Positional encoding
• Teacher forcing during training
• BLEU-based model selection

---

Training

Configuration:

Python

BATCH_SIZE = 64
LEARNING_RATE = 3e-4
EPOCHS = 15
OPTIMIZER = Adam

Hardware: Kaggle GPU (NVIDIA T4)
Time: 3.5 hours

---

Results

Metrics

Metric	Score
BLEU	4.45
Train Loss	6.74 → 3.45 (↓48.8%)
Val Loss	6.24 → 4.82 (↓22.8%)

Key Achievement: 28x BLEU improvement (0.16 → 4.45) through data engineering

Introduction

I built a conversational chatbot for Urdu using custom Transformer architecture in PyTorch. This article covers the key challenges, solutions, and lessons learned.

Tech Stack:

• PyTorch 2.0
• Custom Transformer (no pre-trained models)
• 15,000 Urdu conversation pairs
• Trained on Kaggle GPU (3.5 hours)

Links:

• 🔗 Live Demo: [Gradio Link]
• 📂 GitHub: https://github.com/asheeradnan/urdu-chatbot-transformer
• 📓 Kaggle: [Your notebook link]

---

The Challenge

Problem: Build an Urdu chatbot without pre-trained models.

Dataset Issue: Available data had 20,000 individual sentences, not conversation pairs.

Solution: Created 15,000 pairs using:

• 1,000 rule-based conversations (6.7%) - greetings, thanks, etc.
• 3,000 question-statement pairs (20%)
• 11,000 sequential pairs (73.3%)

---

Architecture

Code

Transformer Model:
├── Vocabulary: ~10,000 Urdu words
├── Embedding: 256 dimensions
├── Attention Heads: 2
├── Encoder Layers: 2
├── Decoder Layers: 2
├── Parameters: ~12M
└── Dropout: 0.2

Key Features:

• Multi-head attention
• Positional encoding
• Teacher forcing during training
• BLEU-based model selection

---

Training

Configuration:

Python

BATCH_SIZE = 64
LEARNING_RATE = 3e-4
EPOCHS = 15
OPTIMIZER = Adam

Hardware: Kaggle GPU (NVIDIA T4)
Time: 3.5 hours

---

Results

Metrics

Metric	Score
BLEU	4.45
Train Loss	6.74 → 3.45 (↓48.8%)
Val Loss	6.24 → 4.82 (↓22.8%)

Key Achievement: 28x BLEU improvement (0.16 → 4.45) through data engineering