System Design: Video Conferencing with Transcription and AI

Introduction

This system combines real-time communication, speech-to-text, and AI inference — three workloads with completely different latency and infrastructure requirements running simultaneously. Getting the architecture wrong means captions lag, meetings drop, or AI costs bankrupt the product.

Step 1: Requirements

Functional:

1. Video conferencing — multiple participants, real-time
2. Real-time transcription — speech to text during the call
3. AI inference during call — live captions, speaker identification, sentiment analysis
4. Post-call processing — full transcript, AI summary, action items, searchable transcript
5. Storage — recordings, transcripts, AI-generated insights
6. Search across all past meetings

Non-functional:

1. Captions appear within 500ms of speech
2. High availability — meetings cannot drop
3. Scale — 50,000 teams
4. AI processing must not slow down the call
5. Security — meetings are private and sensitive
6. Cost efficiency — AI inference is expensive at scale

Step 2: Scale Estimation

50,000 teams
Average team size: 20 people
Average meetings per team per day: 5
Average meeting duration: 45 minutes

Meetings per day: 250,000
Peak concurrent (20% overlap): 50,000 concurrent meetings

Audio streams for transcription:
50,000 meetings × 5 active speakers = 250,000 audio streams

Transcription volume:
45 min × 150 words/min × 5 speakers = 33,750 words/meeting
250,000 meetings × 33,750 = ~8 billion words/day

Recording storage:
250,000 meetings × 500 MB = 125 TB/day

This tells you:

• Real-time audio processing is the hardest challenge
• AI inference at 250K concurrent streams is expensive
• Storage is massive — S3 mandatory
• Search across billions of words needs Elasticsearch

Step 3: Three Phases, Three Architectures

The system has three distinct phases with different requirements:

Phase 1 — During Meeting (Real Time):
→ Video/audio transmission
→ Real-time transcription
→ Live captions (< 500ms)

Phase 2 — End of Meeting (Near Real Time):
→ Full transcript assembly
→ AI summary, action items
→ Recording storage (< 2 minutes)

Phase 3 — Post Meeting (Async):
→ Search indexing
→ Analytics, topic modeling
→ Insights delivery (minutes to hours)

Each phase uses different infrastructure optimized for its latency requirement. Mixing them makes all three worse.

Step 4: Video Conferencing Layer

WebRTC

Industry standard for real-time video — browser-native, end-to-end encrypted, used by Google Meet, Zoom, Teams.

2-person call:
→ Direct peer-to-peer
→ No server needed for media
→ Lowest latency

3+ person call:
→ P2P doesn't scale — 10 participants = 45 connections each ❌
→ Need a media server (SFU)

Selective Forwarding Unit (SFU)

Participant A sends video → SFU
Participant B sends video → SFU
Participant C sends video → SFU

SFU selectively forwards:
→ A receives B and C streams
→ B receives A and C streams
→ C receives A and B streams

Each participant:
→ Uploads 1 stream (their own)
→ Downloads N-1 streams (others)

SFU doesn’t mix or process — just routes. Very low CPU, very low latency. Used by Discord, Zoom, Twilio.

SFU Infrastructure at Scale

50,000 concurrent meetings × 5 participants = 250,000 video streams
Each stream: ~1-2 Mbps → 375 Gbps total bandwidth

Single SFU server: ~500 concurrent meetings
50,000 / 500 = 100 SFU servers at peak

Geographic distribution:
→ Deploy SFU in every major region (Mumbai, Singapore, Frankfurt, Virginia)
→ User connects to nearest SFU — latency sensitive

Step 5: Real-Time Transcription Pipeline

Audio Extraction

For transcription you need raw audio, not video. SFU already has all audio streams.

SFU → Audio Tap Service
→ Extract audio stream per participant
→ Convert to PCM 16kHz mono (speech models trained on this format)
→ Send to transcription pipeline

Distributed Transcription Workers

250,000 audio streams → cannot send all to one service
Each stream → dedicated transcription worker

Transcription Worker:
→ Receives audio chunks (every 100ms)
→ Sends to Speech-to-Text model (Whisper)
→ Publishes to Kafka "transcription.chunks":
  {
    meetingId, participantId,
    text: "let's discuss the Q3 results",
    timestamp, confidence: 0.94
  }

Speech-to-Text: Cloud vs Self-Hosted

Cloud API (Google/AWS/Azure):
250K streams × 45 min = 11.25M minutes/day
At $0.006/min = ~$67,500/day = $2M/month

Self-hosted Whisper on GPU (A100):
Each GPU handles ~100 concurrent streams
250,000 / 100 = 2,500 GPUs at peak
= ~$180,000/day

Reality:
→ Start with cloud APIs — simpler, no ML ops
→ Switch to self-hosted when volume justifies it
→ Use spot/preemptible GPU instances
→ Scale fleet up during peak, down off-peak

Streaming Transcription Flow

Audio chunk (100ms) arrives
      ↓
Transcription Worker (GPU) → partial transcript
      ↓
Kafka "transcription.chunks"
      ↓
      ├── Real Time Caption Service
      │   → WebSocket → participants see captions (200-500ms latency) ✅
      │
      └── Transcript Assembler
          → Collects chunks for meeting
          → Assembles ordered transcript in Redis
          → Persisted to Cassandra at meeting end

Step 6: AI Inference Pipeline

Three types of AI processing with different latency requirements:

Type 1 — Real Time (During Call)

Speaker identification:

Each participant voice-fingerprinted at call start → stored in Redis
Each audio chunk → compare fingerprint → tag transcript with speaker name
Must be fast — < 200ms

Live sentiment analysis:

Text sentiment model on transcript chunks
→ Update meeting sentiment score in Redis
→ Dashboard shows real-time meeting health

Type 2 — Near Real Time (End of Call)

Triggered by Kafka event meeting.ended:

Kafka "meeting.ended"
      ↓
AI Processing Queue (Kafka)
      ↓
Parallel AI Workers:

Worker 1 — Summarization:
→ Full transcript → LLM → 3-paragraph summary (30-60 sec)

Worker 2 — Action Items:
→ Full transcript → LLM → structured JSON
  { task: "prepare Q3 report", assignee: "Raj", deadline: "Friday" }

Worker 3 — Key Decisions:
→ Extract decisions made in meeting

Worker 4 — Meeting Score:
→ Aggregate real-time sentiment → overall effectiveness score

All workers run in parallel
Results ready within 60-90 seconds
→ Notification: "Your meeting summary is ready"

Type 3 — Async (Background)

Search Indexing:
→ Full transcript → Elasticsearch
→ "Find all meetings where we discussed pricing"

Topic Modeling:
→ Auto-tag meeting with topics
→ Enables filtering and analytics

Speaker Analytics:
→ Talk time distribution per participant
→ Trends over time per team

Step 7: Data Model

Meeting Record (PostgreSQL)

meetings:
  meeting_id, team_id, title, host_user_id
  started_at, ended_at, duration_seconds
  participant_count, recording_url, status
  (scheduled / live / ended / processing / ready)

Why PostgreSQL: relational (meetings belong to teams), complex queries (“all meetings for team X this month”), manageable volume at 250K meetings/day.

Transcript Storage (Cassandra)

transcripts:
  meeting_id     → partition key
  chunk_sequence → sort key
  speaker_id, speaker_name, text
  timestamp_ms, confidence, sentiment_score

Why Cassandra: 8 billion words/day, access pattern is “all chunks for meeting X”, time series ordered by chunk_sequence, append-only.

Live assembly in Redis during meeting:

Key: "transcript:meetingId"
Value: ordered list of chunks
→ Fast assembly during live meeting
→ Persisted to Cassandra at meeting end
→ Redis key deleted after persistence confirmed

AI Outputs (PostgreSQL)

meeting_insights:
  meeting_id, summary, action_items (JSON)
  key_decisions (JSON), topics (array)
  sentiment_score, processing_status, created_at

One record per meeting. Complex queries: “all action items assigned to Raj this week.”

Search Index (Elasticsearch)

{
  "meeting_id", "team_id", "title", "date",
  "participants": ["Raj", "Priya", "Kumar"],
  "full_transcript_text",
  "summary", "topics": ["Q3", "sales", "product"],
  "action_items_text"
}

Enables: “Find meetings about Q3 planning”, “Show meetings where Kumar spoke”, “Find pricing discussions last month.”

Step 8: Recording Storage

During meeting:
SFU records mixed audio/video → stored temporarily on SFU server

After meeting ends:
→ Upload to S3: s3://recordings/teamId/meetingId/recording.mp4
→ Update PostgreSQL with S3 URL

Cost optimization (S3 lifecycle):
→ Recent (< 30 days): S3 Standard
→ 30-180 days: S3 Infrequent Access (40% cheaper)
→ Archive (> 180 days): S3 Glacier (80% cheaper)

CDN for playback:
→ CloudFront in front of S3
→ Frequently accessed recordings served from edge
→ Never hammers S3 directly

Step 9: Security and Privacy

Encryption in transit:
→ WebRTC: DTLS-SRTP (built in)
→ API calls: TLS 1.3
→ Kafka: TLS encrypted topics

Encryption at rest:
→ S3: AES-256, Database: encrypted at rest, Redis: TLS

Access control:
→ Only meeting participants access transcript
→ Team admin accesses all team meetings
→ Row-level security in PostgreSQL per team_id

AI data privacy:
→ Transcripts not sent to third-party AI without consent
→ Self-hosted models option for sensitive teams
→ Data residency — EU data stays in EU

Compliance (GDPR):
→ User requests deletion → purge from PostgreSQL, Cassandra,
  Elasticsearch, S3. Kafka events anonymised.

Step 10: Complete Architecture

[DURING MEETING]

Participants (Browser/Mobile)
      ↓ WebRTC
Regional SFU Servers (100 servers globally)
      ↓
Audio Tap Service
      ↓
Audio Chunks → Kafka "audio.chunks"
      ↓
Transcription Workers (GPU fleet — Whisper)
      ↓
Kafka "transcription.chunks"
      ↓
      ├── Real Time Caption Service → WebSocket → live captions
      ├── Transcript Assembler → Redis (live assembly)
      └── Real Time AI Service → speaker ID, sentiment → Redis

[END OF MEETING]

Kafka "meeting.ended"
      ↓
      ├── Recording Service → S3 → update PostgreSQL
      ├── Transcript Persistence → Redis → Cassandra → Elasticsearch
      └── AI Processing Queue
          ↓
          Parallel Workers: Summary, Action Items, Topics
          ↓
          Notification: "Your meeting summary is ready"

[USER ACCESSES MEETING DATA]

App Server
      ↓
      ├── Meeting metadata → PostgreSQL
      ├── Transcript → Cassandra
      ├── AI Insights → PostgreSQL
      ├── Recording → CDN → S3
      └── All cached in Redis (TTL 1 hour)

User searches:
→ Search Service → Elasticsearch → meeting IDs → PostgreSQL for details

Step 11: Cost Optimization

AI inference is the biggest cost. Manage it with tiers:

1. Tiered transcription quality:
   Free tier → cheaper model
   Paid tier → Whisper large
   Enterprise → dedicated GPU instances

2. Async where possible:
   Real-time captions → expensive GPU (must be instant)
   Post-meeting summary → can take 60 sec → cheaper batch GPU

3. Smart scaling:
   9 AM - 6 PM → full GPU fleet
   Midnight → 10% of fleet

4. Cache AI results:
   Same summary requested multiple times → Redis cache
   AI never called twice for same content

5. Batch processing:
   Topics, analytics → batch every hour, not per meeting
   Higher GPU utilization, lower cost per meeting

Everything Connects

Key Takeaways

1. Three latency tiers — real time (< 500ms), near real time (< 2 min), async (hours) — must use separate infrastructure. Never put batch AI on the hot path.
2. SFU, not P2P, for group calls. Each participant uploads one stream, downloads N-1. Route via nearest regional SFU.
3. Kafka sits between audio capture and transcription — same shock-absorber pattern as the feed system. Spikes when many meetings start simultaneously never hit GPU workers directly.
4. Cassandra for transcripts (billions of words/day, partition by meeting_id). PostgreSQL for meetings and insights (relational, complex queries).
5. Self-hosted Whisper vs cloud APIs is a cost decision, not a correctness decision. Start cloud, migrate when volume justifies ML ops.
6. AI is the dominant cost — tier quality by plan, batch what you can, cache results, scale GPU fleet with meeting load.

Part of the system design series.