CocoTagger

   ___                   _____                            
  / __\___   ___ ___   /__   \__ _  __ _  __ _  ___ _ __ 
 / /  / _ \ / __/ _ \    / /\/ _` |/ _` |/ _` |/ _ \ '__|
/ /__| (_) | (_| (_) |  / / | (_| | (_| | (_| |  __/ |   
\____/\___/ \___\___/   \/   \__,_|\__, |\__, |\___|_|   
                                   |___/ |___/            
   🎵  AI-Powered Music Tagging with MusicCocoa Embeddings

Automatically tag audio files with genre, mood, instruments, vocals, texture, and more using semantic similarity in 768-dimensional embedding space.

---

Built with Google's Magenta Realtime 2 | Zero-Shot Capable | Mac M1-M5 Optimized

---

Features

• 178 predefined tags across 10 categories
• Multi-segment analysis: extracts N segments per file (uniform or random)
• Debug mode: saves all similarity scores for analysis
• Batch processing: parallel workers for large collections
• All audio formats: MP3, WAV, FLAC, M4A, OGG, OPUS, AAC, etc.
• Mac M1-M5 optimized: runs on CPU, no GPU needed
• Custom tags: add your own descriptors

Quick Start

# Setup
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Tag single file (models auto-download on first run)
python tag_music.py song.mp3

# Example output (Kendrick Lamar - HUMBLE.)
# Kendrick Lamar - HUMBLE..mp3:
#   chorus-effect        (texture     ) 0.369
#   raspy-vocals         (vocals      ) 0.365
#   delay-effect         (texture     ) 0.360
#   hip-hop              (genre       ) 0.359
#   spoken-word          (vocals      ) 0.358

# Tag directory (batch)
python tag_music.py /music/*.mp3 --workers 4

# Custom parameters
python tag_music.py audio.flac \
  --segments 10 \
  --top-k 15 \
  --extra-tags "shoegaze,post-rock"

Output

Tags saved as JSON in tags/ directory:

Example: Kendrick Lamar - HUMBLE.

{
  "filename": "Kendrick Lamar - HUMBLE..mp3",
  "duration_sec": 177.4,
  "num_segments": 5,
  "segment_strategy": "uniform",
  "aggregation": "weighted_mean",
  "tags": [
    {"name": "chorus-effect",  "category": "texture",  "confidence": 0.369},
    {"name": "raspy-vocals",   "category": "vocals",   "confidence": 0.365},
    {"name": "delay-effect",   "category": "texture",  "confidence": 0.360},
    {"name": "hip-hop",        "category": "genre",    "confidence": 0.359},
    {"name": "spoken-word",    "category": "vocals",   "confidence": 0.358},
    {"name": "dark-timbre",    "category": "texture",  "confidence": 0.355},
    {"name": "anxious",        "category": "mood",     "confidence": 0.351},
    {"name": "turntables",     "category": "instruments", "confidence": 0.350},
    {"name": "muddy",          "category": "texture",  "confidence": 0.345},
    {"name": "bitcrushed",     "category": "texture",  "confidence": 0.344}
  ]
}

Tag Taxonomy

178 tags across 10 categories. All tags are zero-shot — no training required to add new ones via --extra-tags.

Genre (57)

Subgroup	Tags
Core	rock pop jazz classical electronic hip-hop country blues reggae metal folk punk soul funk disco techno house dubstep drum-and-bass ambient indie alternative r-and-b gospel latin world experimental soundtrack new-age
Cinematic	orchestral cinematic epic minimalist neoclassical
Ambient / Atmospheric	dark-ambient drone atmospheric soundscape
Art / Progressive	post-rock post-metal math-rock art-rock shoegaze
Electronic subgenres	IDM glitch trip-hop grime
World	bossa-nova samba flamenco afrobeats
Retro / Garage	krautrock garage-rock psychedelic-rock
Experimental	avant-garde noise musique-concrete

Mood (24)

happy sad energetic calm aggressive melancholic uplifting dark dreamy intense playful romantic mysterious anxious euphoric nostalgic peaceful angry hopeful chill tense bittersweet triumphant hypnotic

Instruments (27)

Subgroup	Tags
String	guitar electric-guitar acoustic-guitar bass violin cello harp mandolin banjo ukulele pedal-steel
Keys / Synths	piano organ synthesizer keyboard
Wind	saxophone trumpet flute clarinet trombone tuba
Rhythm / Electronic	drums percussion 808 turntables
World	sitar oud accordion

Vocals (12)

male-vocals female-vocals clean-vocals harsh-vocals choir spoken-word vocal-harmony layered-vocals no-vocals instrumental-only raspy-vocals falsetto

Texture (14)

distorted clean-tone reverb-heavy chorus-effect delay-effect warm bright dark-timbre muddy compressed filtered bitcrushed sparse dense

Harmonic (9)

minor-key major-key modal atonal dissonant chromatic pentatonic blues-scale consonant

Tempo (5)

very-slow slow moderate fast very-fast

Era (7)

vintage retro modern contemporary futuristic timeless 2000s

Production (17)

acoustic electric live studio lo-fi hi-fi raw polished sample-heavy synthesizer-based guitar-driven bass-heavy groove-oriented analog-recording digital-recording 8-bit field-recording

Energy (5)

building steady fading explosive meditative

CLI Options

--segments N              Number of segments per file (default: 5)
--segment-length S        Segment length in seconds (default: 10.0)
--segment-strategy S      Segment position: uniform (default) or random
--aggregation A           Embedding merge: weighted_mean (default), mean, max_pool
--top-k K                 Number of top tags to return (default: 10)
--workers W               Parallel workers for batch processing (default: 1)
--output-dir DIR          Output directory for JSON files (default: tags)
--extra-tags TAGS         Comma-separated custom tags
--rebuild-cache           Rebuild tag embedding cache
--cache-audio             Cache audio embeddings to disk (default: on)
--no-audio-cache          Disable audio embedding cache
--save-debug              Save all similarity scores to debug/ (default: on)
--no-debug                Disable debug output

How It Works (Simple Explanation)

The Process

1. Extract segments (ffmpeg)

   • Pick 5 evenly-spaced 10-second segments from your song (uniform strategy)
   • Example: 3-min song → clips at 0:00, 0:42, 1:25, 2:07, 2:50

2. Convert audio to meaning (MusicCocoa audio encoder)

   • Each 10s clip → 768 numbers representing its "musical meaning"
   • Merge 5 clips via variance-weighted mean → single 768-number vector
   • This vector captures genre, mood, instruments, texture, etc.

3. Convert tags to meaning (MusicCocoa text encoder, done once)

   • Each tag ("rock", "aggressive") → 768 numbers
   • Pre-computed at startup, cached for reuse

4. Compare similarity (vectorised cosine similarity)

   • Single matrix multiply: audio vector vs all 178 tag vectors at once
   • Score from -1 (opposite) to +1 (identical)

5. Pick winners

   • Sort tags by similarity score
   • Top 10 = best matches for your song

Example 1: Kendrick Lamar - HUMBLE.

What you hear: Hard-hitting hip-hop beat, aggressive vocals, heavy bass, distorted production

Terminal output:

Kendrick Lamar - HUMBLE..mp3:
  chorus-effect        (texture     ) 0.369
  raspy-vocals         (vocals      ) 0.365
  delay-effect         (texture     ) 0.360
  hip-hop              (genre       ) 0.359
  spoken-word          (vocals      ) 0.358
  dark-timbre          (texture     ) 0.355
  anxious              (mood        ) 0.351
  turntables           (instruments ) 0.350
  muddy                (texture     ) 0.345
  bitcrushed           (texture     ) 0.344

What the model captured:

• Texture: Chorus/delay effects, dark timbre, muddy and bitcrushed signal chain — nails the heavy, processed production
• Vocals: Raspy vocals + spoken-word delivery — accurate for Kendrick's style on this track
• Genre: Hip-hop correctly identified
• Instruments: Turntables — picks up the DJ scratching and sample-based production
• Mood: Anxious — captures the tense, confrontational energy

Example 2: Erykah Badu - Rim Shot (rs.mp3)

What you hear: Neo-soul vocals, emotional intensity, R&B production

Terminal output:

rs.mp3:
  angry                (mood        ) 0.321
  raspy-vocals         (vocals      ) 0.298
  female-vocals        (vocals      ) 0.288
  soul                 (genre       ) 0.284
  falsetto             (vocals      ) 0.276
  fading               (energy      ) 0.272
  chorus-effect        (texture     ) 0.270
  r-and-b              (genre       ) 0.259
  delay-effect         (texture     ) 0.248
  vocal-harmony        (vocals      ) 0.243

What the model captured:

• Vocals: Female vocals, raspy delivery, falsetto, vocal harmony — richly detailed vocal profile with no duplicate tags
• Genre: Soul (0.284) and R&B (0.259) — accurate for neo-soul
• Mood: Angry/intense emotional delivery
• Energy: Fading — picks up the song's dynamic arc
• Note: Previously showed female-vocalist (instruments) and female-vocals (vocals) as near-duplicates; the cleaned taxonomy now surfaces more distinct, useful tags instead

Example 3: Björk - All Is Full of Love

What you hear: Electronic art-pop, ethereal vocals, experimental production

Terminal output:

björk - all is full of love.mp3:
  delay-effect         (texture     ) 0.292
  chorus-effect        (texture     ) 0.247
  fading               (energy      ) 0.245
  trip-hop             (genre       ) 0.235
  raspy-vocals         (vocals      ) 0.234
  art-rock             (genre       ) 0.229
  musique-concrete     (genre       ) 0.229
  falsetto             (vocals      ) 0.225
  spoken-word          (vocals      ) 0.219
  female-vocals        (vocals      ) 0.212

What the model captured:

• Texture: Delay and chorus effects dominate — accurate for the heavily processed vocal/synth mix
• Genre: Trip-hop (0.235), art-rock (0.229), musique-concrete (0.229) — captures the experimental, electronic nature
• Vocals: Falsetto, spoken-word, female vocals — detailed and accurate vocal profile
• Energy: Fading — reflects the slow, dissolving quality of the track

Why Multiple Segments?

Songs vary over time (intro ≠ chorus ≠ outro). Taking 5 evenly-spaced segments gives a reproducible, balanced representation of the whole track. Embeddings are merged via variance-weighted mean — segments with more "confident" signal contribute more than near-silent passages.

Technical Deep Dive

Architecture

Audio Pipeline:

Input audio file (any format)
    ↓
ffmpeg extraction → 5 uniform 10s segments (16kHz mono, float32)
    ↓
audio_preprocessor.tflite → mel spectrograms
    ↓
music_encoder.tflite (12-layer ViT, 8 CPU threads) → 5 × 768-dim embeddings
    ↓
Variance-weighted mean → 1 × 768-dim audio embedding (L2-normalized)

Text Pipeline:

Tag text ("hip-hop", "aggressive", etc.)
    ↓
SentencePiece tokenization → token IDs
    ↓
text_encoder.tflite (12-layer Transformer, 8 CPU threads) → 768-dim text embedding (L2-normalized)
    ↓
Stacked into (178, 768) matrix → cached to disk (tag_embeddings.pkl)

Similarity Computation:

# Single matrix multiply — all 178 tags at once
scores = tag_matrix @ audio_embedding   # (178, 768) @ (768,) → (178,)
# Range: -1.0 (opposite) to +1.0 (identical)
# Typical music scores: 0.2 to 0.4

Why This Works

Contrastive Learning: MusicCocoa was trained on 44M audio-text pairs to make similar-sounding music close to matching text descriptions in 768-dimensional space. The model learned:

• "aggressive hip-hop beat" → similar vector to aggressive hip-hop audio
• "calm piano ballad" → different vector from aggressive hip-hop audio

Semantic Space: The 768 dimensions capture abstract musical concepts:

• Some dimensions → tempo (slow vs fast)
• Some dimensions → timbre (distorted vs clean)
• Some dimensions → mood (dark vs bright)
• Combination of dimensions → genre, instruments, production style

Zero-Shot Capability: No fine-tuning needed. Any text description works because training used natural language from YouTube (not rigid taxonomies). Could tag with custom phrases like "80s synth solo" or "underwater bass" without retraining.

Model Details

MusicCocoa (part of Magenta RealTime 2):

• Based on Google MuLan (Music Language Model)
• Architecture: CoCa (Contrastive Captioners)
• Audio: 12-layer Vision Transformer over log-mel spectrograms
• Text: 12-layer Transformer with SentencePiece tokenization
• Training: 44M YouTube music videos (370K hours)
• Objective: Contrastive loss (audio-text pairs attract, non-pairs repel)

Files:

• audio_preprocessor.tflite (8.3 MB): Waveform → mel spectrogram
• music_encoder.tflite (353 MB): Mel spec → 768-dim audio embedding
• text_encoder.tflite (399 MB): Text tokens → 768-dim text embedding
• spm.model (505 KB): SentencePiece tokenizer

Performance Characteristics

Accuracy:

• Well-defined genres (hip-hop, soul, jazz): high confidence (0.28–0.4)
  • Example: Kendrick HUMBLE. → hip-hop 0.359, chorus-effect 0.369
• Niche/experimental genres (art-rock, musique-concrete, trip-hop): moderate (0.2–0.3)
  • Example: Björk → trip-hop 0.235, art-rock 0.229, musique-concrete 0.229
• Vocals: very reliable when vocals are prominent (0.21–0.37)
  • Example: Erykah Badu → female-vocals 0.288, falsetto 0.276, vocal-harmony 0.243
• Mood: consistent for emotionally clear tracks (0.25–0.35)
  • Example: Erykah Badu → angry 0.321; Kendrick → anxious 0.351

Speed (8-core CPU, XNNPACK):

• Tag embedding (178 tags): ~60s first run, then cached permanently
• Single file, cold (no audio cache): ~1s
• Single file, warm (audio cache hit): ~0.6s
• Batch (100 files, 4 workers): ~2–3 minutes

Memory:

• Models loaded: ~800 MB
• Peak inference: ~1.2 GB
• Tag cache: ~1 MB

Requirements

• Python 3.11+
• ffmpeg (install via brew install ffmpeg)
• Mac M1-M5 (or any Mac/Linux with CPU)

Performance

Runs entirely on CPU — no GPU needed. Optimized for Apple Silicon but works on any modern CPU.

Mac M1 Pro timing (actual):

time python tag_music.py song.mp3
# 16.16s user, 2.37s system, 19.1s total

Breakdown per file:

• Model loading: ~1s (first time only)
• Segment extraction: ~2s (ffmpeg I/O)
• Audio encoding: ~14s (5 segments × 2.8s each)
• Tag similarity: <0.1s
• Total: ~19 seconds per song

Batch processing:

• 10 files: ~3 minutes
• 100 files (4 workers): ~8 minutes
• 1000 files (4 workers): ~80 minutes (~5s/file amortized)

Models

Auto-downloaded on first run from google/magenta-realtime-2:

• audio_preprocessor.tflite (8.3 MB)
• music_encoder.tflite (353 MB)
• text_encoder.tflite (399 MB)
• spm.model (505 KB)

Total: 761 MB (downloaded to models/ directory)

Training Data

MusicCocoa is based on Google's MuLan model, trained on:

• 44 million music recordings (370,000 hours)
• Natural language descriptions from YouTube (titles, descriptions, tags, playlists)
• Zero-shot capability: works with any text description, not just predefined tags

This allows flexible tagging beyond rigid taxonomies.

Debugging & Analysis

Check all similarities:

cat tags/debug/song.debug.json | jq '.all_similarities | sort_by(-.score) | .[:20]'

Find why a tag scored low:

cat tags/debug/song.debug.json | jq '.all_similarities[] | select(.tag=="cinematic")'

Compare two songs:

# Extract top genres for comparison
jq '.all_similarities[] | select(.category=="genre")' tags/debug/song1.debug.json
jq '.all_similarities[] | select(.category=="genre")' tags/debug/song2.debug.json

Limitations

• 10-second minimum: Songs shorter than 10s are zero-padded to fill the model's fixed input window, which may reduce embedding quality
• CPU-only: No GPU acceleration — TFLite runs on CPU via XNNPACK (MLX/CoreML conversion would use Apple Neural Engine and speed things up further)
• English-centric: Text encoder was trained primarily on English YouTube descriptions; non-English genre/mood names may embed less accurately
• Genre ambiguity: The model reflects real-world genre overlap — a track can legitimately score high on both trip-hop and art-rock; this is a feature as much as a limitation
• Averaged representation: Segments are merged into a single embedding — per-segment tagging (verse vs chorus vs outro) is not yet supported

Next Steps

• Temporal segmentation: Analyze song structure (intro/verse/chorus/bridge) and tag each segment separately
• Per-segment tagging: Track how tags evolve throughout the song (e.g., calm intro → energetic chorus)
• React UI with drag-drop upload
• Export to CSV/SQLite for music library integration
• CoreML conversion for faster Mac inference
• Custom tag training from user-provided examples

Contributing

Contributions welcome! Areas for improvement:

• Performance: CoreML/MLX conversion for faster inference
• Features: Temporal analysis, per-segment voting, bulk export formats
• Tags: Expand tag taxonomy, add language support beyond English
• Testing: Validate accuracy on diverse music datasets
• Documentation: Add Jupyter notebooks with examples

Submit issues or PRs at github.com/shoegazerstella/cocotagger

License

MIT

Citation

If you use this tool in research, cite the underlying model:

@inproceedings{mulan2022,
  title={MuLan: A Joint Embedding of Music Audio and Natural Language},
  author={Huang, Qingqing and Jansen, Aren and Lee, Joonseok and Ganti, Ravi and Li, Judith Yue and Ellis, Daniel P. W.},
  booktitle={ISMIR},
  year={2022}
}