{“content”:”---\nname: heartmula\ndescription: Set up and run HeartMuLa, the open-source music generation model family (Suno-like). Generates full songs from lyrics + tags with multilingual support.\nversion: 1.0.0\nmetadata:\n hermes:\n tags: [music, audio, generation, ai, heartmula, heartcodec, lyrics, songs]\n related_skills: [audiocraft]\n---\n\n# HeartMuLa - Open-Source Music Generation\n\n## Overview\nHeartMuLa is a family of open-source music foundation models (Apache-2.0) that generates music conditioned on lyrics and tags. Comparable to Suno for open-source. Includes:\n- HeartMuLa - Music language model (3B/7B) for generation from lyrics + tags\n- HeartCodec - 12.5Hz music codec for high-fidelity audio reconstruction\n- HeartTranscriptor - Whisper-based lyrics transcription\n- HeartCLAP - Audio-text alignment model\n\n## When to Use\n- User wants to generate music/songs from text descriptions\n- User wants an open-source Suno alternative\n- User wants local/offline music generation\n- User asks about HeartMuLa, heartlib, or AI music generation\n\n## Hardware Requirements\n- Minimum: 8GB VRAM with --lazy_load true (loads/unloads models sequentially)\n- Recommended: 16GB+ VRAM for comfortable single-GPU usage\n- Multi-GPU: Use --mula_device cuda:0 --codec_device cuda:1 to split across GPUs\n- 3B model with lazy_load peaks at ~6.2GB VRAM\n\n## Installation Steps\n\n### 1. Clone Repository\nbash\ncd ~/ # or desired directory\ngit clone https://github.com/HeartMuLa/heartlib.git\ncd heartlib\n\n\n### 2. Create Virtual Environment (Python 3.10 required)\nbash\nuv venv --python 3.10 .venv\n. .venv/bin/activate\nuv pip install -e .\n\n\n### 3. Fix Dependency Compatibility Issues\n\nIMPORTANT: As of Feb 2026, the pinned dependencies have conflicts with newer packages. Apply these fixes:\n\nbash\n# Upgrade datasets (old version incompatible with current pyarrow)\nuv pip install --upgrade datasets\n\n# Upgrade transformers (needed for huggingface-hub 1.x compatibility)\nuv pip install --upgrade transformers\n\n\n### 4. Patch Source Code (Required for transformers 5.x)\n\nPatch 1 - RoPE cache fix in src/heartlib/heartmula/modeling_heartmula.py:\n\nIn the setup_caches method of the HeartMuLa class, add RoPE reinitialization after the reset_caches try/except block and before the with device: block:\n\npython\n# Re-initialize RoPE caches that were skipped during meta-device loading\nfrom torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE\nfor module in self.modules():\n if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:\n module.rope_init()\n module.to(device)\n\n\nWhy: from_pretrained creates model on meta device first; Llama3ScaledRoPE.rope_init() skips cache building on meta tensors, then never rebuilds after weights are loaded to real device.\n\nPatch 2 - HeartCodec loading fix in src/heartlib/pipelines/music_generation.py:\n\nAdd ignore_mismatched_sizes=True to ALL HeartCodec.from_pretrained() calls (there are 2: the eager load in __init__ and the lazy load in the codec property).\n\nWhy: VQ codebook initted buffers have shape [1] in checkpoint vs [] in model. Same data, just scalar vs 0-d tensor. Safe to ignore.\n\n### 5. Download Model Checkpoints\nbash\ncd heartlib # project root\nhf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'\nhf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'\nhf download --local-dir './ckpt/HeartCodec-oss' 'HeartMuLa/HeartCodec-oss-20260123'\n\n\nAll 3 can be downloaded in parallel. Total size is several GB.\n\n## GPU / CUDA\n\nHeartMuLa uses CUDA by default (--mula_device cuda --codec_device cuda). No extra setup needed if the user has an NVIDIA GPU with PyTorch CUDA support installed.\n\n- The installed torch==2.4.1 includes CUDA 12.1 support out of the box\n- torchtune may report version 0.4.0+cpu — this is just package metadata, it still uses CUDA via PyTorch\n- To verify GPU is being used, look for “CUDA memory” lines in the output (e.g. “CUDA memory before unloading: 6.20 GB”)\n- No GPU? You can run on CPU with --mula_device cpu --codec_device cpu, but expect generation to be extremely slow (potentially 30-60+ minutes for a single song vs ~4 minutes on GPU). CPU mode also requires significant RAM (~12GB+ free). If the user has no NVIDIA GPU, recommend using a cloud GPU service (Google Colab free tier with T4, Lambda Labs, etc.) or the online demo at https://heartmula.github.io/ instead.\n\n## Usage\n\n### Basic Generation\nbash\ncd heartlib\n. .venv/bin/activate\npython ./examples/run_music_generation.py \\\n --model_path=./ckpt \\\n --version=\"3B\" \\\n --lyrics=\"./assets/lyrics.txt\" \\\n --tags=\"./assets/tags.txt\" \\\n --save_path=\"./assets/output.mp3\" \\\n --lazy_load true\n\n\n### Input Formatting\n\nTags (comma-separated, no spaces):\n\npiano,happy,wedding,synthesizer,romantic\n\nor\n\nrock,energetic,guitar,drums,male-vocal\n\n\nLyrics (use bracketed structural tags):\n\n[Intro]\n\n[Verse]\nYour lyrics here...\n\n[Chorus]\nChorus lyrics...\n\n[Bridge]\nBridge lyrics...\n\n[Outro]\n\n\n### Key Parameters\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| --max_audio_length_ms | 240000 | Max length in ms (240s = 4 min) |\n| --topk | 50 | Top-k sampling |\n| --temperature | 1.0 | Sampling temperature |\n| --cfg_scale | 1.5 | Classifier-free guidance scale |\n| --lazy_load | false | Load/unload models on demand (saves VRAM) |\n| --mula_dtype | bfloat16 | Dtype for HeartMuLa (bf16 recommended) |\n| --codec_dtype | float32 | Dtype for HeartCodec (fp32 recommended for quality) |\n\n### Performance\n- RTF (Real-Time Factor) ≈ 1.0 — a 4-minute song takes ~4 minutes to generate\n- Output: MP3, 48kHz stereo, 128kbps\n\n## Pitfalls\n1. Do NOT use bf16 for HeartCodec — degrades audio quality. Use fp32 (default).\n2. Tags may be ignored — known issue (#90). Lyrics tend to dominate; experiment with tag ordering.\n3. Triton not available on macOS — Linux/CUDA only for GPU acceleration.\n4. RTX 5080 incompatibility reported in upstream issues.\n5. The dependency pin conflicts require the manual upgrades and patches described above.\n\n## Links\n- Repo: https://github.com/HeartMuLa/heartlib\n- Models: https://huggingface.co/HeartMuLa\n- Paper: https://arxiv.org/abs/2601.10547\n- License: Apache-2.0\n”}