Machine Learning Engineer, Speech – Joint Audio-Video Modeling
Employment Information
Report this job
Job expired or something wrong with this job?
Job Description
Research/ML Engineer specializing in audio generation and AI systems at Cantina. Leading development of advanced speech and audio models for joint audio-video applications.
Responsibilities:
- Audio Representations: Design, train, and improve the audio VAEs, neural codecs, and vocoders our generative models sit on top of latent design, reconstruction and perceptual objectives, compression-vs-fidelity tradeoffs.
- Model Building: Architect, implement, pre-train, fine-tune, and post-train/alignment (e.g., GRPO/DPO) diffusion and flow-matching transformers for large-scale audio and video generation.
- Joint Audio-Video Modeling: Design the audio conditioning and cross-modal alignment inside joint AV models, audio latents alongside video latents, reference-audio and multi-speaker conditioning, multi shot generation audio/video modeling.
- Experimental Design: Design, run, and analyze scientific experiments to advance our understanding of the models.
- Data Ownership: Define data requirements and collaborate on acquisition, curation, AV-sync and quality filtering, annotation quality, and synthetic data strategies for paired audio-video and speech corpora.
- Rigorous Evaluation: Design automated objective/subjective evaluations audio fidelity and intelligibility metrics, AV-sync, listening and viewing tests, robustness & bias checks, and red-team studies.
- Inference Efficiency: Drive distillation, step-count reduction, quantization, and kernel/memory optimization to meet interactive latency and cost targets.
- Pipeline Delivery: Harden the training → evaluation → inference pipeline; profile latency, memory, and cost; and meet production SLAs with robust monitoring and rollback.
- GPU Scaling: Partner with infrastructure to run distributed training/inference on cloud fleets and productionize models with reliability and observability.
- Project Leadership: Independently lead small research projects while collaborating on larger team initiatives, including cross-team work with video generation.
- Tool Development: Develop and improve dev tooling to enhance team productivity.
- Safety & Responsibility: Contribute to safety/consent guardrails, watermarking, and misuse/abuse mitigation for responsible voice and likeness technology.
Requirements:
- Exceptional research/development experience with large-scale audio models (>8B parameters, >500k hours of data).
- Deep hands-on experience with diffusion and/or flow-matching transformers, including practical knowledge of samplers, schedules, conditioning mechanisms, and distillation.
- Deep hands-on experience training audio VAEs, neural audio codecs, and vocoders latent/tokenizer design, reconstruction and perceptual objectives, adversarial training.
- Strong experience with multi-node, multi-GPU distributed training (FSDP/DeepSpeed or equivalent).
- Strong software engineering skills with a proven track record of building complex systems.
- Strong with PyTorch and performance work (profiling, CUDA/Triton/C++ as needed) and writing reliable production-quality code.
- Shipped large-scale speech/audio or multimodal generative models to production.
- Background in working with large-scale ML data, and the ability to iterate on data and triangulate quality using both subjective and objective signals.
- Experience with voice cloning, speech control/steerability, or expressive speech generation.
- Notable publications and/or open-source contributions in speech/audio/ML.
- Strongly preferred:
- Experience with multimodal audio-video modeling: joint AV generation of multi-shot, multi-speaker scenes with dialogue, music, and sound design generated jointly with video, and the cross-modal alignment that keeps them in sync.
- Experience with video generation: video diffusion/flow-matching transformers, video VAEs, conditioned and multi-shot generation, building data pipelines for video models.
- Streaming or real-time generation, causal distillation (e.g., Self Forcing / Self Forcing++).
Benefits:
- Competitive salary and generous company equity
- Medical, dental, and vision insurance – 99.99% of premiums covered by Cantina
- 42 days of paid time off, including:
- 15 PTO days
- 10 sick days
- 15 company holidays
- 2 floating holidays
- Generous parental leave & fertility support
- 401(k) retirement savings plan
- Lifestyle spending account – $500/month to use however you’d like
- Complimentary lunch and snacks for in-office employees
- One Medical membership, and more!
Similar Jobs
Machine Learning Engineer
Machine Learning Engineer at Interval solving technical problems related to AI, privacy, and distributed systems. Collaborating to build models using enterprise data while maintaining privacy.
Machine Learning Engineer, Especialista II – Dados, Consumidor
Machine Learning Engineer responsible for developing ML solutions and pipelines at Grupo Boticário. Collaborating in a dynamic data ecosystem for beauty brands globally.
Senior Machine Learning Engineer
Senior Machine Learning Engineer developing core AI systems for proactive smart assistant solutions. Collaborating with cross-functional teams to deliver impactful AI products at scale.
Senior Machine Learning Engineer
Senior Machine Learning Engineer at A1 Engineering building AI-native systems. Owning critical ML subsystems, collaborating with teams, and ensuring production reliability.
Senior Machine Learning Engineer
Senior Machine Learning Engineer building and shipping machine learning systems for proactive AI products. Collaborating with research teams and mentoring other engineers.
Senior Machine Learning Engineer
Senior Machine Learning Engineer building and shipping ML systems that are reliable in production. Collaborating with cross-functional teams and mentoring ML engineers to enhance performance.
Member of Technical Staff, Machine Learning
Join A1 Engineering as a Member of Technical Staff, Machine Learning. Build ML components and improve production systems with real-world applications.
Member of Technical Staff, Machine Learning
Member of Technical Staff working on ML components for a smart assistant. Building reliable ML systems and collaborating with senior engineers and product teams.
Staff Machine Learning Engineer
Staff Machine Learning Engineer designing and building backend generative AI solutions at Xero. Collaborating with a remote team to address complex R&D challenges and elevate technical capabilities.
Working Student in Machine Learning – Sustainability Focus
Working student in Machine Learning focusing on sustainability at pacemaker.ai. Supporting development of AI systems and Python data pipelines for carbon footprint calculations.
MLOps Engineer
MLOps Platform Engineer for IRIUM creating a Generative AI platform. Requires 5+ years in MLOps, DevOps, and Azure expertise.
Senior Software Engineer – Machine Learning
Senior Software Engineer developing machine learning systems for product recommendation at Home Depot. Designing architectures that integrate AI into the digital product discovery process.
Senior Machine Learning Engineer
Senior Machine Learning Engineer designing and deploying machine learning solutions for Edelman. Collaborating with cross-functional teams to deliver impactful AI products in a remote-first environment.
Senior Machine Learning Engineer, Ads Content Understanding
Senior Machine Learning Engineer developing scalable ML systems for Ads Content Understanding at Reddit. Key contributor to solutions that enhance monetization impact.
Senior MLOps Engineer
Senior MLOps Engineer responsible for the full lifecycle of machine learning systems. Involved in developing cloud infrastructure and ensuring best practices in software development.
Senior Staff Machine Learning Engineer, Trust
Senior Staff Machine Learning Engineer at Airbnb leading ML initiatives for trust and safety. Collaborating across teams to enhance fraud prevention and community protection through innovative technology.
Machine Learning Engineer
ML Engineer driving machine learning infrastructure from experimentation to production for cybersecurity solutions. Join a well-funded startup building applications of LLMs and AI agents in cybersecurity.
Ingeniero/a MLOps – Teletrabajo
Ingeniero/a MLOps working remotely with IRIUM on projects. Requires high level of English and extensive experience in MLOps or SRE.
Machine Learning Engineer – MLOps, Software Engineering
Machine Learning Engineer developing backend services for operational logistics at flaschenpost. Collaborating with Data Scientists to ensure robust ML services and pipelines.
Machine Learning Scientist, RL – Autonomous Discovery
ML Scientist at Terray Therapeutics developing systems for discovering novel chemical matter using reinforcement learning techniques. Collaborative environment with a focus on machine learning and scientific problem-solving.