HeyGen Remotion 集成指南在 OpenMontage 中生成虚拟人视频并完成帧级精确合成【免费下载链接】OpenMontageWorlds first open-source, agentic video production system. 12 production pipelines, 100 tools, 700 agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.项目地址: https://gitcode.com/GitHub_Trending/op/OpenMontage导读本文以 OpenMontage 仓库内 HeyGen 技能参考文档 remotion-integration.md 为核心系统讲解如何把 HeyGen 云端 AI 虚拟人Avatar视频接入 Remotion 代码化视频合成流水线从输出格式选型MP4 带背景 / WebM 透明通道、并行开发工作流、画幅对齐到 Remotion 中基于OffthreadVideo的帧级精确渲染、Loom 风格圆形头像遮罩、分层合成与动态时长计算。读完本文你将掌握一条「HeyGen 生成 → 轮询取 URL → Remotion 合成 → 命令行渲染」的完整可落地链路并能结合仓库中的TalkingHead合成组件与 HeyGen 工具封装将其嵌入自己的生产管线。Overview一条典型的合成工作流HeyGen 负责生成虚拟人口播视频Remotion 负责把它与其他素材背景、Logo、图表、动画、字幕在代码层面合成最终成片。典型流程如下用 HeyGen 生成虚拟人视频MP4 或 WebM等待生成完成获取视频 URL在 Remotion 中直接使用 URL或先下载到本地再引用与背景、叠加层、动画等其他元素合成。仓库中 SKILL.md 对 HeyGen 技能的整体定位是AI avatar video creation API for generating talking-head videos, explainers, and presentations并在 Quick Reference 中明确将「与 Remotion 搭配使用」指向本文对应的 remotion-integration.md说明该集成方案是仓库内 HeyGen 技能的官方推荐用法之一。在 OpenMontage 的 remotion-composer 子项目中已经存在与本文档同构的实战实现TalkingHead.tsx 以videoSrc为输入用OffthreadVideo承载口播视频层用Sequence按秒级时间轴驱动图表、数据卡、标注框等 overlay 层最上层叠加CaptionOverlay字幕——这正是本文所述分层合成思路在真实仓库中的落地形态下文会多次回指该实现。Quick Start三步跑通最小闭环// 1. Get avatar with default voice const avatar await getAvatarDetails(avatarId); // 2. Generate video (MP4 with background - most common) const videoId await generateVideo({ video_inputs: [{ character: { type: avatar, avatar_id: avatar.id, avatar_style: normal }, voice: { type: text, input_text: script, voice_id: avatar.default_voice_id }, background: { type: color, value: #1a1a2e }, }], dimension: { width: 1920, height: 1080 }, }); // 3. Poll for completion (10-15 min) // 4. Use in Remotion with motion graphics overlaid on top关键点在于使用头像自带的default_voice_id多数 HeyGen 头像自带预匹配的默认音色语音与形象的自然度最好也省去手动挑选音色的步骤详见 avatars.md 与 voices.md。选择正确的输出格式这是整个集成方案中第一个、也是最重要的决策点因为它直接决定后续在 Remotion 中如何合成你的合成方案推荐格式原因虚拟人作为主持人上层叠加动效MP4 背景更简单叠加层直接盖在视频上Loom 风格虚拟人叠加在录屏上WebM closeUp在 Remotion 中做遮罩需要透明通道用 CSS 圆形遮罩虚拟人叠加在其他视频/内容之上WebM透明需要透过虚拟人看到背后内容全屏虚拟人MP4 背景标准做法绝大多数场景使用 MP4 带背景。只有当需要看到虚拟人背后的内容时才使用 WebM。注意WebM 只支持normal与closeUp两种风格。需要圆形构图时请在 Remotion 中用 CSSborder-radius: 50%实现而不是依赖 HeyGen 的circle风格。这一点在 video-generation.md 的 WebM 一节中有同样结论WebM 只用于「虚拟人叠加在录屏上」「虚拟人漂浮在视频背景上」「真正的 alpha 通道合成」三种场景。推荐并行开发工作流HeyGen 视频生成耗时10-15 分钟以上绝不应该干等。推荐的并行策略先启动 HeyGen 生成——把video_id存到文件里立即退出并行构建 Remotion 合成——先用占位素材或头像的preview_video_url一段短视频循环顶着构建过程中周期性检查 HeyGen 状态或者构建完成后再查就绪后把占位素材替换为真实视频 URL。根据脚本估算时长语速约 150 词/分钟因此wordCount / 150 * 60 * fps即可估算出大致帧数。合成设计建议把组件设计成「有无虚拟人视频都能工作」这样动效可以独立于虚拟人进行测试。仓库中的 TalkingHead.tsx 正是这种设计——videoSrc只是众多 props 之一overlay 与字幕层完全不依赖视频是否就位可以在虚拟人视频渲染完成前独立预览全部动效。画幅对齐HeyGen 输出尺寸必须与 Remotion 合成尺寸一致关键约束HeyGen 的输出画幅必须与 Remotion 合成画幅完全一致否则会出现黑边、裁切或缩放失真。通用画幅预设常量// Shared dimension constants for both HeyGen and Remotion const DIMENSIONS { landscape_1080p: { width: 1920, height: 1080 }, landscape_720p: { width: 1280, height: 720 }, portrait_1080p: { width: 1080, height: 1920 }, portrait_720p: { width: 720, height: 1280 }, square_1080p: { width: 1080, height: 1080 }, square_720p: { width: 720, height: 720 }, } as const; type DimensionPreset keyof typeof DIMENSIONS;HeyGen 侧按预设生成视频// Generate HeyGen video with specific dimensions async function generateHeyGenVideo( script: string, avatarId: string, voiceId: string, preset: DimensionPreset ): Promisestring { const dimension DIMENSIONS[preset]; const response await fetch(https://api.heygen.com/v2/video/generate, { method: POST, headers: { X-Api-Key: process.env.HEYGEN_API_KEY!, Content-Type: application/json, }, body: JSON.stringify({ video_inputs: [ { character: { type: avatar, avatar_id: avatarId, avatar_style: normal, }, voice: { type: text, input_text: script, voice_id: voiceId, }, background: { type: color, value: #00FF00, // Green screen for compositing }, }, ], dimension, }), }); const { data } await response.json(); return data.video_id; }Remotion 侧在 Root.tsx 中注册同尺寸合成// remotion/src/Root.tsx import { Composition } from remotion; import { AvatarComposition } from ./AvatarComposition; const DIMENSIONS { landscape_1080p: { width: 1920, height: 1080 }, // ... same as above }; export const RemotionRoot: React.FC () { return ( Composition idAvatarVideo component{AvatarComposition} durationInFrames{300} // Will be set dynamically fps{30} width{DIMENSIONS.landscape_1080p.width} height{DIMENSIONS.landscape_1080p.height} defaultProps{{ avatarVideoUrl: , }} / / ); };仓库中的 Root.tsx 注册了同构的TalkingHead合成其默认画幅为 9:161080x1920与TalkingHead.tsx中POSITION_STYLES按竖屏画幅设计的 overlay 定位如lower_third底部留 320px 给字幕区相互呼应——再次印证「先定画幅再写合成」的工程顺序。为 Remotion 生成虚拟人视频标准做法MP4 带背景绝大多数 Remotion 合成采用 MP4 背景叠加层与动效直接盖在视频上层async function generateAvatarForRemotion( script: string, avatarId: string, voiceId: string, options: { style?: normal | closeUp | circle; backgroundColor?: string; } {} ): Promisestring { const { style normal, backgroundColor #1a1a2e } options; const response await fetch(https://api.heygen.com/v2/video/generate, { method: POST, headers: { X-Api-Key: process.env.HEYGEN_API_KEY!, Content-Type: application/json, }, body: JSON.stringify({ video_inputs: [{ character: { type: avatar, avatar_id: avatarId, avatar_style: style, }, voice: { type: text, input_text: script, voice_id: voiceId, }, background: { type: color, value: backgroundColor, }, }], dimension: { width: 1920, height: 1080 }, }), }); const { data } await response.json(); return data.video_id; }透明背景WebM 接口仅当需要看到虚拟人背后的内容时使用例如虚拟人叠加在录屏之上// Use /v1/video.webm endpoint for transparent background // Note: Different structure than /v2/video/generate const response await fetch(https://api.heygen.com/v1/video.webm, { method: POST, headers: { X-Api-Key: process.env.HEYGEN_API_KEY!, Content-Type: application/json, }, body: JSON.stringify({ avatar_pose_id: avatarPoseId, // Required: avatar pose ID avatar_style: normal, // Required: normal or closeUp only input_text: script, // Required (with voice_id) voice_id: voiceId, // Required (with input_text) dimension: { width: 1920, height: 1080 }, }), });注意 WebM 端点/v1/video.webm的请求结构与/v2/video/generate不同使用扁平字段avatar_pose_id/avatar_style/input_text/voice_id且input_textvoice_id与input_audio二选一不能同时提供。生成后与 MP4 使用相同的状态轮询接口返回的video_url为.webm文件详见 video-generation.md 与 video-status.md。在 Remotion 中使用 HeyGen 视频关键渲染必须用OffthreadVideo而非Video虚拟人视频一律使用OffthreadVideo。基础Video组件走浏览器视频解码器无法做到逐帧精确渲染时会产生抖动jitterOffthreadVideo通过 FFmpeg 逐帧抽取画面保证平滑且帧精确。OffthreadVideo属于remotion核心包无需额外安装。基础用法// remotion/src/AvatarComposition.tsx import { OffthreadVideo, useVideoConfig } from remotion; interface AvatarCompositionProps { avatarVideoUrl: string; } export const AvatarComposition: React.FCAvatarCompositionProps ({ avatarVideoUrl, }) { return ( div style{{ flex: 1, backgroundColor: #1a1a2e }} OffthreadVideo src{avatarVideoUrl} style{{ width: 100%, height: 100%, objectFit: contain, }} / /div ); };仓库实现与此一致TalkingHead.tsx 中的口播视频层即用OffthreadVideoobjectFit: cover承载且通过resolveAsset(videoSrc)统一解析本地静态资源与远程 URL。WebM 透明通道合成推荐使用/v1/video.webm生成的 WebM 自带 alpha 通道无需任何色键处理import { OffthreadVideo, AbsoluteFill, Sequence } from remotion; export const AvatarWithMotionGraphics: React.FC{ avatarWebmUrl: string } ({ avatarWebmUrl }) { return ( AbsoluteFill {/* Layer 1: Your background/content */} AbsoluteFill style{{ backgroundColor: #1a1a2e }} YourMotionGraphics / /AbsoluteFill {/* Layer 2: Avatar with transparent background - use OffthreadVideo for frame-accurate rendering */} OffthreadVideo src{avatarWebmUrl} transparent style{{ position: absolute, bottom: 0, right: 0, width: 50%, height: auto, }} / {/* Layer 3: Overlays on top of avatar */} Sequence from{30} AnimatedTitle textWelcome! / /Sequence /AbsoluteFill ); };要点WebM 加transparent属性用Sequence控制叠加层出现时机此处from{30}表示第 30 帧后出现即第 1 秒。Loom 风格圆形头像叠加录屏使用closeUp风格 WebM再在 Remotion 中做圆形遮罩import { OffthreadVideo, AbsoluteFill } from remotion; export const LoomStyleComposition: React.FC{ screenRecordingUrl: string; avatarWebmUrl: string; // Generated with avatar_style: closeUp via /v1/video.webm } ({ screenRecordingUrl, avatarWebmUrl }) { return ( AbsoluteFill {/* Screen recording fills the frame */} OffthreadVideo src{screenRecordingUrl} style{{ width: 100%, height: 100% }} / {/* Avatar with circular mask - transparent bg shows screen behind */} OffthreadVideo src{avatarWebmUrl} transparent style{{ position: absolute, bottom: 40, left: 40, width: 180, height: 180, borderRadius: 50%, // Circular mask applied in CSS overflow: hidden, objectFit: cover, }} / /AbsoluteFill ); };注意WebM 不支持circle风格——请用normal或closeUp生成再用 CSS 圆形遮罩border-radius: 50%overflow: hiddenobjectFit: cover实现圆形头像效果。遗留方案绿幕 色键如果手里只有绿幕 MP4不推荐应改用 WebM// Note: True chroma key requires WebGL or post-processing // WebM transparent background is much simpler OffthreadVideo src{avatarVideoUrl} style{{ mixBlendMode: multiply, // Basic compositing only }} /真实色键需要 WebGL 或后期处理mixBlendMode只能做基础合成因此 WebM 透明通道是远为简单的方案。分层合成示例import { OffthreadVideo, Sequence, useVideoConfig, Img } from remotion; interface LayeredAvatarProps { avatarVideoUrl: string; backgroundUrl: string; logoUrl: string; title: string; } export const LayeredAvatarComposition: React.FCLayeredAvatarProps ({ avatarVideoUrl, backgroundUrl, logoUrl, title, }) { const { fps } useVideoConfig(); return ( div style{{ position: relative, width: 100%, height: 100% }} {/* Layer 1: Background */} Img src{backgroundUrl} style{{ position: absolute, width: 100%, height: 100%, objectFit: cover, }} / {/* Layer 2: Avatar video - use OffthreadVideo to prevent jitter */} OffthreadVideo src{avatarVideoUrl} style{{ position: absolute, bottom: 0, right: 0, width: 40%, height: auto, }} / {/* Layer 3: Title (appears after 1 second) */} Sequence from{fps} div style{{ position: absolute, top: 50, left: 50, color: white, fontSize: 48, fontWeight: bold, }} {title} /div /Sequence {/* Layer 4: Logo */} Img src{logoUrl} style{{ position: absolute, top: 20, right: 20, width: 100, height: auto, }} / /div ); };分层顺序从底到顶背景图 → 虚拟人视频 → 标题延迟 1 秒出现→ Logo。仓库 TalkingHead.tsx 采用了相同的分层哲学并加以泛化视频层 → overlay 层PositionedOverlay封装了淡入淡出与位置预设→ 最顶层字幕overlay 类型涵盖文本卡、数据卡、标注框、对比卡与柱状/折线/饼图等图表组件见 components。完整工作流生成并合成从脚本到成片import { bundle } from remotion/bundler; import { renderMedia, selectComposition } from remotion/renderer; async function generateAvatarVideoForRemotion( script: string, outputPath: string ) { // 1. Generate HeyGen video console.log(Generating HeyGen avatar video...); const videoId await generateHeyGenVideo( script, josh_lite3_20230714, 1bd001e7e50f421d891986aad5158bc8, landscape_1080p ); // 2. Wait for completion console.log(Waiting for HeyGen video...); const avatarVideoUrl await waitForVideo(videoId); console.log(HeyGen video ready: ${avatarVideoUrl}); // 3. Get video duration for Remotion const avatarDuration await getVideoDuration(avatarVideoUrl); const durationInFrames Math.ceil(avatarDuration * 30); // 30 fps // 4. Bundle Remotion project console.log(Bundling Remotion project...); const bundleLocation await bundle({ entryPoint: ./remotion/src/index.ts, }); // 5. Select composition const composition await selectComposition({ serveUrl: bundleLocation, id: AvatarVideo, inputProps: { avatarVideoUrl, }, }); // 6. Render final video console.log(Rendering final composition...); await renderMedia({ composition: { ...composition, durationInFrames, }, serveUrl: bundleLocation, codec: h264, outputLocation: outputPath, inputProps: { avatarVideoUrl, }, }); console.log(Final video rendered: ${outputPath}); return outputPath; }这条链路与 OpenMontage 的工程实现一一对应仓库中 heygen_video.py 是 HeyGen 视频生成的工具封装要求环境变量HEYGEN_API_KEY走POST /v1/workflows/executions的 GenerateVideoNode 工作流其底层轮询逻辑 poll_heygen 采用 5 秒起始间隔、1.2 倍指数退避、最长 600 秒超时的策略与本文推荐「10-15 分钟生成、周期轮询」完全一致Remotion 渲染器 则对应bundle/selectComposition/renderMedia的可编程渲染路径。动态时长用 calculateMetadata 自动对齐视频长度// remotion/src/AvatarComposition.tsx import { CalculateMetadataFunction } from remotion; export const calculateAvatarMetadata: CalculateMetadataFunction AvatarCompositionProps async ({ props }) { // Fetch video duration from HeyGen video const duration await getVideoDurationInSeconds(props.avatarVideoUrl); return { durationInFrames: Math.ceil(duration * 30), fps: 30, width: 1920, height: 1080, }; }; // In Root.tsx Composition idAvatarVideo component{AvatarComposition} calculateMetadata{calculateAvatarMetadata} defaultProps{{ avatarVideoUrl: , }} /calculateMetadata让合成时长、帧率、画幅在渲染前按真实视频动态确定避免写死durationInFrames{300}造成的截断或空帧。最佳实践1. 用绿幕换取合成灵活性需要合成时让 HeyGen 以纯绿背景生成background: { type: color, value: #00FF00, // Pure green for chroma key }2. 匹配帧率HeyGen 默认输出 25 fps设置 Remotion 帧率时要考虑这一点// Option 1: Match HeyGens 25 fps fps: 25 // Option 2: Use 30 fps with playback rate adjustment OffthreadVideo src{avatarVideoUrl} playbackRate{25/30} // Slow down slightly to match /3. URL 直用 vs 本地下载直接使用 URL当在 Remotion Studio 中预览npm run devURL 在渲染完成前不会过期开发期追求快速迭代。// Direct URL usage - simpler, faster for dev OffthreadVideo src{avatarVideoUrl} /先下载到本地当URL 会过期HeyGen URL 约 24 小时后失效渲染发生在之后或需要重复渲染网络可靠性堪忧需要离线渲染。// Download with retry for reliability async function downloadVideoWithRetry( url: string, outputPath: string, maxRetries 5 ): Promisestring { for (let attempt 0; attempt maxRetries; attempt) { try { const response await fetch(url); if (!response.ok) throw new Error(HTTP ${response.status}); const buffer await response.arrayBuffer(); await fs.promises.writeFile(outputPath, Buffer.from(buffer)); return outputPath; } catch (error) { const delay 2000 * Math.pow(2, attempt); console.log(Retry ${attempt 1}/${maxRetries} in ${delay}ms...); await new Promise((r) setTimeout(r, delay)); } } throw new Error(Download failed after retries); } // Use local file in Remotion const localPath await downloadVideoWithRetry(avatarVideoUrl, ./public/avatar.mp4);混合方案生产环境推荐// Save both URL and local path in metadata const metadata { videoUrl: result.video_url, // For quick preview localPath: ./public/avatar.mp4, // For reliable rendering expiresAt: Date.now() 24 * 60 * 60 * 1000, // URL expiration }; // In Remotion component, prefer local if available const videoSrc fs.existsSync(localPath) ? staticFile(avatar.mp4) : avatarVideoUrl;4. 控制虚拟人位置常见构图位置预设const AVATAR_POSITIONS { fullscreen: { width: 100%, height: 100%, position: center }, bottomRight: { width: 40%, bottom: 0, right: 0 }, bottomLeft: { width: 40%, bottom: 0, left: 0 }, pictureInPicture: { width: 25%, bottom: 20, right: 20 }, leftThird: { width: 33%, left: 0, height: 100% }, };仓库 TalkingHead.tsx 以同样的思路为 9:16 画幅定义了lower_third/upper_third/left_panel/right_panel/full_overlay五档 overlay 位置预设可参考其坐标数值设计自己的竖屏布局。输出格式速查HeyGen 输出格式MP4H.264音频AAC分辨率按请求指定如 1920x1080Remotion 输出编码器H.264默认、VP8、VP9、ProRes质量应与 HeyGen 输出匹配或更高await renderMedia({ codec: h264, crf: 18, // High quality // ... });故障排查视频在 Remotion 中无法播放检查 URL 可访问性CORS 问题确认视频格式兼容性先尝试下载到本地再引用。画幅不匹配确保 HeyGen 与 Remotion 使用完全一致的尺寸// Shared config const VIDEO_CONFIG { width: 1920, height: 1080, fps: 30, }; // HeyGen dimension: { width: VIDEO_CONFIG.width, height: VIDEO_CONFIG.height } // Remotion Composition width{VIDEO_CONFIG.width} height{VIDEO_CONFIG.height} /渲染时视频抖动如果渲染结果中虚拟人画面抖动或卡顿用OffthreadVideo替换Video——基础Video组件走浏览器视频解码器无法逐帧精确更新导入无需额外安装OffthreadVideo就在核心remotion包里// Before (causes jitter) import { Video } from remotion; // After (frame-accurate) import { OffthreadVideo } from remotion;使用带透明通道的 WebM 时加transparent属性OffthreadVideo src{avatarWebmUrl} transparent /音画不同步若虚拟人音频漂移核对源视频帧率检查编码问题考虑用一致的参数重新编码。总结HeyGen Remotion 的组合本质上是「云端虚拟人口播能力」与「代码化合成渲染能力」的分工HeyGen 负责把脚本变成可信的虚拟人视频Remotion 负责在帧级精确的合成器中把视频、图表、字幕、品牌元素编排成最终成片。掌握本文的四条主线——输出格式选型MP4 vs WebM、并行开发工作流、画幅/帧率对齐、OffthreadVideo帧精确渲染——即可把这条链路稳定嵌入 OpenMontage 式的自动化视频生产线。进一步可阅读 video-generation.md多场景视频、WebM 字段详解、video-status.md轮询与下载重试、webhooks.md生产环境替代轮询的方案并对照仓库实现 TalkingHead.tsx 与 heygen_video.py 深化理解。【免费下载链接】OpenMontageWorlds first open-source, agentic video production system. 12 production pipelines, 100 tools, 700 agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.项目地址: https://gitcode.com/GitHub_Trending/op/OpenMontage创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考