Skip to main content
Zhimalab
中文

From Zero to Smooth: A Debugging Record of Building a Web Real-time Video Composer with My AI Partner

Mon Dec 01 2025 08:00:00 GMT+0800 (中国标准时间) · 21 min

This is a debugging record from a developer’s perspective. I wanted a “real-time video compositing on the web” mini-tool: use the camera as the background, then overlay a user-selected window/screen capture on top; also preview, record, go fullscreen, and even export at a specified resolution. With AI as my pair-programming partner, we polished it from an idea into a smooth, usable tool, and along the way solved several key audio/video and Canvas gotchas.

Note: spots marked “[screenshot to be added]” will get actual run screenshots later for reference.

Breaking down the requirements

  • Camera feed as background, screen/window as the overlay layer (picture-in-picture).
  • Support both feeds fullscreen, and split-screen (horizontal/vertical).
  • Support fullscreen preview, real-time recording (WebM), adjustable FPS.
  • Good clarity: small text should be as readable as possible (sharpening/HiDPI).
  • No distortion: selectable output resolution, keep the aspect ratio where possible.

[screenshot to be added: main interface and control panel]

Implementation roadmap (iteration with AI)

  1. React + Astro integration
  • Follow the project’s existing pattern: tools are React components, pages are mounted by Astro.
  • Created src/components/tools/VideoComposer.jsx + src/pages/tools/video-composer.astro.
  1. Building the base capabilities
  • Capture: getUserMedia for the camera, getDisplayMedia for screen/window.
  • Composite: use a <canvas> as the compositing surface, drawImage to overlay the two video streams.
  • Record: canvas.captureStream(fps) + MediaRecorder → WebM download.
  1. Usability enhancements
  • Picture-in-picture overlay: support scaling, positioning, rounded corners, opacity, and drag-to-move.
  • Background source switching: either “camera” or “screen” can act as the background.
  • Fullscreen preview: requestFullscreen to enter/exit fullscreen.
  1. Clarity and layout
  • Sharpening toggle: ctx.imageSmoothingEnabled = !sharp for crisper small text.
  • HiDPI support: scale the backing pixels by devicePixelRatio to stay sharp.
  • Layout modes: picture-in-picture (pip) / horizontal split / vertical split.
  1. Resolution control
  • Dropdown presets: common tiers like 1024x576, 1280x720, 1920x1080.
  • Purpose: reduce distortion and align with recording targets / streaming parameters.

[screenshot to be added: split-screen, picture-in-picture and resolution selection]

Key issues and fixes

1. “ReferenceError: Cannot access ‘hiDPI’ before initialization”

Symptom: after a big change, the component reported hiDPI being accessed before initialization. Root cause: a patch was inserted in the wrong order — a variable was referenced before its declaration, and some code snippets were inserted outside the function.

Fix: reverted and rewrote the affected sections, making sure state declaration order and scope were correct; encapsulated the resolution logic into setupCanvasSize() to avoid scattered “bare code”.

Lesson: large structural changes are best replaced atomically, to avoid repeated small patches corrupting the file structure.

2. Fullscreen preview distortion/blur

Symptom: after entering fullscreen, text in the window became distorted or blurry.

Optimizations:

  • HiDPI: use canvas.width = w * dpr; canvas.height = h * dpr; for the backing canvas, then ctx.scale(dpr, dpr);
  • Sharpening: optionally disable interpolation smoothing (imageSmoothingEnabled=false) to make small text clearer;
  • Resolution presets: keep the output logical size controllable, reducing stretching.

3. At 1920×1080, a “missing edge on the right/bottom” (roughly a few dozen pixels)

Symptom: both the system resolution and output resolution were set to 1920×1080, but the right and bottom edges were “eaten” a bit.

Root cause: on high-DPI screens we scale the ctx (ctx.scale(dpr, dpr)), but the drawing and drag logic used canvas.width/height (device pixels) for calculations — effectively “scaling twice”, clipping the right/bottom edges.

Fix: uniformly use the “logical size” (CSS pixels) for all coordinate/size calculations:

// 1) Set the backing pixels and record the logical size in setupCanvasSize
const dpr = hiDPI ? window.devicePixelRatio || 1 : 1;
canvas.width = Math.floor(w * dpr);
canvas.height = Math.floor(h * dpr);
ctx.setTransform(1, 0, 0, 1, 0, 0);
ctx.scale(dpr, dpr);
setCanvasLogicalSize({ w, h });

// 2) Use the logical size (w/h) in draw/drag, not canvas.width/height
const w = canvasLogicalSize.w || canvas.width;
const h = canvasLogicalSize.h || canvas.height;
ctx.drawImage(bgVideo, 0, 0, w, h);

Result: at 1920×1080 the right/bottom cropping no longer happens, and fullscreen matches the recording.

[screenshot to be added: before/after cropping comparison]

Pair programming with AI: advantages and lessons

  • Idea acceleration: I state the requirement, and AI quickly gives an implementation skeleton and the key APIs (MediaRecorder, captureStream, getDisplayMedia, etc.).
  • Iteration efficiency: from “picture-in-picture” to “split-screen”, “sharpening/HiDPI”, “resolution presets”, I only had to describe the UX problem and AI filled in the change scope and provided patches.
  • Bug localization: when ReferenceError and HiDPI cropping issues appeared, AI quickly reasoned out the root cause and gave a minimal fix surface.
  • Cognitive expansion: around Canvas and media-stream details (like the coordinate system under DPR), AI helped me build a more systematic mental model.

Suggested collaboration posture:

  • Clear goal + expected behavior: describe “what you see” and “what you want it to feel like”, not just “how to implement it”.
  • Small-step commits: split complex changes where possible, and ask AI for “atomic” patches to reduce merge risk.
  • Test with real parameters: e.g. pin to 1920×1080 and observe sharpness, black bars, distortion, cropping and actual FPS.

Caveats (pitfall checklist)

  • Browser permissions: media capture requires HTTPS or localhost, and user authorization must be confirmed.
  • Recording compatibility: MediaRecorder encoding support differs across browsers (vp9/vp8/webm); fall back via isTypeSupported.
  • DPR coordinate system: under HiDPI, distinguish “logical size vs backing pixels” to avoid double scaling.
  • Aspect ratio: when the source and target resolutions differ in ratio, consider offering a “fit mode” (contain/cover/stretch) toggle.
  • Stream-ended events: getDisplayMedia tracks fire ended; remember to clean up UI state and stream objects.
  • Big-patch risk: many small tweaks can scramble the file structure; when necessary, go for a “full rewrite and replace”.

Next steps

  • Custom resolution input and proportional fit modes (contain/cover).
  • Split-screen ratio slider (instead of allocating by source resolution ratio).
  • Overlay edge-snapping/safe margins and guide lines.
  • Audio mixing (e.g. system audio/microphone if needed).

[screenshot to be added: fullscreen preview, recorded output]

Conclusion

This collaboration with AI was both efficient and safe: I own the goals and acceptance criteria, AI owns implementation and fix suggestions; every step explains “why do it this way”. If you’re also building a web audio/video mini-tool, feel free to use this pitfall list and fix experience directly.

If there’s any detail you’d like source snippets for, or spots that need screenshot explanations, leave a message and I’ll add images/code.