How I Built A Video Encoding And Streaming Service

This post walks through how I moved a video processing prototype from a single FFmpeg command to a worker-based encoding pipeline. The project started as a small compression experiment and gradually turned into a backend system for upload handling, adaptive streaming output, and encoding optimization.

Phase 1: Initial Prototype

Tools: FFmpeg, Bash

The first version was a Bash script that accepted a video file and compressed it with H.264. It was intentionally small: one input, one output, and a fixed preset.

That version was useful because it clarified the basic tradeoff: compression is not just about reducing file size. It depends on encode time, codec compatibility, bitrate control, and acceptable visual quality.

ffmpeg -i input.mp4 -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k output.mp4

Phase 2: API-Based Encoding

Tools: Node.js, FFmpeg, Cloudflare R2

The next version exposed the workflow through an API. Users could upload a video, the server ran FFmpeg, and the output was uploaded to Cloudflare R2.

This worked for small inputs, but it coupled request handling and CPU-heavy encoding in the same process. The architecture was simple but limited:

User Upload → Single Server (API + Encoding) → R2 Bucket

The bottleneck was clear: encoding jobs could block the server, and long-running requests were a poor fit for video workloads.

Phase 3: Adaptive Streaming Output

Streaming a single MP4 file through one HTTP request is expensive and does not adapt well to network changes. I moved the output format toward HLS and DASH, where video is split into segments and described through manifest files.

The first implementation used an event-driven flow: an upload event triggered a job, FFmpeg generated HLS/DASH assets, and the resulting segments were stored in R2. A webhook updated job status after processing completed.

#!/bin/bash

set -e

AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
R2_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID
R2_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY
R2_ENDPOINT=$R2_ENDPOINT
R2_BUCKET=$R2_BUCKET

if [ -z "$LOCAL_TEST" ]; then
  S3_BUCKET=${S3_BUCKET}
  S3_KEY=${S3_KEY}
  VIDEO_FILE="/tmp/${S3_KEY}"
  HLS_OUTPUT_DIR="/tmp/hls"
  aws s3 cp s3://${S3_BUCKET}/${S3_KEY} ${VIDEO_FILE}
else
  VIDEO_FILE=${LOCAL_VIDEO_FILE}
  HLS_OUTPUT_DIR=${LOCAL_HLS_OUTPUT_DIR}
fi

mkdir -p ${HLS_OUTPUT_DIR}
OUTPUT_BASE_NAME=$(basename "${VIDEO_FILE%.*}")
UPLOAD_DIR=$(dirname "${S3_KEY}")

ffmpeg -i "${VIDEO_FILE}" \
  -map 0:v -map 0:a -map 0:v -map 0:a \
  -filter:v:0 "scale=-2:480" -c:v:0 libx264 -preset veryfast -crf 23 -c:a:0 aac -b:a 192k \
  -filter:v:1 "scale=-2:720" -c:v:1 libx264 -preset veryfast -crf 23 -c:a:1 aac -b:a 192k \
  -hls_time 10 -hls_playlist_type vod \
  -hls_segment_filename "${HLS_OUTPUT_DIR}/segment_%v%03d.ts" \
  -start_number 0 -var_stream_map "v:0,a:0 v:1,a:1" \
  -master_pl_name master.m3u8 "${HLS_OUTPUT_DIR}/stream%v.m3u8"

echo "Transcoding completed."

mkdir -p /root/.config/rclone
cat <<EOF > /root/.config/rclone/rclone.conf
[myr2]
type = s3
provider = Cloudflare
access_key_id = ${R2_ACCESS_KEY_ID}
secret_access_key = ${R2_SECRET_ACCESS_KEY}
endpoint = ${R2_ENDPOINT}
EOF

rclone sync --transfers 100 ${HLS_OUTPUT_DIR} myr2:${R2_BUCKET}/${UPLOAD_DIR}/${OUTPUT_BASE_NAME} --config /root/.config/rclone/rclone.conf

rm -rf ${VIDEO_FILE} ${HLS_OUTPUT_DIR}

This separated upload handling from packaging work and made the pipeline easier to scale.

Phase 4: Worker-Based Processing In Go

The next step was moving the backend to Go and splitting encoding into independent workers. Go made it easier to model job state, worker availability, and queue processing in a single backend service.

The main design challenge was job distribution. A simple round-robin strategy was not enough because worker nodes can have different CPU capacity and current load.

- Each job enters a Redis-backed pipeline.

- We generate a hash of the job and append it to a Redis list.

- Then, publish a notification to alert workers.

- Each worker listens and accepts work when it has enough available capacity.

This made worker nodes independent and allowed the API layer to focus on job submission and status tracking.

Phase 5: Optimizing the Encoding Process

Once basic encoding was working, quality became the next constraint. Fixed bitrate settings are easy to configure, but they are not always efficient. High-motion scenes and static scenes need different bitrate allocation.

That led to Per-Title Encoding and Two-Pass Encoding:

- Per-Title Encoding adjusts the bitrate ladder based on the video's complexity.

- Two-Pass Encoding analyzes the video first, then optimizes bitrate allocation for quality and efficiency.

I also discovered the concepts of temporal and spatial complexity, both crucial for deciding how many bits are required for a scene.

I wired all this into my pipeline using FFmpeg, which handled:

- Scene analysis

- Two-pass processing

- Smarter bitrate control

The result was better quality at lower bandwidth for the workloads I tested.

Phase 6: Packaging

To reduce duplication between streaming protocols, I explored fragmented MP4 packaging. Bento4 can package media into __fragmented MP4 files__ (.m4s), which are suitable for adaptive streaming.

With fragmented MP4, the same media segments can support both __DASH__ and newer __HLS__ workflows. That simplifies packaging and can reduce duplicated storage.

Conclusion

The project started with a single FFmpeg command and ended with a clearer backend architecture: API layer, job queue, independent workers, adaptive streaming outputs, and packaging decisions based on compatibility and storage cost.

The biggest lesson is that video compression is a systems problem. Codec settings matter, but so do queue design, worker capacity, object storage, packaging format, and playback behavior.

Future work would focus on better observability, hardware acceleration experiments, and more rigorous quality metrics.

LINKS

Live: https://streamscale-dev.aksdev.me/

GitHub: https://github.com/amankumarsingh77/streamscale

X: https://x.com/amankumar404

Email: amankumarsingh7702@gmail.com