Luigi Pizzolito's Logo

TinyMikuServer - HoloCube Part 2

Series: HoloCube

Contents

This is the second part in a three part series on how I built a little HoloCube hologram video player and custom software dedicated to playing back hologram MMD concerts on this device. In this post I will cover the custom video & audio playback format I developed in order to be able to stream to the ESP32S3 in real time and render 240x240 px video plus 16kHz mono audio.

Introduction & Purpose

In the previous post, I covered how I implemented content-aware video cropping to convert the 16:9 MMD hologram videos into a suitable 1:1 video for playback on the HoloCube device; please see the previous post for context.

The next step is to implement a way to stream the generated 240x240px videos and audio to the ESP32S3 MCU in the HoloCube device in a format that is relatively compressed, as to not overload the MCU’s WiFi bandwidth, while at the same time not taking so much computational power to decode that the MCU fails to decode the video/audio for playback in real-time.

These are the main goals and features for this stage of the project:

First I’ll detail the layout and organisation of this code repository as it contains packaged versions of several tools, then I’ll go in depth on the implementation of the custom MikV video format. The repository is organised as follows:

TinyMikuServer/
├── main.go                 # Main server application
├── pkg/
│   ├── mikvformat/        # MikV format library
│   │   ├── MikVFormat.go  # Core format implementation
│   │   └── cmd/
│   │       └── mikv-converter/  # MP4 to MikV converter CLI
│   ├── mikvserver/        # HTTP streaming server
│   │   ├── MikVServer.go  # Server implementation
│   │   └── server.go      # HTTP handlers
│   └── holocropper/       # Video cropping utility
│       ├── holocropper.go # Core cropping logic
│       └── cmd/
│           └── holocropper/     # HoloCropper CLI
├── bin/                   # Compiled binaries (generated)
└── assets/                # Video assets (not tracked)
    ├── mp4/              # Processed MP4 files
    ├── mp4-full/         # Original full videos
    └── mikv/             # Converted MikV files

Additionally, a Makefile is included to facilitate building all the different tools. For more details see the repository on GitHub:

The overall goal is to develop a software framework that allows for easy conversion from a downloaded MMD hologram-ready video to a MikV format that is ready to stream for real-time playback on the HoloCube. Let’s start by organising and cleaning up what was implemented before.

A Much-Needed Code Refactor

I began by refactoring much of the code I had developed for the previous part, the content-aware 1:1 video cropper. I started by stripping out the UI elements and converting the code into a library that I can import and use in this video server project. A standalone command-line tool was left as a bare-bones cropping tool in pkg/holocropper/cmd/holocropper.

Then I proceeded by also implementing my new MikV video format encoder as a library as well. Accompanied by it’s respective command-line conversion tool in pkg/mikvformat/cmd/mikv-converter so that videos can be pre-converted into MikV format for the server to stream.

This left a simple interface for our main.go, simply select a .mikv file to stream and the ESP32S3 on the HoloCube connects to this server to stream the video over HTTP.

HoloCropper

After stripping out the Ebitten UI that was used heavily during the cropping algorithm development detailed in the previous post in this series, the command line arguments were streamlined and set with sensible defaults for the HoloCube device. Below is the help prompt and a full usage example:

$ ./holocropper -h
flag needs an argument: -h
Usage of ./holocropper:
  -fps float
        Frames per second for the cropped video (default 20)
  -h int
        Height of the cropped video (default 240)
  -i string
        Path to the input video file (default: current folder) (default ".")
  -o string
        Directory to save the cropped video
  -w int
        Width of the cropped video (default 240)
exit status 2

$ ./holocropper -i ../../assets/miku_snippet.mp4 -o ../../assets/out
Input Video Path: ../../assets/miku_snippet.mp4
Output Directory: ../../assets/out
Crop Width: 240
Crop Height: 240
Frames Per Second: 20.00
Starting processing...
/usr/sbin/ffmpeg -i ~/Documents/Github/TinyMikuServer/assets/miku_snippet.mp4 -vf unsharp=5:5:1.0:5:5:0.0,scale=426:240 -sws_flags lanczos+accurate_rnd -r 20 /tmp/tinyvideo3164542692/miku_snippet.mp4 -y
 100% |███████████████████████████████████████████████████████████| (768/768, 631 it/s)        
Starting crop processing for ~/Documents/Github/TinyMikuServer/assets/miku_snippet.mp4
 100% |███████████████████████████████████████████████████████████| (768/768, 1167 it/s)        
/usr/sbin/ffmpeg -r 20 -framerate 20 -i /tmp/tinyframes2312036518/%d.jpg -i ~/Documents/Github/TinyMikuServer/assets/miku_snippet.mp4 -map 0:v:0 -map 1:a:0 -c:v mjpeg -q:v 6 -c:a copy ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4 -y
 100% |███████████████████████████████████████████████████████████| (642/642, 7759 it/s)        
Processing complete for ../../assets/miku_snippet.mp4
Processing complete!

MikV Format

The same approach was taken with the MikV format converter CLI, using sensible defaults and simple command line arguments to convert a holo-cropped video straight into the required format. Below is the help prompt and a full usage example:

$ ./mikv-converter -h
Usage of ./mikv-converter:
  -bitsPerSample uint
        Audio bits per sample (default: 16)
  -channels uint
        Number of audio channels (default: 1)
  -i string
        Input MP4 file or directory (default: current directory) (default ".")
  -o string
        Output directory for MikV files (default: current directory) (default ".")
  -sampleRate uint
        Audio sample rate (default: 16000)
  -silent
        Disable verbose output

$ ./mikv-converter -i ../../../../assets/out/miku_snippet.mp4 -o ../../../../assets/out
Processing file: ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4
Converting MP4 file ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4 to MikV format
Converting video frames 100% |███████████████████████████████████████████████████████████| (642/642, 4073 it/s)        
Transcoding audio 100% |███████████████████████████████████████████████████████████| (1027200/1027200, 29290731 it/s)        
Adding frames to MikV  84% |███████████████████████████████████████████████████████████| (541/642, 2761 it/s) [0s:0s]Audio data shorter than video, leaving tail as silence
Loaded 641 frames from MP4 file
Successfully converted ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4 to ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mikv

MikV Server

Last but not least, the MikV server itself is a simple HTTP server that holds an internal MikV player which plays the frames back in real time. Then a channel is used to grab the current frame and serve it over the /video endpoint. A special X-Video-Length HTTP header is used to indicate the length in bytes of the current JPEG frame; with the rest of the data appended after that byte position being the current PCM encoded audio slice. This way the ESP32S3 can easily separate the video and audio elements from the response. This is done using the functions below:

server.go L32-L83
32func (ms *MikVServer) streamer() {
33	// Take input frames and set lastframe
34	for {
35		select {
36		case frame, ok := <-ms.Stream:
37			if !ok {
38				// Input channel closed
39				return
40			}
41
42			// Grab frame lock & set last frame
43			ms.frameLock.Lock()
44			ms.lastFrame = frame
45			ms.frameLock.Unlock()
46		}
47	}
48}
49
50func (ms *MikVServer) videoHandler(w http.ResponseWriter, r *http.Request) {
51	// Grab frame lock
52	ms.frameLock.Lock()
53	defer ms.frameLock.Unlock()
54
55	// Check if we have a frame
56	if len(ms.lastFrame.Video) == 0 {
57		w.WriteHeader(http.StatusNoContent)
58		return
59	}
60
61	// Set headers
62	w.Header().Set("Content-Type", "application/octet-stream")
63
64	audioFrameLength := len(ms.lastFrame.Audio)
65	videoFrameLength := len(ms.lastFrame.Video)
66	totalFrameLength := audioFrameLength + videoFrameLength
67	w.Header().Set("Content-Length", strconv.Itoa(totalFrameLength))
68	w.Header().Set("X-Video-Length", strconv.Itoa(videoFrameLength))
69
70	// Prepare the response
71	totalFrameBuffer := make([]byte, totalFrameLength)
72	copy(totalFrameBuffer, ms.lastFrame.Video)
73	copy(totalFrameBuffer[videoFrameLength:], ms.lastFrame.Audio)
74
75	// Write the response
76	w.WriteHeader(http.StatusOK)
77	num, err := w.Write(totalFrameBuffer)
78	if err != nil {
79		fmt.Printf("Error streaming frame: %v\n", err)
80	} else if num != totalFrameLength {
81		fmt.Printf("Error streaming frame: Only %d/%d bytes sent.\n", num, totalFrameLength)
82	}
83}

The reason that this endpoint simply streams the current frame from the MikV file and has it’s own internal video playback on the server side is simple; to ensure real-time playback even if lag or slow-downs occur on the ESP32S3 side.

The ESP32S3 MCU simply fetches the current frame and audio fragment continuously, and even if frames are dropped, the video playback remains locked onto real-time playback. This causes a minor audio-glitch as the audio samples don’t quite line up if frames are dropped; but its a valid architectural decision to ensure the ESP32S3 doesn’t have to try and keep up with buffering the full MikV file or keeping it’s own playback real-time locked.

This is all exposed with a very straight forward interface for us to call in our main.go:

main.go L10-L24
10func main() {
11	fmt.Println("Starting...")
12
13	mikv := mikvformat.NewMikVFormat()
14	mikv.Load("./assets/mikv/39.mikv")
15
16	fmt.Println("Loaded MikVFormat")
17
18	server := mikvserver.NewMikVServer()
19	server.Start()
20
21	fmt.Println("Started MikVServer")
22
23	mikv.Stream(server.Stream, nil)
24}

Now that the tooling and organisation is out of the way, I’ll focus the rest of this post into going more in depth on the design an implementation of the MikV video format.

Developing a Custom Video Format

The 2 most common video formats used for video live-streaming on lower-end devices are MP4 and MJPEG. MP4 is way more ubiquitous, but also requires comparatively high levels of compute to decode as it’s an advanced and efficient algorithm that performs inter-frame compression. MJPEG or Motion JPEG on the other hand, which is often used by low end cameras as well, is simply a continuous stream of individual JPEG encoded images; no inter-frame compression.

I initially considered using MJPEG for the HoloCube, even though it carries only video and audio would need to be managed separately; there exist optimised JPEG decoding libraries ready to use, such as the excellent PicoJPEG by Rich Geldreich.

I started playing around with PicoJPEG on the ESP32S3. But when it came down to performance profiling, the ESP32S3 simply didn’t have enough juice to run PicoJPEG decode on a 240x240 px frame and meet the 41.6ms frame budget needed for 24 FPS video playback.

Trying to find a solution, I remembered that a lot of web content is served using GZIP; with the browser decompressing on the go. I also tried this using the ZLib compression library on the ESP32S3. But once again, the full GZIP algorithm was just too heavy for it to handle in real-time.

Digging a bit further, I discovered that the GZIP format is simply a wrapper around thee DEFLATE algorithm; adding a GZIP header plus a CRC32 footer. Since we don’t really care about a corrupted frame as the next one will come very soon during live video streaming, we can ditch the GZIP format and CRC checksum and just use raw DEFLATE.

This is where I found a working compromise. Running just the DEFLATE algorithm taken from ZLib, finally the ESP32S3 was able to meet the frame budget. DEFLATE is based on LZ77 and Huffman coding, which oddly enough are the first 2 steps of JPEG decoding, before the inverse discrete cosine transform (IDCT) and color quantization/conversion steps. By compressing the raw 240x240 px RGB565 image frame using just DEFLATE the ESP32S3 is just fast enough to decode it on one of it’s cores and output the image over SPI DMA using the TFT_eSPI library (more info on the ESP32S3 side in the next blog post). Effectively, I managed to skip the IDCT part of the JPEG algorithm which is the most computationally expensive, and just work with DEFLATE directly.

Now that we had a suitable compression algorithm, the next step was to create a custom video format that could hold the appropriate header information for parsing and all of the video frames & audio slices pre-encoded for easy streaming to the ESP32S3.

MikV Header

Keeping things simple, with minimal overhead for playing MikV files, I established the following file structure:

[HEADER (16 bytes)] → [FRAME 1] → [FRAME 2] → ... → [FRAME N]

The 16 byte header encodes basic information about the video and audio, so that the playback device can setup it’s peripherals accordingly; as well as a magic number header for validation. The header format is as follows:

Offset Size Type Description
0 4 char Magic number “MikV”
4 1 uint8 Frames per second (FPS)
5 2 uint16 Video frame height
7 2 uint16 Video frame width
9 2 uint16 Audio sample rate
11 1 uint8 Audio channels
12 2 uint16 Audio bits per sample
14 2 uint16 Reserved (padding)

This is then followed by repeating frames, which each contain 1 encoded image and 1 slice of raw PCM audio (with a proportional amount of samples for the given FPS of the MikV file).

MikV Frame

Each frame has both video (1 frame) and audio slice data. Due to LZ77 + Huffman coding, the picture frame encoded length is variable, and a length header is included per frame block. The frames are organised as follows:

Offset Size Type Description
0 4 uint32 Video frame length (compressed)
4 N []byte Compressed video frame data (DEFLATE)
4+N 4 uint32 Audio frame length (uncompressed)
8+N M []byte Uncompressed audio frame data (PCM)

Where:

In this case, although the video frame data compressed data length is variable, the uncompressed audio frame data length is fixed and can be worked out from the sample rate, number of channels and FPS from the file header:

$$ M = N_{\text{channels}} \cdot N_{\text{bits/sample}} \cdot f_{s} \cdot \frac{1}{f_{\text{FPS}}} $$

For example; with the default 16-bit 16kHz mono audio at 20 FPS configuration:

$$ M = 1 \cdot 2 \cdot 16,000 \cdot \frac{1}{20} = 1600 \text{ bytes} $$

This gives us the following hex layout for the MikV format:

0x0000:  4D 69 6B 56 14 00 F0 00 F0 00 80 3E 01 10 00 00 00  (Header: "MikV", FPS=20, 240x240, 16kHz, 1ch, 16-bit)
0x0010:  70 2F 00 00 [...12144 bytes compressed video data...] 40 06 00 00 [...1600 bytes PCM audio data...]
0x4000:  EC 2E 00 00 [...12012 bytes compressed video data...] 40 06 00 00 [...1600 bytes PCM audio data...]
0x8000:  28 2D 00 00 [...11560 bytes compressed video data...] 40 06 00 00 [...1600 bytes PCM audio data...]

One important thing to note: I am using the Zlib implementation of DEFLATE here, which is not compatible with the libdeflate implementation. Both the ESP32S3 code and the Go code uses ZLib’s DEFLATE module.

Go Lang Implementation

The Go lang library implementation for MikV format follows a classic OOP implementation, providing the following methods:

 1func (m *MikVFormat) Load(filename string) error
 2func (m *MikVFormat) Save(filename string) error
 3func (m *MikVFormat) Stream(frameChan chan<- MikVFrame, stopChan <-chan struct{})
 4func (m *MikVFormat) AddFrame(rawImage []byte, audioSamples []byte) error
 5func (m *MikVFormat) SetAudioParams(sampleRate uint16, channels uint8, bitsPerSample uint8)
 6func (m *MikVFormat) LoadFromMP4(filename string) error
 7func (m *MikVFormat) readHeader(file *os.File) error
 8func (m *MikVFormat) readFrame(file *os.File) (*MikVFrame, error)
 9func (m *MikVFormat) writeHeader(file *os.File) error
10func (m *MikVFormat) writeFrame(file *os.File, frame MikVFrame) error

The file and frame headers are implemented via a directly mapped go structs:

MikVFormat.go L132-L163
132// Frame represents a single video and audio frame
133type MikVFrame struct {
134	Video []byte
135	Audio []byte
136}
137
138// MikVFormat is a struct that represents a video/audio file
139type MikVFormat struct {
140	Frames             []MikVFrame
141	FPS                uint8
142	Height             uint16
143	Width              uint16
144	AudioSampleRate    uint16
145	AudioChannels      uint8
146	AudioBitsPerSample uint8
147	Verbose            bool
148}
149
150// Header structure
151var MikVHeader = struct {
152	Magic              [4]byte
153	FPS                uint8
154	Height             uint16
155	Width              uint16
156	AudioSampleRate    uint16
157	AudioChannels      uint8
158	AudioBitsPerSample uint8
159	Reserved           [3]byte
160}{
161	Magic:    [4]byte{'M', 'i', 'k', 'V'},
162	Reserved: [3]byte{0, 0, 0}, // Default reserved bytes
163}

And compress/zlib is imported to perform the DEFLATE compression.

We can see above the internal private (starting with lower-case) functions, as well as some higher-level exported external public functions. I’d like to cover here 3 public functions; Stream, AddFrame and LoadFromMP4.

The Stream function is a helper function which takes in a MikVFrame channel, which it then reads a MikV file and pushes the individual frames into. The implementation is simple and straight-forward in Go:

MikVFormat.go L228-L248
228// -- Stream function
229func (m *MikVFormat) Stream(frameChan chan<- MikVFrame, stopChan <-chan struct{}) {
230	frameTime := time.Duration(float64(time.Second) / float64(m.FPS))
231
232	ticker := time.NewTicker(frameTime)
233	defer ticker.Stop()
234
235	for _, frame := range m.Frames {
236		select {
237
238		case <-stopChan: // Stop if requested
239			// fmt.Println("Stopping stream")
240			return
241
242		case <-ticker.C: // Send the frame at the correct time
243			frameChan <- frame
244		}
245	}
246
247	// Do not close the channel here so that it can be reused
248}

This allows for easy integration with the MikVServer class, as the MikVFormat class itself provides a streaming function taking advantage of those private functions to avoid code replication.

The AddFrame function encodes a single raw []byte frame into the MikVFrame struct and appends it to the currently open MikV file:

MikVFormat.go L252-L290
252// -- AddFrame function
253func (m *MikVFormat) AddFrame(rawImage []byte, audioSamples []byte) error {
254	// Swap byte order for zlib
255	for i := 0; i < len(rawImage); i += 2 {
256		rawImage[i], rawImage[i+1] = rawImage[i+1], rawImage[i]
257	}
258
259	// Create a new deflate writer
260	var compressedFrame bytes.Buffer
261	writer, err := zlib.NewWriterLevel(&compressedFrame, zlib.BestSpeed)
262	if err != nil {
263		return fmt.Errorf("Error creating DEFLATE writer: %v", err)
264	}
265
266	// Write the raw image data to the DEFLATE writer
267	if _, err := writer.Write(rawImage); err != nil {
268		return fmt.Errorf("Error writing DEFLATE data: %v", err)
269	}
270
271	// Make sure all data is flushed to the buffer
272	if err := writer.Flush(); err != nil {
273		return fmt.Errorf("Error flushing DEFLATE data: %v", err)
274	}
275
276	// Close the writer
277	if err := writer.Close(); err != nil {
278		return fmt.Errorf("Error closing DEFLATE writer: %v", err)
279	}
280
281	// Create a new frame with compressed video data and raw audio data
282	frame := MikVFrame{
283		Video: compressedFrame.Bytes(),
284		Audio: audioSamples,
285	}
286
287	// Append the frame to the MikVFormat
288	m.Frames = append(m.Frames, frame)
289	return nil
290}

This function takes an already pre-processed RGB565 frame and PCM audio slice, and simply populates the frame header and performs the DEFLATE compression using ZLib and the zlib.BestSpeed preset to ensure fast decoding times on the client (ESP32S3) side.

Last but not least, the LoadFromMP4 function includes a whole process to convert an MP4 file’s frames into RGB565 format, before calling AddFrame repeatedly to insert every frame into a MikVFormat struct. It also calls ffmpeg to do the audio conversion as required to obtain the appropriate audio slices:

MikVFormat.go L299-L444
299// -- LoadFromMP4 function
300func (m *MikVFormat) LoadFromMP4(filename string) error {
301	// Load an MP4 file and extract frames
302	if m.Verbose {
303		fmt.Printf("Converting MP4 file %s to MikV format\n", filename)
304	}
305
306	// 0. Check if MikVFormat has audio params
307	if m.AudioSampleRate == 0 || m.AudioChannels == 0 || m.AudioBitsPerSample == 0 {
308		return fmt.Errorf("Audio parameters not set")
309	}
310
311	// 1. Use VidIO to extract video frames
312	video, err := vidio.NewVideo(filename)
313	if err != nil {
314		return fmt.Errorf("Error opening video file: %v", err)
315	}
316	m.FPS = uint8(video.FPS())
317	m.Width = uint16(video.Width())
318	m.Height = uint16(video.Height())
319
320	var imageBytes [][]byte
321	imageBytes = make([][]byte, video.Frames())
322	vbuffer := image.NewRGBA(image.Rect(0, 0, int(m.Width), int(m.Height)))
323	video.SetFrameBuffer(vbuffer.Pix)
324	currentFrameI := 0
325
326	var vbar *progressbar.ProgressBar
327	if m.Verbose {
328		vbar = progressbar.Default(int64(video.Frames()), "Converting video frames")
329	}
330
331	for video.Read() {
332		// 2. Convert the frames to RGB565 format
333		rgb565buf := make([]uint16, m.Width*m.Height)
334
335		for y := 0; y < int(m.Height); y++ {
336			for x := 0; x < int(m.Width); x++ {
337				r, g, b, _ := vbuffer.At(x, y).RGBA()
338				r5 := uint16((r >> 11) & 0x1F)
339				g6 := uint16((g >> 10) & 0x3F)
340				b5 := uint16((b >> 11) & 0x1F)
341				rgb565buf[y*int(m.Width)+x] = (r5 << 11) | (g6 << 5) | b5
342			}
343		}
344		rawImage := make([]byte, len(rgb565buf)*2)
345		for i, v := range rgb565buf {
346			rawImage[i*2] = byte(v)
347			rawImage[i*2+1] = byte(v >> 8)
348		}
349
350		if m.Verbose {
351			vbar.Add(1)
352		}
353
354		imageBytes[currentFrameI] = rawImage
355		// imageBytes = append(imageBytes, rawImage)
356		currentFrameI++
357	}
358
359	if m.Verbose {
360		vbar.Finish()
361	}
362
363	// 3. Use ffmpeg to convert the audio to the right format
364	audioFrameLength := int((int(m.AudioSampleRate) / int(m.FPS)) * int(m.AudioChannels) * int(m.AudioBitsPerSample) / 8)
365
366	var abar *progressbar.ProgressBar
367	if m.Verbose {
368		abar = progressbar.Default(int64(len(imageBytes)*audioFrameLength), "Transcoding audio")
369	}
370	var audioFormat string
371	if m.AudioBitsPerSample == 16 {
372		audioFormat = "s16le"
373	} else if m.AudioBitsPerSample == 8 {
374		audioFormat = "u8"
375	}
376	cmd := exec.Command("ffmpeg", "-i", filename, "-ar", fmt.Sprintf("%d", m.AudioSampleRate), "-f", audioFormat, "-ac", fmt.Sprintf("%d", m.AudioChannels), "pipe:1")
377	audioReader, err := cmd.StdoutPipe()
378	if err != nil {
379		return fmt.Errorf("Error creating audio pipe: %v", err)
380	}
381	if err := cmd.Start(); err != nil {
382		return fmt.Errorf("Error starting ffmpeg: %v", err)
383	}
384	// 3.1 Read audio frames from the pipe
385	var audioBytes []byte
386	stdoutBuffer := make([]byte, 4096)
387	for {
388		n, err := audioReader.Read(stdoutBuffer)
389		if n > 0 {
390			audioBytes = append(audioBytes, stdoutBuffer[:n]...)
391
392			if m.Verbose {
393				abar.Add(n)
394			}
395		}
396		if err != nil {
397			if err == io.EOF {
398				break
399			}
400			return fmt.Errorf("Error reading audio data: %v", err)
401		}
402	}
403	if err := cmd.Wait(); err != nil {
404		return fmt.Errorf("Error waiting for ffmpeg: %v", err)
405	}
406
407	if m.Verbose {
408		abar.Finish()
409	}
410
411	// 4. Add the  frames to the MikVFormat
412	var mbar *progressbar.ProgressBar
413	if m.Verbose {
414		mbar = progressbar.Default(int64(len(imageBytes)), "Adding frames to MikV")
415	}
416
417	for i, rawImage := range imageBytes {
418		start := i * audioFrameLength
419		end := start + audioFrameLength
420		if end > len(audioBytes) {
421			end = len(audioBytes)
422		}
423		if start > len(audioBytes) {
424			fmt.Println("Audio data shorter than video, leaving tail as silence")
425			break
426		}
427
428		audioFrame := make([]byte, audioFrameLength)
429		copy(audioFrame, audioBytes[start:end])
430		if err := m.AddFrame(rawImage, audioFrame); err != nil {
431			return fmt.Errorf("Error adding frame: %v", err)
432		}
433
434		if m.Verbose {
435			mbar.Add(1)
436		}
437	}
438
439	if m.Verbose {
440		fmt.Printf("Loaded %d frames from MP4 file\n", len(m.Frames))
441	}
442
443	return nil
444}

This function provides an easy-to-use run-time converter from MP4 to MikV format conversion, in case the videos aren’t pre-processed into MikV format using the CLI tool ahead of time.

And that is all for the implementation of my custom MikV video format for this project, see all implementation details in pkg/mikvformat/MikVFormat.go on the GitHub repository. I coded this on the high-speed bullet train on a trip back to university from Nanjing to Beijing, and I am quite proud of this clean implementation I achieved while coding mostly offline during the four and a half hour journey.

Next Steps

This was quite a code-heavy post, but sometimes knowing how to code properly is the make-or-break factor for tight requirement embedded projects. I hope it was an enjoyable journey and exploration into compression formats and file encoding. On the next post in this series I will cover the software and hardware on the ESP32S3 side of things and finalise (maybe?) my hologram-ready concert playing HoloCube project.

MikV format and frame, side by side.

Categories
Tags