TinyMikuServer - HoloCube Part 2
Contents
This is the second part in a three part series on how I built a little HoloCube hologram video player and custom software dedicated to playing back hologram MMD concerts on this device. In this post I will cover the custom video & audio playback format I developed in order to be able to stream to the ESP32S3 in real time and render 240x240 px video plus 16kHz mono audio.
Introduction & Purpose
In the previous post, I covered how I implemented content-aware video cropping to convert the 16:9 MMD hologram videos into a suitable 1:1 video for playback on the HoloCube device; please see the previous post for context.
The next step is to implement a way to stream the generated 240x240px videos and audio to the ESP32S3 MCU in the HoloCube device in a format that is relatively compressed, as to not overload the MCU’s WiFi bandwidth, while at the same time not taking so much computational power to decode that the MCU fails to decode the video/audio for playback in real-time.
These are the main goals and features for this stage of the project:
- Custom MikV Format: A lightweight binary format optimized for storing synchronized video and audio frames with DEFLATE compression
- Streaming Server: HTTP-based server for real-time streaming of MikV content
- Video Converter: Tool to convert MP4 files to the MikV format
- HoloCropper: Content-aware video cropping tool for preparing hologram-style videos (1:1 aspect ratio)
- Optimized for Edge Devices: Designed for efficient streaming to embedded systems and IoT devices
First I’ll detail the layout and organisation of this code repository as it contains packaged versions of several tools, then I’ll go in depth on the implementation of the custom MikV video format. The repository is organised as follows:
TinyMikuServer/
├── main.go # Main server application
├── pkg/
│ ├── mikvformat/ # MikV format library
│ │ ├── MikVFormat.go # Core format implementation
│ │ └── cmd/
│ │ └── mikv-converter/ # MP4 to MikV converter CLI
│ ├── mikvserver/ # HTTP streaming server
│ │ ├── MikVServer.go # Server implementation
│ │ └── server.go # HTTP handlers
│ └── holocropper/ # Video cropping utility
│ ├── holocropper.go # Core cropping logic
│ └── cmd/
│ └── holocropper/ # HoloCropper CLI
├── bin/ # Compiled binaries (generated)
└── assets/ # Video assets (not tracked)
├── mp4/ # Processed MP4 files
├── mp4-full/ # Original full videos
└── mikv/ # Converted MikV files
Additionally, a Makefile is included to facilitate building all the different tools. For more details see the repository on GitHub:
The overall goal is to develop a software framework that allows for easy conversion from a downloaded MMD hologram-ready video to a MikV format that is ready to stream for real-time playback on the HoloCube. Let’s start by organising and cleaning up what was implemented before.
A Much-Needed Code Refactor
I began by refactoring much of the code I had developed for the previous part, the content-aware 1:1 video cropper. I started by stripping out the UI elements and converting the code into a library that I can import and use in this video server project. A standalone command-line tool was left as a bare-bones cropping tool in pkg/holocropper/cmd/holocropper.
Then I proceeded by also implementing my new MikV video format encoder as a library as well. Accompanied by it’s respective command-line conversion tool in pkg/mikvformat/cmd/mikv-converter so that videos can be pre-converted into MikV format for the server to stream.
This left a simple interface for our main.go, simply select a .mikv file to stream and the ESP32S3 on the HoloCube connects to this server to stream the video over HTTP.
HoloCropper
After stripping out the Ebitten UI that was used heavily during the cropping algorithm development detailed in the previous post in this series, the command line arguments were streamlined and set with sensible defaults for the HoloCube device. Below is the help prompt and a full usage example:
$ ./holocropper -h
flag needs an argument: -h
Usage of ./holocropper:
-fps float
Frames per second for the cropped video (default 20)
-h int
Height of the cropped video (default 240)
-i string
Path to the input video file (default: current folder) (default ".")
-o string
Directory to save the cropped video
-w int
Width of the cropped video (default 240)
exit status 2
$ ./holocropper -i ../../assets/miku_snippet.mp4 -o ../../assets/out
Input Video Path: ../../assets/miku_snippet.mp4
Output Directory: ../../assets/out
Crop Width: 240
Crop Height: 240
Frames Per Second: 20.00
Starting processing...
/usr/sbin/ffmpeg -i ~/Documents/Github/TinyMikuServer/assets/miku_snippet.mp4 -vf unsharp=5:5:1.0:5:5:0.0,scale=426:240 -sws_flags lanczos+accurate_rnd -r 20 /tmp/tinyvideo3164542692/miku_snippet.mp4 -y
100% |███████████████████████████████████████████████████████████| (768/768, 631 it/s)
Starting crop processing for ~/Documents/Github/TinyMikuServer/assets/miku_snippet.mp4
100% |███████████████████████████████████████████████████████████| (768/768, 1167 it/s)
/usr/sbin/ffmpeg -r 20 -framerate 20 -i /tmp/tinyframes2312036518/%d.jpg -i ~/Documents/Github/TinyMikuServer/assets/miku_snippet.mp4 -map 0:v:0 -map 1:a:0 -c:v mjpeg -q:v 6 -c:a copy ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4 -y
100% |███████████████████████████████████████████████████████████| (642/642, 7759 it/s)
Processing complete for ../../assets/miku_snippet.mp4
Processing complete!
MikV Format
The same approach was taken with the MikV format converter CLI, using sensible defaults and simple command line arguments to convert a holo-cropped video straight into the required format. Below is the help prompt and a full usage example:
$ ./mikv-converter -h
Usage of ./mikv-converter:
-bitsPerSample uint
Audio bits per sample (default: 16)
-channels uint
Number of audio channels (default: 1)
-i string
Input MP4 file or directory (default: current directory) (default ".")
-o string
Output directory for MikV files (default: current directory) (default ".")
-sampleRate uint
Audio sample rate (default: 16000)
-silent
Disable verbose output
$ ./mikv-converter -i ../../../../assets/out/miku_snippet.mp4 -o ../../../../assets/out
Processing file: ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4
Converting MP4 file ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4 to MikV format
Converting video frames 100% |███████████████████████████████████████████████████████████| (642/642, 4073 it/s)
Transcoding audio 100% |███████████████████████████████████████████████████████████| (1027200/1027200, 29290731 it/s)
Adding frames to MikV 84% |███████████████████████████████████████████████████████████| (541/642, 2761 it/s) [0s:0s]Audio data shorter than video, leaving tail as silence
Loaded 641 frames from MP4 file
Successfully converted ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mp4 to ~/Documents/Github/TinyMikuServer/assets/out/miku_snippet.mikv
MikV Server
Last but not least, the MikV server itself is a simple HTTP server that holds an internal MikV player which plays the frames back in real time. Then a channel is used to grab the current frame and serve it over the /video endpoint. A special X-Video-Length HTTP header is used to indicate the length in bytes of the current JPEG frame; with the rest of the data appended after that byte position being the current PCM encoded audio slice. This way the ESP32S3 can easily separate the video and audio elements from the response. This is done using the functions below:
32func (ms *MikVServer) streamer() {
33 // Take input frames and set lastframe
34 for {
35 select {
36 case frame, ok := <-ms.Stream:
37 if !ok {
38 // Input channel closed
39 return
40 }
41
42 // Grab frame lock & set last frame
43 ms.frameLock.Lock()
44 ms.lastFrame = frame
45 ms.frameLock.Unlock()
46 }
47 }
48}
49
50func (ms *MikVServer) videoHandler(w http.ResponseWriter, r *http.Request) {
51 // Grab frame lock
52 ms.frameLock.Lock()
53 defer ms.frameLock.Unlock()
54
55 // Check if we have a frame
56 if len(ms.lastFrame.Video) == 0 {
57 w.WriteHeader(http.StatusNoContent)
58 return
59 }
60
61 // Set headers
62 w.Header().Set("Content-Type", "application/octet-stream")
63
64 audioFrameLength := len(ms.lastFrame.Audio)
65 videoFrameLength := len(ms.lastFrame.Video)
66 totalFrameLength := audioFrameLength + videoFrameLength
67 w.Header().Set("Content-Length", strconv.Itoa(totalFrameLength))
68 w.Header().Set("X-Video-Length", strconv.Itoa(videoFrameLength))
69
70 // Prepare the response
71 totalFrameBuffer := make([]byte, totalFrameLength)
72 copy(totalFrameBuffer, ms.lastFrame.Video)
73 copy(totalFrameBuffer[videoFrameLength:], ms.lastFrame.Audio)
74
75 // Write the response
76 w.WriteHeader(http.StatusOK)
77 num, err := w.Write(totalFrameBuffer)
78 if err != nil {
79 fmt.Printf("Error streaming frame: %v\n", err)
80 } else if num != totalFrameLength {
81 fmt.Printf("Error streaming frame: Only %d/%d bytes sent.\n", num, totalFrameLength)
82 }
83}The reason that this endpoint simply streams the current frame from the MikV file and has it’s own internal video playback on the server side is simple; to ensure real-time playback even if lag or slow-downs occur on the ESP32S3 side.
The ESP32S3 MCU simply fetches the current frame and audio fragment continuously, and even if frames are dropped, the video playback remains locked onto real-time playback. This causes a minor audio-glitch as the audio samples don’t quite line up if frames are dropped; but its a valid architectural decision to ensure the ESP32S3 doesn’t have to try and keep up with buffering the full MikV file or keeping it’s own playback real-time locked.
This is all exposed with a very straight forward interface for us to call in our main.go:
10func main() {
11 fmt.Println("Starting...")
12
13 mikv := mikvformat.NewMikVFormat()
14 mikv.Load("./assets/mikv/39.mikv")
15
16 fmt.Println("Loaded MikVFormat")
17
18 server := mikvserver.NewMikVServer()
19 server.Start()
20
21 fmt.Println("Started MikVServer")
22
23 mikv.Stream(server.Stream, nil)
24}Now that the tooling and organisation is out of the way, I’ll focus the rest of this post into going more in depth on the design an implementation of the MikV video format.
Developing a Custom Video Format
The 2 most common video formats used for video live-streaming on lower-end devices are MP4 and MJPEG. MP4 is way more ubiquitous, but also requires comparatively high levels of compute to decode as it’s an advanced and efficient algorithm that performs inter-frame compression. MJPEG or Motion JPEG on the other hand, which is often used by low end cameras as well, is simply a continuous stream of individual JPEG encoded images; no inter-frame compression.
I initially considered using MJPEG for the HoloCube, even though it carries only video and audio would need to be managed separately; there exist optimised JPEG decoding libraries ready to use, such as the excellent PicoJPEG by Rich Geldreich.
I started playing around with PicoJPEG on the ESP32S3. But when it came down to performance profiling, the ESP32S3 simply didn’t have enough juice to run PicoJPEG decode on a 240x240 px frame and meet the 41.6ms frame budget needed for 24 FPS video playback.
Trying to find a solution, I remembered that a lot of web content is served using GZIP; with the browser decompressing on the go. I also tried this using the ZLib compression library on the ESP32S3. But once again, the full GZIP algorithm was just too heavy for it to handle in real-time.
Digging a bit further, I discovered that the GZIP format is simply a wrapper around thee DEFLATE algorithm; adding a GZIP header plus a CRC32 footer. Since we don’t really care about a corrupted frame as the next one will come very soon during live video streaming, we can ditch the GZIP format and CRC checksum and just use raw DEFLATE.
This is where I found a working compromise. Running just the DEFLATE algorithm taken from ZLib, finally the ESP32S3 was able to meet the frame budget. DEFLATE is based on LZ77 and Huffman coding, which oddly enough are the first 2 steps of JPEG decoding, before the inverse discrete cosine transform (IDCT) and color quantization/conversion steps. By compressing the raw 240x240 px RGB565 image frame using just DEFLATE the ESP32S3 is just fast enough to decode it on one of it’s cores and output the image over SPI DMA using the TFT_eSPI library (more info on the ESP32S3 side in the next blog post). Effectively, I managed to skip the IDCT part of the JPEG algorithm which is the most computationally expensive, and just work with DEFLATE directly.
Now that we had a suitable compression algorithm, the next step was to create a custom video format that could hold the appropriate header information for parsing and all of the video frames & audio slices pre-encoded for easy streaming to the ESP32S3.
MikV Header
Keeping things simple, with minimal overhead for playing MikV files, I established the following file structure:
[HEADER (16 bytes)] → [FRAME 1] → [FRAME 2] → ... → [FRAME N]
The 16 byte header encodes basic information about the video and audio, so that the playback device can setup it’s peripherals accordingly; as well as a magic number header for validation. The header format is as follows:
| Offset | Size | Type | Description |
|---|---|---|---|
| 0 | 4 | char | Magic number “MikV” |
| 4 | 1 | uint8 | Frames per second (FPS) |
| 5 | 2 | uint16 | Video frame height |
| 7 | 2 | uint16 | Video frame width |
| 9 | 2 | uint16 | Audio sample rate |
| 11 | 1 | uint8 | Audio channels |
| 12 | 2 | uint16 | Audio bits per sample |
| 14 | 2 | uint16 | Reserved (padding) |
This is then followed by repeating frames, which each contain 1 encoded image and 1 slice of raw PCM audio (with a proportional amount of samples for the given FPS of the MikV file).
MikV Frame
Each frame has both video (1 frame) and audio slice data. Due to LZ77 + Huffman coding, the picture frame encoded length is variable, and a length header is included per frame block. The frames are organised as follows:
| Offset | Size | Type | Description |
|---|---|---|---|
| 0 | 4 | uint32 | Video frame length (compressed) |
| 4 | N | []byte | Compressed video frame data (DEFLATE) |
| 4+N | 4 | uint32 | Audio frame length (uncompressed) |
| 8+N | M | []byte | Uncompressed audio frame data (PCM) |
Where:
- $N$ = Video frame length (compressed)
- $M$ = Audio frame length (uncompressed)
In this case, although the video frame data compressed data length is variable, the uncompressed audio frame data length is fixed and can be worked out from the sample rate, number of channels and FPS from the file header:
$$ M = N_{\text{channels}} \cdot N_{\text{bits/sample}} \cdot f_{s} \cdot \frac{1}{f_{\text{FPS}}} $$
For example; with the default 16-bit 16kHz mono audio at 20 FPS configuration:
$$ M = 1 \cdot 2 \cdot 16,000 \cdot \frac{1}{20} = 1600 \text{ bytes} $$
This gives us the following hex layout for the MikV format:
0x0000: 4D 69 6B 56 14 00 F0 00 F0 00 80 3E 01 10 00 00 00 (Header: "MikV", FPS=20, 240x240, 16kHz, 1ch, 16-bit)
0x0010: 70 2F 00 00 [...12144 bytes compressed video data...] 40 06 00 00 [...1600 bytes PCM audio data...]
0x4000: EC 2E 00 00 [...12012 bytes compressed video data...] 40 06 00 00 [...1600 bytes PCM audio data...]
0x8000: 28 2D 00 00 [...11560 bytes compressed video data...] 40 06 00 00 [...1600 bytes PCM audio data...]
One important thing to note: I am using the Zlib implementation of DEFLATE here, which is not compatible with the libdeflate implementation. Both the ESP32S3 code and the Go code uses ZLib’s DEFLATE module.
Go Lang Implementation
The Go lang library implementation for MikV format follows a classic OOP implementation, providing the following methods:
1func (m *MikVFormat) Load(filename string) error
2func (m *MikVFormat) Save(filename string) error
3func (m *MikVFormat) Stream(frameChan chan<- MikVFrame, stopChan <-chan struct{})
4func (m *MikVFormat) AddFrame(rawImage []byte, audioSamples []byte) error
5func (m *MikVFormat) SetAudioParams(sampleRate uint16, channels uint8, bitsPerSample uint8)
6func (m *MikVFormat) LoadFromMP4(filename string) error
7func (m *MikVFormat) readHeader(file *os.File) error
8func (m *MikVFormat) readFrame(file *os.File) (*MikVFrame, error)
9func (m *MikVFormat) writeHeader(file *os.File) error
10func (m *MikVFormat) writeFrame(file *os.File, frame MikVFrame) error
The file and frame headers are implemented via a directly mapped go structs:
132// Frame represents a single video and audio frame
133type MikVFrame struct {
134 Video []byte
135 Audio []byte
136}
137
138// MikVFormat is a struct that represents a video/audio file
139type MikVFormat struct {
140 Frames []MikVFrame
141 FPS uint8
142 Height uint16
143 Width uint16
144 AudioSampleRate uint16
145 AudioChannels uint8
146 AudioBitsPerSample uint8
147 Verbose bool
148}
149
150// Header structure
151var MikVHeader = struct {
152 Magic [4]byte
153 FPS uint8
154 Height uint16
155 Width uint16
156 AudioSampleRate uint16
157 AudioChannels uint8
158 AudioBitsPerSample uint8
159 Reserved [3]byte
160}{
161 Magic: [4]byte{'M', 'i', 'k', 'V'},
162 Reserved: [3]byte{0, 0, 0}, // Default reserved bytes
163}And compress/zlib is imported to perform the DEFLATE compression.
We can see above the internal private (starting with lower-case) functions, as well as some higher-level exported external public functions. I’d like to cover here 3 public functions; Stream, AddFrame and LoadFromMP4.
The Stream function is a helper function which takes in a MikVFrame channel, which it then reads a MikV file and pushes the individual frames into. The implementation is simple and straight-forward in Go:
228// -- Stream function
229func (m *MikVFormat) Stream(frameChan chan<- MikVFrame, stopChan <-chan struct{}) {
230 frameTime := time.Duration(float64(time.Second) / float64(m.FPS))
231
232 ticker := time.NewTicker(frameTime)
233 defer ticker.Stop()
234
235 for _, frame := range m.Frames {
236 select {
237
238 case <-stopChan: // Stop if requested
239 // fmt.Println("Stopping stream")
240 return
241
242 case <-ticker.C: // Send the frame at the correct time
243 frameChan <- frame
244 }
245 }
246
247 // Do not close the channel here so that it can be reused
248}This allows for easy integration with the MikVServer class, as the MikVFormat class itself provides a streaming function taking advantage of those private functions to avoid code replication.
The AddFrame function encodes a single raw []byte frame into the MikVFrame struct and appends it to the currently open MikV file:
252// -- AddFrame function
253func (m *MikVFormat) AddFrame(rawImage []byte, audioSamples []byte) error {
254 // Swap byte order for zlib
255 for i := 0; i < len(rawImage); i += 2 {
256 rawImage[i], rawImage[i+1] = rawImage[i+1], rawImage[i]
257 }
258
259 // Create a new deflate writer
260 var compressedFrame bytes.Buffer
261 writer, err := zlib.NewWriterLevel(&compressedFrame, zlib.BestSpeed)
262 if err != nil {
263 return fmt.Errorf("Error creating DEFLATE writer: %v", err)
264 }
265
266 // Write the raw image data to the DEFLATE writer
267 if _, err := writer.Write(rawImage); err != nil {
268 return fmt.Errorf("Error writing DEFLATE data: %v", err)
269 }
270
271 // Make sure all data is flushed to the buffer
272 if err := writer.Flush(); err != nil {
273 return fmt.Errorf("Error flushing DEFLATE data: %v", err)
274 }
275
276 // Close the writer
277 if err := writer.Close(); err != nil {
278 return fmt.Errorf("Error closing DEFLATE writer: %v", err)
279 }
280
281 // Create a new frame with compressed video data and raw audio data
282 frame := MikVFrame{
283 Video: compressedFrame.Bytes(),
284 Audio: audioSamples,
285 }
286
287 // Append the frame to the MikVFormat
288 m.Frames = append(m.Frames, frame)
289 return nil
290}This function takes an already pre-processed RGB565 frame and PCM audio slice, and simply populates the frame header and performs the DEFLATE compression using ZLib and the zlib.BestSpeed preset to ensure fast decoding times on the client (ESP32S3) side.
Last but not least, the LoadFromMP4 function includes a whole process to convert an MP4 file’s frames into RGB565 format, before calling AddFrame repeatedly to insert every frame into a MikVFormat struct. It also calls ffmpeg to do the audio conversion as required to obtain the appropriate audio slices:
299// -- LoadFromMP4 function
300func (m *MikVFormat) LoadFromMP4(filename string) error {
301 // Load an MP4 file and extract frames
302 if m.Verbose {
303 fmt.Printf("Converting MP4 file %s to MikV format\n", filename)
304 }
305
306 // 0. Check if MikVFormat has audio params
307 if m.AudioSampleRate == 0 || m.AudioChannels == 0 || m.AudioBitsPerSample == 0 {
308 return fmt.Errorf("Audio parameters not set")
309 }
310
311 // 1. Use VidIO to extract video frames
312 video, err := vidio.NewVideo(filename)
313 if err != nil {
314 return fmt.Errorf("Error opening video file: %v", err)
315 }
316 m.FPS = uint8(video.FPS())
317 m.Width = uint16(video.Width())
318 m.Height = uint16(video.Height())
319
320 var imageBytes [][]byte
321 imageBytes = make([][]byte, video.Frames())
322 vbuffer := image.NewRGBA(image.Rect(0, 0, int(m.Width), int(m.Height)))
323 video.SetFrameBuffer(vbuffer.Pix)
324 currentFrameI := 0
325
326 var vbar *progressbar.ProgressBar
327 if m.Verbose {
328 vbar = progressbar.Default(int64(video.Frames()), "Converting video frames")
329 }
330
331 for video.Read() {
332 // 2. Convert the frames to RGB565 format
333 rgb565buf := make([]uint16, m.Width*m.Height)
334
335 for y := 0; y < int(m.Height); y++ {
336 for x := 0; x < int(m.Width); x++ {
337 r, g, b, _ := vbuffer.At(x, y).RGBA()
338 r5 := uint16((r >> 11) & 0x1F)
339 g6 := uint16((g >> 10) & 0x3F)
340 b5 := uint16((b >> 11) & 0x1F)
341 rgb565buf[y*int(m.Width)+x] = (r5 << 11) | (g6 << 5) | b5
342 }
343 }
344 rawImage := make([]byte, len(rgb565buf)*2)
345 for i, v := range rgb565buf {
346 rawImage[i*2] = byte(v)
347 rawImage[i*2+1] = byte(v >> 8)
348 }
349
350 if m.Verbose {
351 vbar.Add(1)
352 }
353
354 imageBytes[currentFrameI] = rawImage
355 // imageBytes = append(imageBytes, rawImage)
356 currentFrameI++
357 }
358
359 if m.Verbose {
360 vbar.Finish()
361 }
362
363 // 3. Use ffmpeg to convert the audio to the right format
364 audioFrameLength := int((int(m.AudioSampleRate) / int(m.FPS)) * int(m.AudioChannels) * int(m.AudioBitsPerSample) / 8)
365
366 var abar *progressbar.ProgressBar
367 if m.Verbose {
368 abar = progressbar.Default(int64(len(imageBytes)*audioFrameLength), "Transcoding audio")
369 }
370 var audioFormat string
371 if m.AudioBitsPerSample == 16 {
372 audioFormat = "s16le"
373 } else if m.AudioBitsPerSample == 8 {
374 audioFormat = "u8"
375 }
376 cmd := exec.Command("ffmpeg", "-i", filename, "-ar", fmt.Sprintf("%d", m.AudioSampleRate), "-f", audioFormat, "-ac", fmt.Sprintf("%d", m.AudioChannels), "pipe:1")
377 audioReader, err := cmd.StdoutPipe()
378 if err != nil {
379 return fmt.Errorf("Error creating audio pipe: %v", err)
380 }
381 if err := cmd.Start(); err != nil {
382 return fmt.Errorf("Error starting ffmpeg: %v", err)
383 }
384 // 3.1 Read audio frames from the pipe
385 var audioBytes []byte
386 stdoutBuffer := make([]byte, 4096)
387 for {
388 n, err := audioReader.Read(stdoutBuffer)
389 if n > 0 {
390 audioBytes = append(audioBytes, stdoutBuffer[:n]...)
391
392 if m.Verbose {
393 abar.Add(n)
394 }
395 }
396 if err != nil {
397 if err == io.EOF {
398 break
399 }
400 return fmt.Errorf("Error reading audio data: %v", err)
401 }
402 }
403 if err := cmd.Wait(); err != nil {
404 return fmt.Errorf("Error waiting for ffmpeg: %v", err)
405 }
406
407 if m.Verbose {
408 abar.Finish()
409 }
410
411 // 4. Add the frames to the MikVFormat
412 var mbar *progressbar.ProgressBar
413 if m.Verbose {
414 mbar = progressbar.Default(int64(len(imageBytes)), "Adding frames to MikV")
415 }
416
417 for i, rawImage := range imageBytes {
418 start := i * audioFrameLength
419 end := start + audioFrameLength
420 if end > len(audioBytes) {
421 end = len(audioBytes)
422 }
423 if start > len(audioBytes) {
424 fmt.Println("Audio data shorter than video, leaving tail as silence")
425 break
426 }
427
428 audioFrame := make([]byte, audioFrameLength)
429 copy(audioFrame, audioBytes[start:end])
430 if err := m.AddFrame(rawImage, audioFrame); err != nil {
431 return fmt.Errorf("Error adding frame: %v", err)
432 }
433
434 if m.Verbose {
435 mbar.Add(1)
436 }
437 }
438
439 if m.Verbose {
440 fmt.Printf("Loaded %d frames from MP4 file\n", len(m.Frames))
441 }
442
443 return nil
444}This function provides an easy-to-use run-time converter from MP4 to MikV format conversion, in case the videos aren’t pre-processed into MikV format using the CLI tool ahead of time.
And that is all for the implementation of my custom MikV video format for this project, see all implementation details in pkg/mikvformat/MikVFormat.go on the GitHub repository. I coded this on the high-speed bullet train on a trip back to university from Nanjing to Beijing, and I am quite proud of this clean implementation I achieved while coding mostly offline during the four and a half hour journey.
Next Steps
This was quite a code-heavy post, but sometimes knowing how to code properly is the make-or-break factor for tight requirement embedded projects. I hope it was an enjoyable journey and exploration into compression formats and file encoding. On the next post in this series I will cover the software and hardware on the ESP32S3 side of things and finalise (maybe?) my hologram-ready concert playing HoloCube project.