Luigi Pizzolito's Logo

Contents

This is the third part in a three part series on how I built a little HoloCube hologram video player and custom software dedicated to playing back hologram MMD concerts on this device. In this post I will cover the hardware used and the client software on the ESP32S3 which receives, decodes, and display the MikV format video streams we developed in part 2.

Hardware

For the hardware, I am mostly sticking to what was used by Peng ZhiHui’s original HoloCube. However since I am not using a custom PCB, I opted for off the shelf modules. I also added a MAX9835 I2S DAC and speaker amplifier, and a MSM261S4030H0 I2S microphone for future use. The main MCU board is the Seed Xiao ESP32S3.

Hardware Connections
1" Beam-Splitter 50/50 Prism -
Xiao ESP32S3 -
1.3" 240x240px SPI TFT ST7789 Display BL(D5), MOSI(D10), SCK(D8), DC(D7), RST(D6)
MAX9835 I2S DAC & Speaker Amp WS(D0), SCK(D1), DO(D3)
MSM261S4040H0 I2S MEMS Microphone WS(D0), SCK(D1), DI(D2)
37x18mm PC Speaker MAX9835 Speaker Out

The connections are straight forward and listed above. You can also change the pins used from the code; the display pins are setup in main.c and the I2S pins are setup in a separate lib/i2s_settings/i2s_settings.h file.

3D Printed Case

Inspired by the original HoloCube, I also designed in Fusion360 a 3D printed case which holds the screen in place at a nice viewing angle and allows me to mount the 1" beam-splitter prism. The speaker is down-firing and has some grill holes at the bottom of the case, along with some holes to add rubber feet.

The flex PCB WiFi antenna of the ESP32S3 can be mounted inside the case by adhering to one of the side walls. But I found that it decreases frame-rate performance, as we are already pushing the bandwidth close to it’s maximum. I simply left the antenna outside the case for most of the testing.

A small slot is left at the front of the case under the display, this is for the sound input to the MEMS microphone, as I eventually plan to make this into a full AI assistant kind of device.

An attempt with JPEG

Before coming up with the custom MikV video format of the last post, I started by setting up the display using the TFT_eSPI library and tested displaying images to the screen with PicoJPEG.

However, this presented two issues; first as you can see in the image above, for some reason JPEG decode at these small image sizes left some artifacts around the edges of the picture where there is a hard transition to the black background, I was never able to figure out why this was and assumed it was something to do with the original video’s MP4 compression and the conversion process from that to JPEG. Second, after testing the performance it became clear that I would not be able to meet the 50ms frame budget for 20 FPS playback doing a full JPEG decode.

Nonetheless, I continued the project using my MikV format.

Streaming & Decoding MikV

The software setup is simple and organises each module of the implementation into a separate library:

TinyMikuHolo
├── lib
│   ├── AudioBuffer
│   │   ├── AudioBuffer.cpp
│   │   └── AudioBuffer.h
│   ├── AudioPlayer
│   │   ├── AudioPlayer.cpp
│   │   └── AudioPlayer.h
│   ├── fps
│   │   └── fps.h
│   ├── i2s_settings
│   │   ├── i2s_settings.cpp
│   │   └── i2s_settings.h
│   ├── video_source
│   │   ├── video_source.cpp
│   │   └── video_source.h
│   └── zlib_turbo
├── src
│   └── main.cpp
└── platformio.ini

i2s_settings is simply an utility to define the pins used and setup I2S audio on the ESP32S3. Then AudioPlayer performs audio playback through the use of the AudioBuffer class which was necessary to ensure the playback didnt stutter. Lastly, video_source performs the decoding of MikV video and pushes the final image to TFT_eSPI which then uses DMA to output to the screen.

All of these routines run concurrently using FreeRTOS tasks, this allows me not only to simply run concurrent functions, but also to pin them to core 0 or core 1 of the ESP32S3. This way it becomes feasible to achieve real-time video and audio playback.

Video Source

video_source performs the bulk of the work of this implementation, its responsible for grabbing the MikV frame from the TinyMikuServer, as well as decoding and screen output. It exposes a simple initialiser which takes the TFT_eSPI instance, the AudioBuffer instance, and a String to the server URL:

video_source.h L50-L50
50  video_source(TFT_eSPI *tft, AudioBuffer &audioBuffer, String frame_url);

Inside the initialiser, we declare pointers to instances of TFT_eSPI, AudioBuffer, zlib_turbo and HTTPClient. As well as pointers for 2 buffers which are allocated at runtime on the ESP32S3’s external PSRAM; one compressed frame buffer uint8_t *pCompressed;, and one uncompressed frame buffer to push directly to the display uint8_t *pUncompressed;.

video_source.h L17-L40
17class video_source {
18private:
19  zlib_turbo *zt;
20  TFT_eSPI *tft;
21  FPS *fps;
22
23  uint8_t *pDL;
24  size_t pDLSize;
25
26  uint8_t *pUncompressed;
27  size_t pUncompressedSize;
28
29  uint8_t *pCompressed;
30  size_t pCompressedSize;
31
32  AudioBuffer &audioBuffer;
33
34  volatile bool frame_ready = false;
35
36  unsigned long l;
37  unsigned long lt;
38
39  String frame_url;
40  HTTPClient *http;

The class then exposes a single void start_tasks(); function which starts 2 FreeRTOS tasks; grab_frame() which runs on core 0, and decode_and_display() which runs on core 1. This function simply initiates the 2 tasks with the appropriate parameters, the tasks themselves are just simple wrappers around our two functions:

video_source.cpp L260-L280
260void video_source::start_tasks() {
261  // Create the grab task
262  // xTaskCreate(grab_task, "GrabTask", 8192, this, 1, NULL);
263  xTaskCreatePinnedToCore(grab_task, "GrabTask", 8192, this, 1, NULL,
264                          GRAB_TASK_CORE);
265
266  // Create the decode and display task
267  // xTaskCreate(decode_task, "DecodeTask", 8192, this, 1, NULL);
268  xTaskCreatePinnedToCore(decode_task, "DecodeTask", 8192, this, 1, NULL,
269                          DECODE_TASK_CORE);
270}
271
272void video_source::grab_task(void *pvParameters) {
273  video_source *vs = static_cast<video_source *>(pvParameters);
274  vs->grab_frame();
275}
276
277void video_source::decode_task(void *pvParameters) {
278  video_source *vs = static_cast<video_source *>(pvParameters);
279  vs->decode_and_display();
280}

Grab Frame

The grab frame task starts by setting up the WiFi connection, and then enters a while(true) loop. Here we wait for the bool frame_ready to be set to false which signals the other decode & display task has already taken in the current frame buffer and we can start fetching the next frame.

If it’s time to fetch a new frame, we follow these steps:

  1. Send a GET request to the server and get the size of the payload from HTTP headers:

    video_source.cpp L54-L74
    54      // Step 2: Send GET request
    55
    56      //? Set Request header
    57      // http->addHeader("X-Audio-Frame", String(audioFrame++));
    58      //? Set response headers to parse
    59      const char *headerkeys[] = {"X-Video-Length"};
    60      const size_t headerkeyscount = sizeof(headerkeys) / sizeof(headerkeys[0]);
    61      http->collectHeaders(headerkeys, headerkeyscount);
    62
    63      int httpResponseCode = http->GET();
    64      // stepTime = micros() - startTime;
    65      // Serial.printf("Step 2 (Send GET request): %lu us\n", stepTime);
    66
    67      if (httpResponseCode == 200) {
    68        // Step 3: Get the size of the payload
    69        // startTime = micros();
    70        int contentLength = http->getSize();
    71        if (contentLength <= 0) {
    72          Serial.println("Content-Length not found");
    73          continue;
    74        }

  2. Allocate or resize allocation of the memory for the compressed data buffer:

    video_source.cpp L79-L92
    79        // Step 4: Allocate or resize memory for compressed data
    80        // startTime = micros();
    81        if (pDL == nullptr) {
    82          pDL = (uint8_t *)heap_caps_malloc(contentLength, MALLOC_CAP_INTERNAL);
    83          pDLSize = contentLength;
    84        } else if (pDLSize < (size_t)contentLength) {
    85          uint8_t *newBuffer = (uint8_t *)realloc(pDL, contentLength);
    86          if (!newBuffer) {
    87            Serial.println("Failed to reallocate memory for DL data");
    88            continue;
    89          }
    90          pDL = newBuffer;
    91          pDLSize = contentLength;
    92        }

  3. Read in the HTTP response payload into our *pCompressed buffer:

    video_source.cpp L96-L121
     96        // Step 5: Read data from the stream
     97        // startTime = micros();
     98        WiFiClient *stream = http->getStreamPtr();
     99        int bytesRead = 0;
    100
    101        while (http->connected() && (bytesRead < contentLength)) {
    102          // Check how many bytes are available in the stream
    103          int bytesAvailable = stream->available();
    104          if (bytesAvailable > 0) {
    105            // Calculate the number of bytes to read in this iteration
    106            int bytesToRead = min(bytesAvailable, contentLength - bytesRead);
    107
    108            // Read directly into the pDL buffer at the correct offset
    109            int bytesReadNow = stream->read(pDL + bytesRead, bytesToRead);
    110
    111            // Update the total number of bytes read
    112            bytesRead += bytesReadNow;
    113          }
    114        }
    115
    116        if (bytesRead != contentLength) {
    117          Serial.printf("Read %d bytes instead of %d\n", bytesRead,
    118                        contentLength);
    119          Serial.println("Failed to read the entire payload");
    120          continue;
    121        }

  4. Read the special X-Video-Length HTTP header to split the payload into our video and audio buffers. *pCompressed is then re-allocated to contain just the video frame data, and the audio frame data is pushed to our audioBuffer instance:

    video_source.cpp L123-L164
    123        // Get video and audio part length
    124        String pDLVideoSizeStr = http->header("X-Video-Length");
    125        int pDLVideoSize = 0;
    126        if (pDLVideoSizeStr == "") {
    127          Serial.println("Video size not found");
    128          continue;
    129        } else {
    130          pDLVideoSize = pDLVideoSizeStr.toInt();
    131        }
    132        int pDLAudioSize = contentLength - pDLVideoSize;
    133        // Serial.printf("Video size: %d, Audio size: %d\n", pDLVideoSize,
    134        // pDLAudioSize);
    135
    136        // Allocate memory for compressed video data
    137        if (pCompressed == nullptr) {
    138          // pCompressed = (uint8_t *)heap_caps_aligned_alloc(2, pDLSize,
    139          // MALLOC_CAP_INTERNAL);
    140          pCompressed =
    141              (uint8_t *)heap_caps_malloc(pDLVideoSize, MALLOC_CAP_INTERNAL);
    142          pCompressedSize = pDLVideoSize;
    143        } else if (pCompressedSize < (size_t)pDLVideoSize) {
    144          uint8_t *newBuffer = (uint8_t *)realloc(pCompressed, pDLVideoSize);
    145          if (!newBuffer) {
    146            Serial.println("Failed to reallocate memory for compressed data");
    147            continue;
    148          }
    149          pCompressed = newBuffer;
    150          pCompressedSize = pDLVideoSize;
    151        }
    152        // Copy compressed video data to buffer for decompression
    153        memcpy(pCompressed, pDL, pDLVideoSize);
    154        frame_ready = true;
    155
    156        startTime = millis() - startTime;
    157        // Serial.printf("Frame ready in %lu ms\n", startTime);
    158        if (int(startTime) < (1000 / 20)) {
    159          vTaskDelay(max((1000 / 20) - int(startTime), 0) /
    160                     portTICK_PERIOD_MS);
    161        }
    162
    163        // Pass audio data to audio buffer
    164        audioBuffer.updateBuffer(pDL + pDLVideoSize, pDLAudioSize);
    Notice here that on line 154, when we have pushed the video frame and audio frame buffers, we set frame_ready = true;. This alerts the other decode & display task that it can start decoding the next frame.

At the end of this loop, we simply have a non-blocking vTaskDelay(1/portTICK_PERIOD_MS); so we give other tasks some time to run. Any errors are also handled in a non-blocking way by inserting a placeholder vTaskDelay of roughly the remaining runtime of the failed step. This way the entire task loop recovers seamlessly if a particular frame fails to be received and parsed.

Decode & Display

The decode & display task follows a similar structure to the grab frame task, however this time we run the main loop only when frame_ready is true rather than false. The steps are as follows:

  1. Set frame_ready = false to indicate to grab frame that it can start fetching the next frame. Then allocate the *pUncompressed raw display frame buffer:
    video_source.cpp L198-L205
    198      frame_ready = false;
    199      // Step 6: Decompress the data
    200
    201      int pUncompressedSize = TFT_WIDTH * TFT_HEIGHT * sizeof(uint16_t);
    202      if (pUncompressed == nullptr) {
    203        pUncompressed =
    204            (uint8_t *)heap_caps_malloc(pUncompressedSize, MALLOC_CAP_INTERNAL);
    205      }
  2. Call zlib_turbo’s inflate function to decompress our DEFLATE encoded RGB565 video frame and output to the buffer we allocated:
    video_source.cpp L206-L208
    206      zt->inflate_init(pUncompressed, pUncompressedSize);
    207
    208      int rc = zt->inflate(pCompressed, pCompressedSize + 8);
  3. Check the return status of the DEFLATE call, and if it succeeded, push the image to the TFT display and draw a frame counter overlay:
    video_source.cpp L210-L221
    210      if (rc == ZT_SUCCESS) {
    211
    212        // Step 7: Draw the image
    213        // startTime = micros();
    214        tft->pushImage(0, 0, TFT_WIDTH, TFT_HEIGHT, (uint16_t *)pUncompressed);
    215        // stepTime = micros() - startTime;
    216        // Serial.printf("Step 7 (Draw image): %lu us\n", stepTime);
    217
    218        fps->frame();
    219        // Serial.printf("FPS: %.2f\n", fps->getFPS());
    220        tft->setCursor(0, 0, 4);
    221        tft->printf("FPS: %.2f", fps->getFPS());

The end of this loop simply contains 1ms vTaskDelay regardless of success of failure to decode a frame, this is different than grab frame because this task needs to signal when to fetch the next frame; therefore, on a failure, we should signal to fetch the next frame straight away.

Audio Player

The audio player class is very simple, as we are receiving raw 16-bit PCM encoded audio; there is no decode needed and we can push directly to the I2S for the DAC to decode. It follows a similar task structure, with a simple initialiser taking in the audio buffer AudioPlayer(AudioBuffer &audioBuffer); and a single void start_task() public method.

The void start_task() is a simple wrapper which creates a task from the internal void audio_task() function:

AudioPlayer.cpp L10-L15
10void AudioPlayer::start_task() {
11  // Create the audio playback task on core 1.
12  xTaskCreatePinnedToCore(
13      [](void *param) { ((AudioPlayer *)param)->audioTask(); }, "AudioTask",
14      4096, this, 1, NULL, AUDIO_TASK_CORE);
15}

Then the void audio_task() function itself just has a control loop which waits for incoming data into the audio buffer. If new data came in during the right time frame, we swap the buffers and write the result directly to the I2S bus using DMA; if a timeout occurred (new audio frame didn’t come in), we fill the buffer with silence to avoid audio glitches:

AudioPlayer.cpp L17-L38
17void AudioPlayer::audioTask() {
18  size_t bytesWritten = 0;
19  while (true) {
20    // Wait up to AUDIO_FRAME_TIME_MS for new audio data.
21    if (!buffer.waitForNewData(AUDIO_FRAME_TIME_MS)) {
22      // If new data isn't available in time, fill the inactive buffer with
23      // silence.
24      buffer.fillInactiveBufferWithSilence();
25    }
26    // Swap the buffers so that the fresh (or silence-filled) data becomes
27    // active.
28    buffer.swapBuffers();
29    // Write the active buffer to I2S (this call blocks until the DMA transfer
30    // completes).
31    esp_err_t result =
32        i2s_write(I2S_NUM, (const char *)buffer.getActiveBuffer(),
33                  STEREO_BUFFER_SIZE, &bytesWritten, portMAX_DELAY);
34    if (result != ESP_OK) {
35      Serial.printf("i2s_write error: %d\n", result);
36    }
37  }
38}

Most of the heavy lifting here is done by the audio buffer itself, here we simply control the DMA sends with the correct timing.

Audio Buffer

The audio buffer class is critical for glitch-free audio playback. It’s a simple ping-ping style double buffer implementation. It contains all of the methods we should expect; such as updateBuffer, waitForNewData, getActiveBuffer and swapBuffers:

AudioBuffer.h L7-L34
 7// ===== AudioBuffer Class =====
 8// This class manages two buffers (double buffering) for stereo audio.
 9// It provides a method to update the inactive buffer with new mono PCM data
10// (converted to stereo) and a semaphore to signal when new data is available.
11class AudioBuffer {
12public:
13  AudioBuffer();
14  // Update the inactive buffer with fresh mono PCM data (in uint8_t form)
15  void updateBuffer(const uint8_t *monoData, size_t byteCount);
16  // Wait for new data (with timeout in milliseconds); returns true if data is
17  // ready.
18  bool waitForNewData(uint32_t timeout_ms);
19  // Returns a pointer to the active (ready to play) buffer.
20  const uint8_t *getActiveBuffer();
21  // Swap the active and inactive buffers.
22  void swapBuffers();
23  // Fill the inactive buffer with silence (all zeros).
24  void fillInactiveBufferWithSilence();
25
26private:
27  uint8_t buffer0[STEREO_BUFFER_SIZE];
28  uint8_t buffer1[STEREO_BUFFER_SIZE];
29  // Pointers to the active (to be played) and inactive (to be updated) buffers.
30  volatile uint8_t *activeBuffer;
31  volatile uint8_t *inactiveBuffer;
32  // Semaphore used to signal that new data is available.
33  SemaphoreHandle_t semaphore;
34};

The only unique features here are the fillInactiveBufferWithSilence() method which is used to silence a buffer when an audio frame fails to be received, and the SemaphoreHandle_t semaphore which is used to indicate that new data is available. This is needed here because we have 2 tasks concurrently accessing this buffer; the grab_frame task which pushes new audio data to the buffer, and the audio_task task which send data from the buffer to I2S.

updateBuffer fills the inactive buffer in after converting the stereo PCM samples into mono, and then signals that new data is ready:

AudioBuffer.cpp L19-L35
19void AudioBuffer::updateBuffer(const uint8_t *monoData, size_t byteCount) {
20  if (byteCount != BUFFER_SAMPLES * SAMPLE_SIZE) {
21    Serial.println("Warning: Unexpected byte count.");
22  }
23  // Convert each 16-bit mono sample (2 bytes) into stereo (copy the sample to
24  // both channels).
25  for (size_t i = 0; i < BUFFER_SAMPLES; i++) {
26    size_t monoOffset = i * 2;   // Two bytes per mono sample
27    size_t stereoOffset = i * 4; // Four bytes per stereo sample
28    inactiveBuffer[stereoOffset] = monoData[monoOffset];         // Left LSB
29    inactiveBuffer[stereoOffset + 1] = monoData[monoOffset + 1]; // Left MSB
30    inactiveBuffer[stereoOffset + 2] = monoData[monoOffset];     // Right LSB
31    inactiveBuffer[stereoOffset + 3] = monoData[monoOffset + 1]; // Right MSB
32  }
33  // Signal that new data is available.
34  xSemaphoreGive(semaphore);
35}

waitForNewData simply awaits for the semaphore signal with a given timeout in ms:

AudioBuffer.cpp L37-L39
37bool AudioBuffer::waitForNewData(uint32_t timeout_ms) {
38  return (xSemaphoreTake(semaphore, pdMS_TO_TICKS(timeout_ms)) == pdTRUE);
39}

swapBuffers simple swaps the pointer references of the active and inactive buffers:

AudioBuffer.cpp L45-L49
45void AudioBuffer::swapBuffers() {
46  volatile uint8_t *temp = activeBuffer;
47  activeBuffer = inactiveBuffer;
48  inactiveBuffer = temp;
49}

And last but not least, the fillInactiveBufferWithSilence takes the last valid audio sample received and fills the inactive buffer with that sample; in order to create silence without any sudden jump to 0 (would result in an audio click):

AudioBuffer.cpp L51-L62
51void AudioBuffer::fillInactiveBufferWithSilence() {
52  // memset((void *)inactiveBuffer, 0, STEREO_BUFFER_SIZE);
53
54  // Get the last sample (4 bytes) from the active buffer
55  uint8_t lastSample[4];
56  memcpy(lastSample, (const uint8_t *)&activeBuffer[STEREO_BUFFER_SIZE - 4], 4);
57
58  // Fill the inactive buffer with the last sample
59  for (size_t i = 0; i < STEREO_BUFFER_SIZE; i += 4) {
60    memcpy((uint8_t *)&inactiveBuffer[i], lastSample, 4);
61  }
62}

Main Sketch

With all of our functions modularised into libraries and wrapped into ready to go FreeRTOS tasks, the main.cpp sketch becomes very simple. We start by declaring our classes:

main.cpp L16-L23
16#include "video_source.h"
17video_source *vs;
18
19#include "AudioBuffer.h"
20#include "AudioPlayer.h"
21#include "i2s_settings.h"
22AudioBuffer audioBuffer;
23AudioPlayer *audioPlayer;

Then we initialise them:

main.cpp L57-L60
57  vs = new video_source(&tft, audioBuffer, "http://192.168.8.114:8000/video");
58
59  setupI2S();
60  audioPlayer = new AudioPlayer(audioBuffer);

And lastly, we start our tasks:

main.cpp L65-L66
65  vs->start_tasks();
66  audioPlayer->start_task();

The main loop simply contains a 1000ms vTaskDelay to yield time for the tasks to run.

Showcase

Finally, after uploading the sketch and starting our TinyMikuServer from the previous post, the HoloCube comes to life.

HoloCube with MMD display

Below are some video demos:

Next Steps

There we have it, full circle from video pre-processing for cropping to a 1:1 aspect ratio, to a custom video encoding format, to playback on the HoloCube.

In the future I wish to extend this project, by using the MEMS microphone I included to stream audio back the the server on a PC, where we can then run an AI agent to give Miku’s hologram some personality and live interaction.

Categories
Tags