Luigi Pizzolito's Logo

Hologram Video Cropper - HoloCube Part 1

Series: HoloCube

Contents

This is the first in a three part series on how I built a little HoloCube hologram video player and custom software dedicated to playing back hologram MMD concerts on this device. In this post I will cover the content-aware video cropper tool I built for the first video pre-processing step required to make this project come true.

Introduction to HoloCube & MMD Holograms

The HoloCube is a DIY mini hologram device originally created by the popular Chinese hobbyist 稚晖 (Peng ZhiHui) under the name HoloCubic; which I first stumbled upon on Bilibili.

He combined an ESP32S3 with a 1.3" ST7789 240x240px TFT display, with a 1" beam-splitter prism to create the hologram illusion. I love this project and especially the form factor. However, it was missing a few key features in my view to be useful as a holo-concert viewer:

Below is the GitHub repository for his original implementation.

MikuMikuDance Holo Concerts

Miku Miku Dance is a free homebrew software created by 樋口M/ヒグチエム(Yu Higuchi) in 2008 to create dance animations as part of the VOCALOID promotion video project (VPVP). Thousands of fans have used it to create animations and hologram-ready videos using it. Just like in Hatsune Miku’s live MIKU EXPO concerts, a black background video is used so that when projected on a semi-transparent screen, a hologram is created. Below is an example with the song Anamanaguchi - Miku uploaded by Mekaio:

The Problem

As one can expect from the video above, there are several issues that means these videos cannot simply be cropped in the center to a square aspect ratio needed for the HoloCube. Here are the difficulties:

With this in mind, I started designing a video processing algorithm to perform such content-aware crop work.

Implementing a Solution

I chose to favourite programming language; Go lang. This is because of it’s first-class concurrency primitives and the chan channel datatype, which allows me to easily pass live data concurrently between processing pipeline stages.

Let’s cover the pipeline stages I devised first, and then delve into the pipeline implementation itself later.

Stage 1: Convert to Grayscale

The first step is of course to read the video in and extract each frame, it is there frames which are passed along the pipeline stages. Then the first real processing step is to convert the video to grayscale as we don’t need the color data to perform cropping. This is the simplest and most straight forward part of the pipeline, and it’s done in just a few lines of code:

step2_grayscaleVideo.go L33-L43
33func convertToMonochrome(img *image.RGBA) *image.Gray {
34    monoFrame := image.NewGray(img.Bounds())
35    draw.Draw(monoFrame, monoFrame.Bounds(), img, img.Bounds().Min, draw.Src)
36    return monoFrame
37}
38
39func convertGrayToRGBA(img *image.Gray) *image.RGBA {
40    rgbaFrame := image.NewRGBA(img.Bounds())
41    draw.Draw(rgbaFrame, rgbaFrame.Bounds(), img, img.Bounds().Min, draw.Src)
42    return rgbaFrame
43}

The result is, as expected, a grayscale video:

Stage 2: Convert to Bitmap via Otsu Threshold, Glob Components, Centroids & Bounding Boxes

The next challenge is to perform object detection to obtain a bounding box for the subjects without using computer vision AI (like YOLO) for example, but rather just traditional video DSP algorithms.

To achieve this, we first simplify the problem space even further by converting the grayscale video into a bitmap video (pixels are only fully on or fully off). In order to obtain this without dithering a simple threshold is used, so that all edges stay as sharp as they can be. A threshold operation will simply set all pixels above a certain value to 1, and all pixels below to 0. In order to ensure proper contrast so that even very dark parts (like Miku’s black boots) and light parts (like skin) are captured in the outline. This is where we can apply the Otsu algorithm.

Otsu’s method, fittingly developed by Nobuyuki Otsu, is a method that first calculates a histogram over the image and then finds the optimal split between the two main area peaks by calculating the inner-class variances. Below is a visualisation of this method from Wikipedia:

Otsu’s Method Visualisation

This is implemented with the Go code below:

step3_binaryVideo.go L79-L139
 79func otsuThreshold(img *image.Gray) uint8 {
 80    // Calculate histogram
 81    histogram := make([]int, 256)
 82    totalPixels := float64(img.Bounds().Dx() * img.Bounds().Dy())
 83    for y := img.Bounds().Min.Y; y < img.Bounds().Max.Y; y++ {
 84        for x := img.Bounds().Min.X; x < img.Bounds().Max.X; x++ {
 85            intensity := img.GrayAt(x, y).Y
 86            histogram[intensity]++
 87        }
 88    }
 89
 90    // Normalize histogram
 91    var normalizedHistogram [256]float64
 92    for i := range histogram {
 93        normalizedHistogram[i] = float64(histogram[i]) / totalPixels
 94    }
 95
 96    // Calculate cumulative sum and cumulative mean
 97    var cumulativeSum [256]float64
 98    var cumulativeMean [256]float64
 99    cumulativeSum[0] = normalizedHistogram[0]
100    cumulativeMean[0] = 0
101    for i := 1; i < 256; i++ {
102        cumulativeSum[i] = cumulativeSum[i-1] + normalizedHistogram[i]
103        cumulativeMean[i] = cumulativeMean[i-1] + float64(i)*normalizedHistogram[i]
104    }
105
106    // Calculate between-class variance and find optimal threshold
107    var maxBetweenClassVariance float64
108    var optimalThreshold uint8
109    for t := 0; t < 256; t++ {
110        omega := cumulativeSum[t]
111        mu := cumulativeMean[t]
112        omega2 := 1 - omega
113        mu2 := (cumulativeMean[255] - mu) / omega2
114        betweenClassVariance := omega * omega2 * (mu - mu2) * (mu - mu2)
115        if betweenClassVariance > maxBetweenClassVariance {
116            maxBetweenClassVariance = betweenClassVariance
117            optimalThreshold = uint8(t)
118        }
119    }
120
121    return optimalThreshold
122}
123
124func applyThreshold(img *image.Gray, threshold uint8) *image.Gray {
125    binaryImg := image.NewGray(img.Bounds())
126    black := color.Gray{Y: 0}
127    white := color.Gray{Y: 255}
128
129    for y := img.Bounds().Min.Y; y < img.Bounds().Max.Y; y++ {
130        for x := img.Bounds().Min.X; x < img.Bounds().Max.X; x++ {
131            if img.GrayAt(x, y).Y < threshold {
132                binaryImg.SetGray(x, y, black)
133            } else {
134                binaryImg.SetGray(x, y, white)
135            }
136        }
137    }
138    return binaryImg
139}

After the threshold is applied, we now have a binary bitmap video. The next step is to label all connected components (adjacent pixels) with bounding boxes. This is done via a custom algorithm that utilised depth-first search (DFS) to label all connected components:

step3_binaryVideo.go L205-L284
205// Connected component labeling algorithm
206func connectedComponentLabeling(binaryMap *image.Gray) (map[int]image.Rectangle, [][]int) {
207	labelMatrix := make([][]int, binaryMap.Bounds().Dy())
208	for i := range labelMatrix {
209		labelMatrix[i] = make([]int, binaryMap.Bounds().Dx())
210	}
211
212	boundingBoxes := make(map[int]image.Rectangle)
213
214	label := 1
215	for y := binaryMap.Bounds().Min.Y; y < binaryMap.Bounds().Max.Y; y++ {
216		for x := binaryMap.Bounds().Min.X; x < binaryMap.Bounds().Max.X; x++ {
217			if binaryMap.GrayAt(x, y).Y > 0 && labelMatrix[y][x] == 0 {
218				boundingBox := dfs(binaryMap, labelMatrix, x, y, label)
219				boundingBoxes[label] = boundingBox
220				label++
221			}
222		}
223	}
224
225	return boundingBoxes, labelMatrix
226}
227
228// Depth-first search to label connected components
229func dfs(binaryMap *image.Gray, labelMatrix [][]int, x, y, label int) image.Rectangle {
230    minX, minY, maxX, maxY := x, y, x, y
231
232    stack := []image.Point{{x, y}}
233    for len(stack) > 0 {
234        current := stack[len(stack)-1]
235        stack = stack[:len(stack)-1]
236
237        cx, cy := current.X, current.Y
238        labelMatrix[cy][cx] = label
239
240        if cx < minX {
241            minX = cx
242        }
243        if cx > maxX {
244            maxX = cx
245        }
246        if cy < minY {
247            minY = cy
248        }
249        if cy > maxY {
250            maxY = cy
251        }
252
253        // Check neighboring pixels
254        for dx := -1; dx <= 1; dx++ {
255            nx, ny := cx+dx, cy-1
256            if nx >= 0 && nx < binaryMap.Bounds().Dx() && ny >= 0 && ny < binaryMap.Bounds().Dy() {
257                if binaryMap.GrayAt(nx, ny).Y > 0 && labelMatrix[ny][nx] == 0 {
258                    stack = append(stack, image.Point{nx, ny})
259                }
260            }
261            ny = cy + 1
262            if nx >= 0 && nx < binaryMap.Bounds().Dx() && ny >= 0 && ny < binaryMap.Bounds().Dy() {
263                if binaryMap.GrayAt(nx, ny).Y > 0 && labelMatrix[ny][nx] == 0 {
264                    stack = append(stack, image.Point{nx, ny})
265                }
266            }
267        }
268        nx := cx - 1
269        ny := cy
270        if nx >= 0 && nx < binaryMap.Bounds().Dx() && ny >= 0 && ny < binaryMap.Bounds().Dy() {
271            if binaryMap.GrayAt(nx, ny).Y > 0 && labelMatrix[ny][nx] == 0 {
272                stack = append(stack, image.Point{nx, ny})
273            }
274        }
275        nx = cx + 1
276        if nx >= 0 && nx < binaryMap.Bounds().Dx() && ny >= 0 && ny < binaryMap.Bounds().Dy() {
277            if binaryMap.GrayAt(nx, ny).Y > 0 && labelMatrix[ny][nx] == 0 {
278                stack = append(stack, image.Point{nx, ny})
279            }
280        }
281    }
282
283    return image.Rect(minX, minY, maxX+1, maxY+1)
284}

Since there may be gaps in the islands created by threshold-ing the video, we need to then group adjacent bounding boxes within a certain radius into one master bounding box for our subject. This is done by simply combining bounding box bounds:

step3_binaryVideo.go L184-L203
184// CombineBounds combines the bounds of all rectangles in the slice
185// that overlap with the given rectangle and removes them from the slice.
186func CombineBounds(rect image.Rectangle, rects []image.Rectangle, tol int) (combined image.Rectangle, remainingRects []image.Rectangle) {
187    // Initialize the combined rectangle with the rectangle passed as an argument
188    combined = rect
189
190    // Iterate over the rectangles in the slice
191    for _, r := range rects {
192        // Check if the current rectangle overlaps with the given rectangle
193        if ExpandRectangle(combined, tol).Overlaps(r) {
194            // Combine the bounds of overlapping rectangles
195            combined = combined.Union(r)
196        } else {
197            // Add the non-overlapping rectangle to the remaining rectangles slice
198            remainingRects = append(remainingRects, r)
199        }
200    }
201
202    return combined, remainingRects
203}

Finally, we have achieved bounding-box labelling of subjects in the video without using AI. However, one issue remains, the geometric center of the bounding box is not necessarily the visual center of our subject. Particulairly with Miku’s long twin-tail hair, which takes up more width than usual for a human-shaped subject; the geometric center does not accurately reflect the visual center.

To solve this, we use a centroid calculation algorithm, which performs a 2D weighted sum within the bounding box to find the mass center. Each white pixel effectively pulls the new center towards it, starting from the geometric center:

step3_binaryVideo.go L301-L318
301func calculateCentroid(img *image.Gray, bbox image.Rectangle) image.Point {
302    totalX, totalY, count := 0, 0, 0
303    for y := bbox.Min.Y; y < bbox.Max.Y; y++ {
304        for x := bbox.Min.X; x < bbox.Max.X; x++ {
305            if img.GrayAt(x, y).Y == 255 { // white pixel
306                totalX += x
307                totalY += y
308                count++
309            }
310        }
311    }
312
313    if count > 0 {
314        return image.Point{totalX / count, totalY / count}
315    }
316
317    return image.Point{0, 0}
318}

It is this centroid center that we use for all further calculations down the pipeline. Below is the final output from this stage:

Here we can clearly see the bitmap video, a purple bounding box calculated for each connected component island, and the cyan subject bounding box and cyan centroid center.

Stage 3: Caculate Zoom and Pan & Interpolate using PID loops.

The last algorithmic step is to then use the detected subject bounding box and centroid center to control both the pan and zoom of the square crop camera. An active control loop is used with a PID each to control and smooth the zoom and pan. A PID class was implemented for this in pid_controller.go and it is initialised with some experimentally determined PID values which resulted in smooth camera motion:

step4_calcVideo.go L19-L20
19    pid = NewPIDController(0.1, 0.001*10e-10, 0.002*10e10, 0)
20    pidz = NewPIDController(0.2, 0.003*10e-10, 0.002*10e10, 0)

The PID target positions are set so that the centroid of the subjects (or average centroids for multiple subjects) is kept near the center of the square camera view:

step4_calcVideo.go L103-L124
103func calculatePanAdjustment(centroid image.Point, frameWidth int) int {
104    // Calculate how much you need to pan to keep the centroid in the center
105	panAdj := centroid.X - frameWidth/2
106
107	// smoothedPanAdj := ma.Add(panAdj)
108    pid.SetSetpoint(float64(panAdj))
109    smoothedPanAdj := pid.Update()
110    pid.SetDelta(pid.delta+smoothedPanAdj)
111
112    return int(pid.delta)
113}
114
115func calculateCropRect(centroid image.Point, frameWidth, cropWidth, panAdj int) image.Rectangle {
116	frameCtr := frameWidth/2
117	min := (cropWidth/2)-frameCtr
118	max := -min
119	panAdj = clamp(panAdj, min, max)
120
121	cropRect := image.Rect((frameCtr+panAdj)-cropWidth/2, 0, (frameCtr+panAdj)+cropWidth/2, cropWidth)
122
123	return cropRect
124}

Some additional safe-guards and custom logic is added to handle the zoom level, as well as some boundary conditions to avoid jerky motion if the bounding boxes computed from the previous step are invalid:

step4_calcVideo.go L30-L65
30            // Process the received image
31            bbox := mask.Bbox
32            centroid := calculateCentroid(mask.Mask, bbox)
33            
34			panAdj := calculatePanAdjustment(centroid,mask.Image.Bounds().Dx())
35			bboxArea := bbox.Dx()*bbox.Dy()
36            frameArea := mask.Image.Bounds().Dx()*mask.Image.Bounds().Dy()
37            perCovered := float64(float64(bboxArea)/float64(frameArea))*100
38            bboxC := color.RGBA{0,0,255,255}
39            if perCovered < 0.1 {
40				// bounding box too small (< .1%), do not pan
41				panAdj = 0
42                bboxC = color.RGBA{255,0,255,255}
43			} else if perCovered > 75 {
44                // bounding box too big (> 80%), do not pan
45                panAdj = 0
46                bboxC = color.RGBA{255,0,255,255}
47            }
48            // fmt.Println(bboxArea, float64(float64(bboxArea)/float64(frameArea)))
49			cropRect := calculateCropRect(centroid,mask.Image.Bounds().Dx(),mask.Image.Bounds().Dy(),panAdj)
50
51            //? Experimental zoom out if width of bbox > cropRect
52            zoom := false
53            padding := int(cropRect.Dx()/8) // padding width of 1/10th of the crop rectangle width
54            perOut := float64(float64(bbox.Dx()) / float64(cropRect.Dx())) * 100
55            difOut := minBoxHDist(bbox, cropRect) //bbox.Dx()-cropRect.Dx()
56            difOut = clamp(difOut+padding, 0, cropRect.Dx()*5)
57            zoompx := 0
58            zoompx = difOut
59            pidz.SetSetpoint(float64(zoompx))
60            smoothedZoomAdj := pidz.Update()
61            pidz.SetDelta(pidz.delta+smoothedZoomAdj)
62            
63            // Zoom
64            cropRect = ExpandRectangle(cropRect, clamp(int(pidz.delta), 0, cropRect.Dx()*5))
65            zoom = true

The final result is shown in the video below:

Where the blue rectangle shows the aggregated subjects bounding box, the red line and dot shows the subject centroid target, and the green bounding box shows the square camera view position (yellow showing the same but with zoom applied) returned from the PID controllers.

Notice here how smooth the movement of the green camera is, and how it accurately tracks Miku even when she walks to the left and right sides of the original 16:9 frame.

Since an absolute centroid center is calculated across all tracked subjects, this whole algorithm also works to track videos where multiple character’s are present. Below is the result from the song DiosP - Cendrillon, with the hologram-ready video from MMD Music:

Here the zoom part of the algorithm needs to do a lot more work in order to keep both Miku and KAITO in frame at the same time. Although it looks a little stuttery in this view, the final output after the last crop step is quite smooth.

Stage 4: Video Crop

The last stage is to actually apply the video zoom and crop calculated by the previous step and generate an output video. This is very straightforward:

step5_cropVideo.go L21-L27
21            // Process the received image
22			croppedImg := cropToBBox(mask.Image, mask.CBox)
23            //? Experimental zoom out and aspect ratio when bbox > cropRect
24            if mask.zoom {
25                // croppedImg = ResizeWithVerticalLetterbox(croppedImg, squareSize, squareSize)
26                croppedImg = ResizeRGBA(croppedImg, squareSize, squareSize)
27            }

At long last, we obtain our smoothly cropped square videos as the final output:

And the same goes for our multi-character videos, taking advantage of the zoom controller:

The PID controllers could probably do with some finer tuning, especially on the zoom side. But as most videos only feature one character anyway, I deemed this to be good enough.

Now we have a full content-aware square video cropping pipeline and tool which outputs videos ready for playback on the HoloCube’s square display. The next part of this series will cover more details about that.

Pipeline Setup Details

Lastly, before we finish off. I want to touch on some details of the pipeline implementation in Go lang, and just how easy it made to do such kind of concurrent processing.

We start setting up the pipeline by declaring all of the input/output image chennels between each step, as well as mutexes to ensure the read/write processes stay in sync between the different pipeline steps:

pipeline.go L14-L41
14var (
15	inputFrame 		*image.RGBA
16	inputFrameRW 	sync.Mutex
17
18	procFrameQueue1	chan *image.RGBA
19
20	grayFrame		*image.RGBA
21	grayFrameRW		sync.Mutex
22
23	procFrameQueue2	chan *ImageGrayMask
24
25	binaryFrame		*image.RGBA
26	binaryFrameRW	sync.Mutex
27
28	procFrameQueue3	chan *ImageGrayBoundMask
29
30	calcFrame		*image.RGBA
31	calcFrameRW		sync.Mutex
32
33	procFrameQueue4	chan *ImageCropMask
34
35	croppedFrame	*image.RGBA
36	croppedFrameRW	sync.Mutex
37
38
39	tempDirTinyVid  string
40    tempDirFrames   string
41)

Then, all we have to do to execute the pipeline is open the video and run all of the pipeline steps in concurrent go routines:

pipeline.go L86-L100
 86func startPipeline() {
 87	video, _ := vidio.NewVideo(filepath.Join(tempDirTinyVid,getBasenameWithoutExt(inputVideo)+".mp4"))
 88    tframes = video.Frames()
 89    cframes = 0
 90    if !playOnlyMode {
 91		bar = progressbar.Default(int64(tframes))
 92	}
 93	video.Close()
 94
 95	go readVideo()
 96	go grayscaleVideo()
 97	go binaryVideo()
 98	go calcVideo()
 99	go cropVideo()
100}

Everything is handled automatically. The receive and send loops inside of each pipeline step look like this:

step2_grayscaleVideo.go L10-L29
10func grayscaleVideo() {
11	for {
12        select {
13        case img, ok := <-procFrameQueue1:
14            if !ok {
15                // Channel is closed
16                return
17            }
18
19            // Process the received image
20			maskFrame := convertToMonochrome(img)
21            if drawUI && !playOnlyMode {
22                grayFrameRW.Lock()
23                grayFrame = convertGrayToRGBA(maskFrame)
24                grayFrameRW.Unlock()
25            }
26			procFrameQueue2 <- &ImageGrayMask{Mask:maskFrame,Image:img}
27        }
28    }
29}

Where a select statement in a for loop is used to capture the incoming input frames in an event-driven manner, the mutex is locked/unlocked during the processing phase for UI display, and the output frame is sent to the queue channel of the next pipeline step.

Using such Go primitives, the pipeline processing became easy to implement rather than an asynchronous nightmare. As an added bonus, I included an Ebitten Engine based backend which can render an UI in real-time showing all of the processing steps as they happen:

All of the options for running the tool are documented and configurable via command line arguments:

$ ./1-1-Content-Aware-Video-Crop -h
Usage of ./1-1-Content-Aware-Video-Crop:
  -i string
        Input video or directory. Required.
  -o string
        Output directory (default current directory)
  -play
        Player only mode, do not write output. Just displays a preview of the result. (default false)
  -rt
        Whether to run in real-time (default false)
  -s int
        Size of square output video (default 256)
  -ui
        Whether to draw UI (default false)

More info can be found on my GitHub repo:

Next Steps

So there we have it, the first part of this three part saga to play MMD holographic concerts on the HoloCube. The next part will cover a custom video encoding format I developed in order to stream video and audio to the ESP32C3 MCU in a format that is optimally compressed while still being able to be decoded live with the limited processing power of this MCU.

I hope this was an interesting read and a little dive in the the world of classical video DSP. Any suggestions or improvements are more then welcome, just send me a PR on GitHub.

Categories
Tags