Week 7: Acquiring and Manipulating Videos#

Laboratory 4
Last updated July 25, 2023

00. Content #

Mathematics

  • N/A

Programming Skills

  • Using Software Documentation

  • OpenCV (cv2)

Embedded Systems

  • Thonny and MicroPython

0. Required Hardware #

  • Raspberry Pi Pico

  • Breadboard

  • USB connector

  • Camera (Arducam HM01B0)

  • 8 Wires

Write your name and email below:

Name:

Email:

import numpy as np
import matplotlib.pyplot as plt
import cv2

1. Reading Videos with OpenCV #

In this lab, we will be using a Python library called OpenCV (cv2). According to the official documentation, “OpenCV is an open-source library that includes several hundreds of computer vision algorithms.” Today, we will primarily utilize it to read and display modified video frames, and we will manually handle the “vision” aspect of computer vision with our own functions.

Run the following code cell to read the video file. The vid.read() function returns two values: a Boolean variable success, which is True if the frame was read without any errors, and frame, which represents the captured image from the video. Since the camera hardware connected to your Pico can only provide grayscale images, we will convert each frame to grayscale using the cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) function. To improve computation speed, we have scaled down each frame by a factor of 4.

Once you run the cell, a separate window will pop up (potentially appearing behind other windows on your screen), displaying the smaller, grayscale video. The separate window will automatically close after reading the entire video. However, you can override this behavior by pressing the Q key on your keyboard.

To begin, download the video file we’ll be using for this lab: test_vid.mov. This video is sourced from YouTube and depicts a bouncing ball.

vid = cv2.VideoCapture('test_vid.mov')                  
height = vid.get(cv2.CAP_PROP_FRAME_HEIGHT)
width  = vid.get(cv2.CAP_PROP_FRAME_WIDTH) 
scale = 0.25
new_size = (int(width*scale),int(height*scale))         # new frame dimensions, 4 times smaller than the original

while vid.isOpened():                                    
    success, frame = vid.read()                         # get the current frame (if there is one)
    if not success:
        print("Unable to read frame. Exiting ...")
        break
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)     # change the video from color to grayscale
    frame = cv2.resize(frame,dsize=new_size)            # resize the frame to the new dimensions
    cv2.imshow('frame', frame)                          # display 'frame' in the popup window
    if cv2.waitKey(25) == ord('q'):                     # press Q on keyboard to stop
        break
vid.release()
cv2.destroyAllWindows()                                 # closes the popup when the video is over

Exercise 1 #

What type of variable is frame? What kind of datatype is stored in frame?

Write Answers for Exercise 1 Below

2. Modifying Videos #

Exercise 2 #

Read the documentation for the waitKey function.

Part 1

Play the video at a slower speed than its original playback speed.

Write Answers for Exercise 2 Part 1 Below

Part 2

Play the video at a faster speed than its original playback speed.

Write Answers for Exercise 2 Part 2 Below

Exercise 3 #

Play the video in reverse (backward). You don’t need to refer to any additional documentation for this exercise.

Write Answers for Exercise 3 Below

Exercise 4 #

Part 1

Flip the visuals of the video over the y-axis.

Hint: There are multiple ways to accomplish this, but you can explore the flip functions in NumPy or OpenCV.

Write Answers for Exercise 4 Part 1 Below

Part 2

Flip the visuals of the video over the x-axis.

Write Answers for Exercise 4 Part 2 Below

Being able to extract specific frames from a video is a useful skill. It can be used, for example, to trim the ends of the video or to remove a section in the middle.

Exercise 5 #

Play only the first 200 frames of the video.

Write Answers for Exercise 5 Below

3. Connecting the Camera #

This time, we will record our own videos using the Arducam HM01B0, which is a small camera that can be connected to the Pico.

Wiring Instructions#

Please ensure that your microcontroller is not connected to the computer while you are wiring components together. If you are unsure about your wiring, please consult the instructor. Use your jumper wires to establish the following connections:

HM01B0

Pico

VCC

3V3

SCL

GP5

SDA

GP4

VSYNC

GP16

HREF

GP15

PCLK

GP14

DO

GP6

GND

GND

Here is an image of the completed breadboard:

img

To find the names of the pins on the Raspberry Pi Pico, you can refer to its pinout diagram located here or in the Extra Materials section. The HM01B0, on the other hand, should have its pins labeled.

After confirming that the wiring is correct, press and hold the BOOTSEL button on the Pico while plugging it in. Download the arducam.uf2 file and copy it onto the Pico’s drive using your computer’s file manager (it should be listed as an external drive: “RPI-RP2”) and not with Thonny. Once the file transfer is complete, the Pico will automatically disconnect, and its LED will start blinking rapidly.

Once the Pico has been successfully connected, please execute the following cell to ensure that we have successfully detected the Pico.

import time
import serial
from serial.tools import list_ports

PICO_HWID = "2E8A"


def get_pico_port():
    pico_ports = list(list_ports.grep(PICO_HWID))
    if len(pico_ports) == 0:
        raise Exception(
            "No Raspberry Pi Pico was detected. Check to make sure it is plugged in, and that no other programs are accessing it"
        )
    return pico_ports[0].device


print("Here are all the serial devices detected:")
for port in list_ports.comports():
    print(port.device, port.hwid)

port = get_pico_port()
print(f"\nselected port {port} as most likely to have a raspberry pi pico")

Capturing a still image#

Now that the Pico and camera have been connected, execute the following cell to capture a still image.

buffer = bytearray(96 * 96)
img = np.zeros(shape=(96, 96), dtype="uint8")

with serial.Serial(port, timeout=1) as s:
    s.read_until(b"\x55\xAA")
    s.readinto(buffer)
    img.flat[::-1] = buffer

plt.imshow(img, cmap="gray")
plt.show()

Exercise 6 #

Use the camera to capture an image of yourself (or a neighbor) with your hand up and another image of yourself (or your neighbor) with your hands down. You are welcome to use more than two frames if desired. Use these frames to create a video where you move your hands up and down, lasting a few seconds.

Write Answers for Exercise 6 Below

Streaming video#

Now that we can capture a still image, the next goal is to stream a video. Both cells need to be executed before the video is streamed to your screen. After running the first cell, a still image should pop up. Once you run the second cell, it will start streaming real-time video from your camera.

%matplotlib widget

fig, ax = plt.subplots()
render = ax.imshow(img, cmap='gray')
plt.show(block=False)

try:
    with serial.Serial(port, timeout=1) as s:
        while True:
            s.read_until(b"\x55\xAA")
            s.readinto(buffer)
            img.flat[::-1] = buffer
            render.set_data(img)
            fig.canvas.draw()
except KeyboardInterrupt:
    pass

When you are finished, you will need to click the “Interrupt Kernel” button in Jupyter, which can be found at the top of the screen (the stop symbol) or under the Kernel menu.

4. Modifying Your Own Video #

Exercise 7 #

Capture and save a list of frames using the camera.

Part 1: Repeat Exercise 3 and Exercise 4 with your own video.

Write Answers for Exercise 7 Part 1 Below

Part 2: Create an Instagram boomerang effect using a small portion or the entire video. Here’s an example of what a boomerang effect looks like:

Write Answers for Exercise 7 Part 2 Below

Part 3: Choose an editing function we haven’t mentioned yet (e.g., adding text, video filters, etc.) and give it a try on either the pre-recorded video or the ones you recorded using the camera.

Write Answers for Exercise 7 Part 3 Below

Reflection #

  1. What was your favorite exercise in this lab?

  2. Which part of the lab did you find the most challenging?

  3. Which part of the lab was the easiest?

Write Answers for the Reflection Below