Skip to contents

6 - Capture queues

Source: vignettes/z6_queue.Rmd
z6_queue.Rmd

6.1 - Increasing performances with capture queues

Retrieving a single frame from a video or camera stream is fast, but it is not instantaneous because Rvision (and therefore OpenCV in the background) needs to grab and decode each frame before it can be used for further processing. For most applications, that small time penalty is not an issue; losing a few milliseconds per frame will not be felt by a user processing short and/or low resolution videos for instance.

However, the frame decoding time will increase significantly with the resolution of the video/camera stream; and for applications requiring the processing of long videos or live camera feeds, the total time lost retrieving frames will quickly increase. Moreover, the process of retrieving frames is blocking: while Rvision grabs and decodes a frame, it cannot work on other frames that were previously captured.

A solution to this issue is to make use of the multi-tasking ability of most modern computers to perform the frame retrieving process in parallel with the rest of the image processing. The principle is fairly simple: one processing thread (thread 1) is in charge of grabbing, decoding, and storing frames in a shared dynamic queue (or buffer); a second thread (thread 2) is in charge of processing these pre-loaded frames further; while thread 2 is working on a frame, thread 1 can keep filling up the queue with new frames so that thread 2 never has to stop and wait for a new frame to be retrieved. On a multi-core/processor computer (most computers nowadays), thread 1 and 2 can be operated in parallel on separate cores/processors, effectively reducing the frame waiting time to near-zero for thread 2 (but see caveats section below).

Unfortunately, R is a single-threaded language, meaning that it cannot natively create and operate multiple parallel processing threads. Fortunately, the C++ language that is used in the background by Rvision can do it. We used that ability - and some precautions to avoid memory access conflicts with R - to create the Queue class of objects which job is to pre-load in memory video and camera stream frames without blocking the execution of the main (and unique) R processing thread, and then give the main R thread near-instantaneous access to these pre-loaded frames when required. Once a frame has been collected by the main R thread, it is removed from the queue to make space for another frame.

6.2 - Creating a queue

Queue objects are created using the queue() function. For instance:

# Find the path to the Balloon.mp4 video provided with Rvision
path_to_video <- system.file("sample_vid", "Balloon.mp4", package = "Rvision")

# Open the video file stream
my_video <- video(filename = path_to_video)

# Create a queue of frames 
my_buf <- queue(my_video, size = 10, delay = 1000, overflow = "pause")

The queue() function can take 4 parameters:

  • x corresponds to the source of the frames to be queued. It should either be a Video or a Stream object.
  • size corresponds to the number of frames that the queue can store at any one time. By default, a Queue object will be able to hold 10 frames at once. If you increase the value of the size parameter, the Queue object will use up more RAM as a result, so be mindful of your computer’s resources.
  • delay corresponds to the time (in microseconds) between two queue update cycles. During an update cycle, the Queue object checks whether it is full or not and whether frames have been collected by the main R thread. If it is not full, it retrieves a new frame from the source and stores it; if frames have been collected by the main R thread, it removes them from the queue; it then waits the duration set by delay before starting a new update cycle. Reducing the value of delay will increase the frequency of the update cycles but will also increase the computational load of the core/processor running the queuing thread.
  • overflow corresponds to the behavior the Queue object should adopt once it is full. By default, the queuing process will “pause” (that is stop retrieving new frames from the source and storing them) until a frame is collected by the main R thread. This is the behavior of choice for video processing as it does not skip any frame while ensuring that the queue storage memory does not grow more than what the user wants. The queuing process can also “replace” the oldest frame in the queue with a new one. This is usually a good choice for processing live camera stream as it ensures that the queue storage memory does not grow more than what the user wants. However, frames may be skipped if the main R thread cannot collect and process the frames faster than they can be retrieved and stored by the queuing thread. Finally, the queuing process can “grow” the queue by doubling its size each time it fills up. This behavior allows for not pausing the retrieval process and avoids frame skipping but it is very much NOT recommended unless you know what you are doing. Indeed, this can lead to excessive RAM usage and decreased computing performance across the board.

Once a Queue object is created, it starts immediately filling up with images retrieved from the Video or a Stream source object. Note that, if you had previously read frames from a Video source object before creating the Queue object, the latter will start retrieving frames from where you left off (e.g. if you have already read the first 10 frames, the queue will start filling up from the 11th).

Once a Queue object is not required anymore, it can be released from memory as follows:

release(my_buf)

6.3 - Using a Queue object

The main purpose of a Queue object is to pre-load and store frames for fast access later on. The pre-loading and storing happens by itself in the background so you do not need to take care of this. Collecting a frame from the queue into the main R thread can be done using the readNext() function. For example:

# Collect the next available frame from the queue and store it in a new 
# Image object
frame <- readNext(my_buf)

# Collect the next available frame from the queue and store it in an existing 
# Image object
readNext(my_buf, target = frame)

At any time, you can check the state of the queue as follows:

# Is the queue empty?
empty(my_buf)

# Is the queue full?
full(my_buf)

# What is the current number of frames in the queue? 
length(my_buf)

# What is the maximum number of frames that the queue can hold?
capacity(my_buf)

# What is the index of the next frame available? (for video queues only)
frame(my_buf)

# What are the dimensions of the queue? 
dim(my_buf)
nrow(my_buf)
ncol(my_buf)

6.4 - Caveats to using Queue objects

Queue objects can be very useful to speed up the processing of camera streams and long videos, especially if their resolution is high (HD or above). For short videos or low resolution videos and camera streams, the benefits of using a Queue object will be very limited because retrieving frames from such sources is already very fast.

If the processing of the frames on the main R thread is much faster than the retrieving and storing of the frames inside a Queue object, you risk emptying the queue faster than in can fill up. If that is the case, readNext() will display a warning to this effect but will not return an error (as it would, for instance, when reaching the end of a video file). In addition, it will not create a new image or modify the target image. You will need to make sure that your code can catch these warnings and act accordingly (e.g., wait a turn before trying again). You can reduce the risks of the queue becoming empty by (1) increasing the size of the queue when creating it, (2) making sure the queue has completely filled up before starting the processing of the frames, and (3) reducing the delay between to queue updates. In any case, Queue objects are better suited for and more beneficial in cases where the processing of the frames is as slow or slower than the frame retrieving process.

Finally, Queue objects read frames from existing Video and Stream objects. This means that they modify the state of these source objects the same way a user would by reading directly from them. This has a few of consequences that require some attention when using Queue object. First, a Queue object will start reading the source object from whichever state you have left it off. For instance, if you have read the first 10 frames of a video before passing it to a Queue object, the Queue object will start reading it from the 11th frame. Conversely, if you pass a newly created Video object to a Queue object with a size of 10, the Queue object will immediately start reading and storing the first 10 frames. If you then decide to read a frame directly from the Video object, it will return its 11th frame, not its first one. Therefore, to avoid competing with a Queue object on a given Video object, it is strongly recommended to avoid creating code that mix reading frames from a Queue object and reading frames from the Video object that Queue object is using.