Cost-effective solution to synchronised audio-visual data capture using multiple sensors

TLDR: This work centralises the synchronisation task by recording all trigger- or timestamp signals with a multi-channel audio interface, and shows that a consumer PC can currently capture 8-bit video data with 1024x1024 spatial- and 59.1Hz temporal resolution, from at least 14 cameras, together with 8 channels of 24-bit audio at 96kHz.

About: This article is published in Image and Vision Computing.The article was published on 2011-09-01 and is currently open access. It has received 32 citations till now. The article focuses on the topics: Timestamp & Sensor fusion.

Figures

Figure 7: 5 tracks recorded in parallel by MOTU 8pre audio interface. From top to bottom: (1) room microphone; (2) head microphone; (3) serial port timestamp output (transmitted at 9600bps), showing 2 timestamp signals; (4) measured infrared light in front of eye tracker; (5) camera trigger.

Table 5: Statistics of estimated root mean square errors (RMSE) in 87 recordings of approximately 5 minutes long, measured in number of Audio Samples (AS) at 48kHz or in µs. From left to right, this table shows the average, standard deviation, minimum and maximum of the RMSE estimated in 87 recordings.

Table 2: Overview of multi-sensor audio-visual data capture solutions. A ‘unit’ is a system in which sensor data is collected in real-time. For most cases, this is a PC. However, for Zitnick et al. [6] it was a ‘concentrator unit’. ‘camera#/unit’ indicates the maximum number of cameras that can be connected to a unit, ‘audio ch#/unit’ indicates the maximum number of audio channels per unit, ‘sync unit#’ shows the maximum number of units that can be synchronised, ‘unit sync’ the type or accuracy (if known) of synchronisation between units, ‘camera sync’ the the type or accuracy of synchronisation between cameras and ‘A/V sync’ the accuracy of synchronisation between audio and video.

Figure 8: A binary (on/off) timestamp signal output from a serial port, recorded as an audio channel. A high-pass filter used in the audio processing causes vertical skew along the timestamp, while an anti-aliasing filter causes ripple around the step edges. The skew is compensated before reading the timestamp message, by a linear interpolation of the steady state level before and after the timestamp.

Figure 6: Sequential disk write performance of the 6 SATA ports on the motherboard (ICH9R controller) for different configurations. From left to right, the WTR of ports 1 to 6 are shown, shaded from black to white. ‘1 dsk’: writing to 1 HDD at a time. ‘1 pair’: writing to 2 disks simultaneously, through port 1&2, 3&4 or 5&6, respectively. ‘3 dsk’: writing to 3 disks simultaneously, through port 1&3&5 or 2&4&6, respectively. ‘6 dsk’: writing through all 6 ports at the same time. ‘6&Cam.’: same as ‘6 dsk’ but simultaneously streaming image data to memory, from 18 cameras of 780x580 pixels at 60fps.

Figure 12: Estimated gaze data timestamp error over time, in comparison to the interval centre of the closest IR flash, measured in the audio recording.

Frequently asked questions

Q1. What have the authors contributed in "Cost-effective solution to synchronised audio-visual data capture using multiple sensors" ?

Furthermore, the authors show that a consumer PC can currently capture 8-bit video data with 1024x1024 spatialand 59. The authors thus improve the quality/cost ratio of multi-sensor systems data capture systems. 

Q2. What is the way to improve image sharpness with low cost lenses?

If a monochrome camera is used, a monochrome colour source can improve image sharpness with lowcost lenses, by preventing chromatic abberation. 

Q3. What is the importance of the microphone setup for accurate multimodal data capture?

Since many audio processing methods are vulnerable to noise, the microphone setup is an important factor for accurate multimodal data capture. 

Q4. What type of sensor can be synchronised with the audio data?

Any type of sensor can be synchronised with the audio data, as long as it produces a measurable signal at the data capture moment, and its output data include reliable sample counts or timestamps relative to the first sample. 

Q5. What is the primary choice for computer vision applications involving moving objects?

For computer vision applications involving moving objects, such as human beings or parts of the human body, progressive scan global shutter sensors are the primary choice. 

Q6. What is the way to record the timestamp signals?

The timestamp signals from multiple PCs can be recorded as separate channels in a multi-channel audio interface, making use of the hardware-synchronisation between the different audio channels. 

Q7. How long does the delay of the slave cameras last?

When the trigger output of themaster camera is used as the input to the slave cameras, the resulting delay of the slave cameras is approximately 30µs. 

Q8. Why do some sensor locations have a higher or lower read-out value than the correct measurements?

Due to irregularities in sensor production, or the influence of radiation, some sensor locations have a defect that causes their pixel read-out values to be significantly higher (hot) or lower (cold) than the correct measurements. 

Q9. How can a PC synchronise its CPU cycle count?

Capture software running on different PCs can be synchronised by letting each PC transmit its CPU cycle count as timestamp signals outputted by the serial port. 

Q10. How did the authors configure the port to transmit at 9600 bits per second?

In their recordings, the authors used the MOTU8pre at 48kHz sampling rate and the authors configured the serial port to transmit at 9600 bits per second (bps). 

Q11. How did the authors disable displaying the live video?

to prevent the communication to the PCI graphics card from reducing the storage WTR, the authors had to disable displaying the live video. 

Q12. Why did Zitnick and others seek custom solutions?

Because of the shortcomings and high costs of commercially available video capture systems, many researchers have already sought custom solutions that meet their own requirements. 

Q13. What is the problem of the high cost of custom solutions and professional hardware?

The problem of the high cost of custom solutions and specialised professional hardware is that it keeps accurately synchronised multi-sensor data capture out of reach for most computer vision and pattern recognition researchers. 

Q14. How many triggers were used to synchronise the cameras?

They used a tree of trigger connections between the processing boards (that each control one camera) to synchronise the cameras with a difference of 200 nanoseconds between subsequent levels of the tree. 

Q15. How can the authors estimate the accuracy of the synchronisation of the gaze data?

Using a photo diode that is sensitive to IR, the authors could record these flashes as a sensor trigger signal in one of the audio channels and estimate the accuracy of synchronisation of the gaze data. 

Q16. What are the main problems of using low-cost commercial COTS components?

To overcome this, the authors propose solutions and present findings regarding the two most important difficulties in using low-cost Commercial Off-The-Shelf (COTS) components: reaching the required bandwidth for data capture and achieving accurate multi-sensor synchronisation. 

Q17. How can the authors find a linear mapping between audio sample number and the time of the external system?

the authors could find a linear mapping between audio sample number and the time of the external system, by applying a linear fit on all two-dimensional time synchronisation points (timestamps with corresponding audio time) that are received during a recording. 

Q18. What can be used as a common time base?

These signals can be recorded in a parallel audio channel as well, and can even be used as a common time base to synchronise multiple asynchronous audio interfaces. 

Q19. How many cameras can be connected to one FireWire bus?

The maximum number of cameras that can be connected to one FireWire bus is typically limited to 4 or 8 (DMA channels), depending on the bus hardware. 

Q20. What is the difference between the transmission and reception time?

Assuming that the process of transmission and reception are symmetric, the transmission latency can be found as half of the time needed for transmitting and receiving the timestamp signal, compensated by the duration of the signal. 

Q21. How accurate is the synchronisation of data?

The above-discussed experiments show that synchronisation by transmitting timestamp signals through the serial port, can be done with an accuracy of approximately 20µs. 

Q22. What is the uncertainty of localising the camera trigger edge?

This means that, with an audio sampling rate of 48kHz, the uncertainty of localising the rising camera trigger edge is around 20µs.