Multi-camera system design for automotive parking assistant systems
Driver Assistance Systems using imaging elements (e. g. parking assistants, night vision systems etc.) are a rapidly growing segment in the market for automotive safety and assistance systems. In order to fulfil the highest prerequisites in terms of quality and video display capabilities, which means the absence of artefacts, a seamless cooperation of all system components is mandatory. The following paper describes a parking assistance system using four camera sensors. This system implements CMOS image sensors supplied by OmniVision Technologies which are connected to an FPGA baseboard delivered by Xilinx through the serialiser/deserialiser (SerDes) chipset DS90UB901QSQ-DS90UB902QSQ from National Semiconductor (NSC). The cameras may be connected for a 360° surround view or used separately – with or without image stitching algorithms which are supplied by IP partner Xylon (brand name: LogicBRICKS). The interface chipset from NSC provides a bidirectional control channel with a minimum latency and general purpose I/O (GPIO) functionalities enabling a seamless frame-synchronous processing of the individual video data streams. This makes it possible to provide the highest image quality when combining the individual images into one single birds-eye view picture.
According to studies conducted by IMS Research the number of automotive camera modules will grow from 6.1 million units in 2010 to more than 34 million units in 2017, whereas parking assistant systems are said to cover the majority of these cameras’ application market. As the resolution of CMOS image sensors within camera imagers permanently increases the video interfaces will have to cope with permanently rising data rates of several Gbit/s. For this application SerDes chipsets (serialiser/deserialiser) using differential signalling provide an outstanding interface solution for thin flexible cable connections without the need for additional protocol overhead of a controller/video processor.
FPD-Link Product Families
The FPD-Link product family is designed for serial interfaces of embedded displays and camera sensor systems (figure 1). The first generation of the FPD-Link product family was designed in the mid-1990s targeted towards a simplification of the video interface between the motherboard in notebook computers and embedded notebook displays. By serialising the video and synchronisation data into 3 to 4 differential pairs and a clock channel running in parallel the design engineers were able to design the hinge much slimmer and they were finally able to achieve a much more EMC-friendly design.
The FPD-Link II family introduced to the market in 2005/2006 was from scratch targeted towards automotive applications. The serialization with embedded pixel clock over just one pair of wires enabled the use of very thin and flexible cables. Furthermore, the ESD resistivity was significantly increased in order to comply with the requirements of the standards ISO 10605 and IEC 61000-4-2. In this application the control data are still transmitted in parallel – e.g. through CAN or LIN bus systems.
Finally, the FPD-Link III product family introduced in 2010 integrates a bidirectional control channel, which is terminated in an I2C controller interface enabling the exchange of control data in addition to the video and synchronisation signals in both directions between the transmit and the receive modules.
Figure 1: Basic Overview about the FPD-Link Product Familie. For full resolution click here.
FPD-Link III Chip Set for Driver Assistance Applications
The video interfaces of the FPD-Link III product generation significantly simplify the architecture of driver assistance systems. In this context the innovative implementation of a full-duplex control channel is of noteworthy importance. In addition to the Forward Channel drivers and receivers the I/Os contain specific Back Channels within the relevant counterparts. In forward direction the control data are embedded into the video data stream. The back channel exchanges its data simultaneously at the same time as the video data over the same set of differential transmission lines. This enables a continuous data exchange in real-time with the lowest latencies – independent of timing considerations like the length of blanking intervals (figure 2).
Parking assistant systems with several distributed camera heads allow for a very precise synchronisation excluding the creation of possible artefacts during the computation of the entire image. In order to enhance the EMC compliance the receiver is equipped with specific reduction methods like spread-spectrum clocking. Doing so, the additional functionalities allow for an implementation with further decreased RF emissions. There is no need for an additional physical transmission channel, and this function makes additional control busses ((s. o.)) like CAN or LIN obsolete for this purpose.
Figure 2: Simultaneous Data Exchange by using the FPD-Link III Family
This allows for a reduced number of connector and cable connections, decreasing in turn the system costs as well as the weight of the wiring harness, which will have a positive ecological effect.
Figure 3 shows a system diagram of the camera-specific FPD-Link III chipset based on DS90UB901QSQ-DS90UB902QSQ. This chipset supports up to 14 bits on top of the timing reference signals HS (Horizontal Sync, or respectively, Line Valid) and VS (Vertical Sync, or respectively, Frame Valid). At the end of the data payload the devices attach a 4-bit CRC (Cyclic Redundancy Check) checksum which allows for supervision in terms of data integrity for safety-critical applications.
Furthermore, up to 6 GPIOs (General-Purpose I/Os) may be programmed – e. g. in order to synchronize several distributed CMOS imager heads in applications like parking assistant systems. These GPIO data are transmitted with the minimum latencies and deviations, which reduces artefacts during the computation of the entire image.
In some applications it is mandatory to address several cameras with the same fixed address on the same I2C bus. For this purpose the chipset offers its “Slave Address Remapping” capability, where every connected element is automatically assigned a unique address by using an identifier. This feature is quite practical whenever camera modules are removed or, respectively, replaced. The system supports up to 8 ID indexes. The chipset synchronizes without a default external reference clock just by using the embedded clock information. This reduces the components costs and potential sources of EMC disturbance. Finally, the serialiser is integrated into a space-saving 32-LLP package with a footprint of 5 mm x 5 mm in order to enable the design of more compact cameras with reduced volumes.
Figure 3: System Diagram of the DS90UB901-DS90UB902 FPD-LINK III Chipset
Reference Design with four Cameras for Parking Assistants
In cooperation with OmiVision Technologies, Xilinx and Xylon a four-camera reference design was created as a multi-camera system for a driver assistance/parking assistance system (figure 4). Four OV10620 CMOS image sensors from OmniVision are connected to serialiser boards by using the DS90UB901QSQ transmitter and standard RJ-45 plugs. The signal is transmitted through a single wire pair from an unshielded CAT6 network cable, which is 10m long. Four DS90UB902QSQ receivers are consolidated on an FMC (FPGA Mezzanine Card). This FMC board is a daughter card stacked on top of a VSK (Video Starter Kit) Spartan FPGA Base-Board from Xilinx. The FPGA contains IP from Xylon (Branding: logicBRICKS in terms of image-stitching and surround-view algorithms. The entire configuration including the dynamic reconfiguration of the OmniVision image sensor is performed solely by the FPGA control unit via the I2C interface of the SerDes chipset. The cameras may be operated individually as well as in combination. Switch-over times between the individual modes are very short because they are in the 10-microseconds range. The bidirectional control channel operating in parallel to the video data enables minimum latencies and finally the synchronisation of the camera video data on a frame-by-frame basis. Consequently, the combined total image avoids any kinds of artefacts due to uncalibrated image delays.
Figure 4: System Diagram of a Reference Design with the FPD-Link III Chipset and four Cameras
Overview: The Basic Properties of the Chipset
The operating frequency range for the pixel clock (PCLK) is between 10 MHz and 43 MHz. On both the send as well as the receive side there is no crystal (quartz) or oscillator needed in order to provide a reference clock for the presynchronisation of the PLL (Phase-Locked Loop). The PLL automatically autonomously locks to any kind of data pattern received as long as start/stop clock bits are embedded into the signal. Furthermore, the PLL provides phase-synchronous impulses to the demultiplexer. As soon as the deserialiser has synchronized to the serialiser data stream it shows the completion of this action at the external LOCK pin. From now on the output data e.g. to the LCD timing controller are provided with the correct polarity. In order to transmit data over long cable distances of up to 15m the receiver stage contains an equalizer, which compensates the frequency-depending attenuation of the transmission media and cleans the signal from deterministic phase jitters. Input and output data busses may be flexibly aligned with either the rising or the falling edge of the pixel clock signal.
The relevant I/O banks may either be powered by a 1.8V or a 3.3V power supply, which means that there is additional flexibility when components on the graphical source and on the graphical drain are connected together. In order to reduce the electromagnetic radiation the deserialiser is equipped with an integrated Spread-Spectrum Clock Generator (SSCG) output, with an adjustable output driver and with staggered outputs which are time-delayed relative to the regained clock signal.
In addition the PLLs of the chipsets are designed in a flexible way which means that they are tolerant towards external spread-spectrum clocking on the transmitter module. This feature is needed in order to use the maximum advantage of the frequency-spreading over the entire data path from the graphics host via the parallel input interface up to the serial and parallel output interface.
The ESD robustness is qualified according to the automotive-relevant standard ISO 10605 and according to the IEC 61000-4-2 standard. The very broad temperature range from -40°C up to +105°C opens up use cases in a multitude of applications, e. g. if the imager camera head needs to be positioned in exposed areas susceptible to intensive heat irradiation like in front of the radiator grille of a passenger car. The components are integrated in space-saving compact LLP packages. This industry-standard package in “no pullback” implementation is excellently suited for processing/handling and inspection and is available in a lead-free version. Of course, all components are qualified according to the standard AEC-Q100 Grade 2.
The product definition of the new LVDS-based family of FPD-Link III Serialiser/Deserialiser chips is oriented towards a total-system concept of a camera-based driver assistance system. The problematic nature of imager configurations at the ECU side was solved elegantly with an embedded control channel function at very low, deterministic latency with minimal timing variations. Through the integration of a continuously working control and return channel conventional control busses are no longer needed, resulting in savings in terms of cost, complexity and weight. The two-wire solution with additional AC-coupling allows for the use of ultrathin and long cable connections at exposed locations and facilitates the placement within the car chassis. The combination of high bandwidth, low EMI, noise immunity, autonomous synchronisation and support of cost-effective twisted pair cable media assure that these chipsets are a real plug-and-play solution. Equipped with these features the SerDES chipsets are well-suited for a variety of video and control data transmission functionalities in a multitude of automotive applications but also in industrial applications.