This issue came up during the design of an SD/SD/3GSDI to IP encoder board. The board takes a SDI signal in input, compresses it using an H.264 encoder and makes it available as RTP streams via an RTSP server. From the user’s point of view, the system looks like an IP camera: it is controlled via the ONVIF API and video is served via an RTSP server. A simple UI is served via a website for users to do some basic configuration.
The board has a Semtec GS3470 front-end device connected to a Lattice Crosslink FPGA. The GS3470 converts the SDI signal to LVDS7:1. The FPGA receives the LVDS7:1 and sends the pixel data to a MIPI CSI2 core connected to the i.MX8MMini. Finally the MIPI signal received by the i.MX8MMini, and the fun starts!
On the i.MX8MMini, we customised the NXP V4L2 driver so that it could receive the data from our FPGA and communicate with the GS3470 to query the video mode.
Development went fairly smoothly for progressive modes (1080p and 720p), but we also needed to support interlaced formats based on PAL and NTSC and that did not go so well. Here the driver was tweaked to interleave the fields rather than providing separate fields. We chose to do this as we thought it would be simpler for the H.264 encoding. The i.MX8MMini uses the Hantro H1 H.264 encoder, and the documentation did not state support for interlaced modes. Looking at the source code for the Hantro library, interlaced support was not included. We could have attempted to modify the library, but there was no guarantee the IP used in the i.MX8MMini would have supported the change.
We eventually managed to acquire and encode the interlaced formats, but the images had a fixed 6 pixel shift which took months to understand and fix the root cause.
As the shift was 100% stable it was clear something was happening at the very beginning of acquisition leaving 6 pixels somewhere in a FIFO or in a counter, causing the shift. But we could not figure out whether it was happening in the FPGA (a likely culprit as there are two FIFOs in the design), or in the SoC and if it was a software issue or an issue with the MIPI CSI IP in the i.MX8MMini. After about 6 weeks going back and fro, we were out of ideas for things to try and the project was stuck.
In order to find the root, we added debug features to the FPGA. Specifically I wanted to be able to control exactly how many pixels the FPGA was sending to the SoC. I also wanted to add a test pattern so that we could analyse the data received in the V4L2 buffers.
With these changes in place, I was able to meticulously analyse the pixel data acquired in the V4L2 buffers after N lines where sent by the FPGA. The V4L2 driver was instrumented so I could dump the content of the buffer to file for offline analysis.
I found that the first 6 pixels of the first buffer acquired where the last pixels of the very first frame sent by the FPGA. Somehow the rest of the frame had been lost and on top of that those pixels had managed to stay in the i.MX8MMini. The initial suspicion was a bug with the FPGA design, causing it to loose most of the very first frame. But after careful review and simulation of the SystemVerilog code there was no indication this was possible.
I started to wonder if there was a ‘compatibility’ issue with the lattice DPHY code and the i.MX’s DPHY and if somehow the very first frame was partially discarded by the SoC, leaving just 6 pixels in the input FIFO.
We added the ability for the FPGA to stop sending data after a set number of lines and frames as well as the ability to tune the number of pixels per line. This effectively allowed us to control exactly how many pixels were sent from the FPGA to the SoC over MIPI. We alse added a test pattern replacing the pixels from the camera, with the frame number / line number in the first 4 bytes, the pixel number in the others and the line number in the last 2 bytes. This enabled us to look at the very first frame and the very first bytes received by the i.MX8MMini. I could see there was indeed 6 bytes in the first buffer we received from the driver that belonged to the previous frame.
This proved that the very first frame was somehow mostly ignored / lost by the SoC, but that 6 bytes from it still remained in the input FIFO.
In the end the solution was to send an extra empty line over MIPI CSI from the FPGA to the SoC. This somehow allowed the first frame to complete and the link worked thereafter.
JASD 