SPRADB0 may   2023 AM62A3 , AM62A3-Q1 , AM62A7 , AM62A7-Q1 , AM67A , AM68A , AM69A

 

  1.   1
  2.   Abstract
  3.   Trademarks
  4. 1Introduction
    1. 1.1 Intended Audience
    2. 1.2 Host Machine Information
  5. 2Creating the Dataset
    1. 2.1 Collecting Images
    2. 2.2 Labelling Images
    3. 2.3 Augmenting the Dataset (Optional)
  6. 3Selecting a Model
  7. 4Training the Model
    1. 4.1 Input Optimization (Optional)
  8. 5Compiling the Model
  9. 6Using the Model
  10. 7Building the End Application
    1. 7.1 Optimizing the Application With TI’s Gstreamer Plugins
    2. 7.2 Using a Raw MIPI-CSI2 Camera
  11. 8Summary
  12. 9References

7.2 Using a Raw MIPI-CSI2 Camera

TI's Edge AI Processors, e.g. AM62A, TDA4VM, AM68A, include an ISP and additional hardware accelerators for scaling images (multiscaler MSC) and dewarping distortions from wide-angle lens (Lens Distortion Correction LDC). The ISP, MSC, and LDC are all TI-designed IP within the Vision Processor Accelerator (VPAC). This section will describe how an IMX219 image sensor was used for this application.

Note: The retail-scanner demo also works with USB cameras in 720 and 1080p. When running, use the -c tag with either usb-720p or usb-1080p for 720 or 1080p USB input, resp. The C270 and C920 logitech USB cameras have both been verified to work. The version of the GST pipeline uses USB input to simplify until this point in the document.

An ISP (Image Signal Processor) is necessary to take the raw output of an image sensor and process it into a more typical format like RGB. Raw output sensors are cheaper and external ISP's (including integrated ones like TI's) are more readily tunable than ISP's built into the image sensor. The ISP can be tuned based on lighting conditions, lens effects, auto-exposure, and so forth. TI has a tuning tool for the integrated ISP and supporting documentation.

The IMX219 is a low-cost sensor that is supported out-of-box, and can capture at up to 8 MP at 15 fps or 2 MP at 30 fps (2MP at 60fps is possible but requires a linux patch that is linked in SDK documentation).

Supporting a sensor includes several components, all three of which are available for the IMX219:

  • A driver, ideally on that is upstreamed to the mainline Linx kernel and actively maintained
  • ISP tuning
  • A hardware module for evaluation

The retail-scanner demo required a 2-megapixel input at 30 fps and at least 70° field-of-view. Using this particular sensor posed several challenges for this use case; these challenges and their resolutions are described below:

  • The driver has limited support for different resolutions and formats
    • The typical 16:9 aspect ratio 1920x1080 (2MP) resolution does not have full field of view – it is a cropping the full 8MP frame, and thus appears artificially zoomed in. The result is far below the 70° FOV requirement.
    • Full FOV with 4:3 aspect ratio is possible with 1640x1232 (half resolution in width and height; still 2 MP). However, the upstream driver only works as-is with 10-bit pixel values (RGGB10). This mode is used for the retail-scanner application, and the setup_cameras.sh script within the demo repository is modified to set this configuration using the linux "media-ctl" command-line tool.
    • 60 fps is not supported at 2MP resolutions - it runs at 30 fps. This is a result of the upstream driver. The Edge AI processors from TI are capable of full 60 fps for this sensor, but it requires a patch to the linux kernel, rebuild for the driver module, and installation on the target.
  • Different resolutions may require changes to the ISP files (typically under /opt/imaging/imx219) – this required the "DCC" files to be regenerated. That process is not covered in this document.
    • The 8.6 Edge AI SDK now includes ISP configuration files for all resolution and bit-depths that are possible with the upstream IMX219 driver.