V4L2 Brainstorming meeting notes (day 1)

Controls are split between standard and implementation (hardware) specific controls. Standard controls need to reference the H.264 spec. Implementation-specific controls could (should ?) be standardized as well, as several vendors can implement the same controls.

V4L2_PIX_FMT_MPEG pix fmt can be used only for multiplexed streams. New 
V4L2_PIX_FMT_MPEG1, V4L2_PIX_FMT_MPEG2, V4L2_PIX_FMT_MPEG4 will define elementary streams.

UVC H.264 proposed implementation: http://www.quickcamteam.net/uvc-h264/USB_Video_Payload_H.264_0.87.pdf

H.264 "elementary stream" packets are called Network Abstraction Layer Units (NALU). They come in two flavors, one with start codes and one without. We would then need two different pixel formats. Microsoft defined two FourCCs for that (http://msdn.microsoft.com/en-us/library/dd757808%28v=vs.85%29.aspx):

'h264', 'X264' and 'x264' FourCCs also exist, and they all refer to H.264 NALUs with start codes.

Codec hardware is available in many SoCs, but also in GPUs (see VA-API and VDPAU).

V4L2 already includes a codec class, but it's mostly an unused stub. The V4L2 codec definition is very similar to V4L2 M2M devices. V4L2 also defines an effect device interface. The codec and effect device interfaces should be merged and be replaced by M2M devices.

M2M devices currently have both capture and output V4L2 capabilities set, so they show up as capture devices in applications. This could be fixed by skipping (at the application level) devices that have both capabilities sets.

Devices that can do both memory-to-memory processing and live capture (such as the OMAP3 ISP) will still flag video capture capability for their capture video nodes. The media controller API should be used by generic applications to find the default capture device.

Codecs need to determine what kind of data is expected. New FourCCs can be defined for "elementary stream" types (H.263, H.264, ...).

DivX formats also need FourCCs. The DivX specification is proprietary, we don't know how the different DivX formats differ ('DIV3', 'DIV4, 'DIVX', 'DX50', 'XVID', ...). More information is needed there.

VC1 comes in two flavors, VC1 (containerless) and VC1 RCV (VC-1 inside and RCV container). More information is needed on the VC1 RCV format: should we consider this as a multiplexed stream or as an elementary stream? Could be similiar to H264 and AVC1 as described above.

http://fourcc.org/codecs.php defines FOURCC codes for various compressed codecs already, but some of them might just be listed because they're output by proprietary codecs and used in AVI containers. We need to be careful on which FourCCs we add to V4L2.

https://oss.renesas.com/modules/document/?libshcodecs - SuperH / sh-mobile compressed video processing library can also be considered to get an idea about video processing parameters, available on different types of hardware.

A V4L2_CTRL_CLASS_CODEC class is  proposed. The existing MPEG class should be used instead even though  the name is misleading. We should create aliases that replace MPEG with  CODEC.

Vendor-specific controls will be defined as vendor controls and can later be standardized if needed. Whether the vendor control can then be removed is a gray area.

Some controls shared by different codecs can have different min/max values. For instance, the QP range is 0..51 for H.264 and 1..31 for MPEG4/H.263. Should we use two different controls, or a single one ? When switching between codecs, min/max changes can be reported to userspace through a V4L2 control event, so a single control can work.Hans would rather see separate controls if there's only a couple of such controls. Action: Kamil will 
make a list of controls and send it to linux-media.

Introduced to overcome hardware limitation (lack of 5th usb endpoint in the chip), additional markers introduced to carry H.264 data. UVC extension units can probably be used to detect whether the "feature" is available, otherwise the driver can hardcode USB VID:PIDs. We need a new pix_fmt for such streams and let applications demultiplex. The driver will report 2 FourCCs: MJPEG (for compatibility, will disable all embedded streams) and logitech MJPEG. Which embedded streams are enabled will be selected through private controls. Action: Laurent will check if the format (width/height) of the embedded streams is identical to the main stream.

Subdev registration is handled by bus-specific methods (I2C, SPI, ...). Host drivers thus need to handle bus-specific configuration themselves. This should be improved.

Tomasz proposes to allow

The base idea behind the subdev pool is to allow subdevs to be registered by board code at startup, and then associated with host devices at runtime. While the idea makes sense (the new device tree support for ARM would then make it possible to get rid of lots of board code), there's no clear consensus on whether the pain is worth the gain. Utility and helper functions should be provided by the V4L2 core to help subdev registration across different physical busses (for instance a function to register a subdev in a bus-agnostic fashion, using a union that provides bus-specific information for all supported bus types).

One important use case of the subdev pool is to link separate drivers. The FIMC device can capture data directly from the frame buffer. The FIMC and frame buffer devices are separate devices with separate drivers, with no standard way to get to eachother at the moment. The subdev pool might not be the best solution for this, but we need a solution.

Isuess to be resolved:
 - some subdevs have dependency on host device in terms of clock management
   which prevents subdevs to be registered autonomously to subdev pool

To be continued tomorrow.

Sub-subdev might be usefull to model some kind of sensors that supports additional operations like scalling. Sub-subdevices can be used to model such scaller and export the possibility to set format on the scaller input and output and the capture ccd.
Should format setting ioctls be propagated to sub-subdevices for applications that do not use media controller. If yes it would improve compatibility layer for non-media controller application. However it would compilcate drivers design because a lot of business logic has to be moved to a driver.
On stream start all formats on pads are verified. Maybe new callback verify_link should be added to subdev's ops.

Call propagation is also a topic, appearing in the MC-compatibility discussion: how do you set formats and other parameters on all subdevices behind the default device node?

The spec requirement about parameter consistency across open-close calls has to be relaxed to cover cases, where different file-handles _have_ to implement different parameter sets.

The framebuffer device must allow other drivers to access its writeback subdevice. How to do it?

Implement a FB driver on top of a V4L2 driver. An option is to extend vb2 to make this easier and perhaps come to a generic solution.

Can be exported in Media Controller, however currently there is very little information exported to userspace. We need to export some basic information about each entity, at least type (scaler, converter, isp, .. etc).
Entity information is needed for UVC driver. However it is just some binary blob that need to be parsed with userspace. Probably not generic enough to add into other drivers. Maybe having a UVC specific subdev with private ioctl will be enough to cover this topic.

Tags: clipping, composing.

- Allow the last ioctl (S_FMT, S_CROP) to take precedence, i.e if during S_CROP format (resolution) has to be changed then it should be changed, but only before buffers were allocated,
- S_FMT, S_CROP atomic transactions seem not to be a preffered approach,
- Add flags in S_CROP/S_FMT ioclt to pass a hint to the driver how it should behave
and also status flags to inform application about possible pipeline setup invalidation? 
- Need a S_COMPOSE operation, probably also a S_EXTCROP operation. Currently S_CROP is doing cropping on capture but composition on output. This is highly confusing.