bcm2835-codec encoder does not expose V4L2_CID_MIN_BUFFERS_FOR_CAPTURE

[FabienDanieau]

Describe the bug

The bcm2835-codec encoder driver does not expose the V4L2_CID_MIN_BUFFERS_FOR_CAPTURE control (0x00980081). GStreamer's v4l2 encoder element queries this control to determine how many frames the hardware holds before producing output, and uses it to compute the encoder latency reported to the pipeline. When the control is missing, GStreamer falls back to 0, causing the encoder to report zero latency.

The driver already tracks its minimum buffer requirement via port->minimum_buffer.num in queue_setup, but this value is not exposed as a V4L2 control.

Steps to reproduce the behaviour

1. Query the control directly:

$ v4l2-ctl -d /dev/video11 --get-ctrl=0x00980081
unknown control '0x00980081'

2. Run a GStreamer pipeline with debug logging:

GST_DEBUG=v4l2videoenc:6 gst-launch-1.0
libcamerasrc ! capsfilter caps=video/x-raw,width=1280,height=720,framerate=60/1,format=YUY2 \
! queue ! v4l2h264enc extra-controls="controls,repeat_sequence_header=1" \
! 'video/x-h264,level=(string)4' ! fakesink

3. Observe in the output:

gst_v4l2_video_enc_decide_allocation: Setting latency: 0:00:00.000000000

Empirical measurement of the QBUF/DQBUF pattern on the capture pool confirms the encoder buffers 1 frame before producing output. Over
1486 frames at 1280x720@60fps:

Encoding latency (sink QBUF → src DQBUF):
median: 9.7 ms mean: 10.9 ms p95: 14.3 ms max: 18.5 ms

Device (s)

Raspberry Pi 4 Mod. B

System

raspinfo.txt

Logs

Codec Controls

               video_b_frames 0x009909ca (int)    : min=0 max=0 step=1 default=0 value=0 flags=update                                   
               video_gop_size 0x009909cb (int)    : min=0 max=2147483647 step=1 default=60 value=60                                     
           video_bitrate_mode 0x009909ce (menu)   : min=0 max=1 default=0 value=0 (Variable Bitrate)                                    
                video_bitrate 0x009909cf (int)    : min=25000 max=25000000 step=25000 default=10000000 value=10000000                   
         sequence_header_mode 0x009909d8 (menu)   : min=0 max=1 default=1 value=1 (Joined With 1st Frame)                               
       repeat_sequence_header 0x009909e2 (bool)   : default=0 value=0                                                                   
              force_key_frame 0x009909e5 (button) : value=0 flags=write-only, execute-on-write                                          
        h264_minimum_qp_value 0x00990a61 (int)    : min=0 max=51 step=1 default=20 value=20                                             
        h264_maximum_qp_value 0x00990a62 (int)    : min=0 max=51 step=1 default=51 value=51                                             
          h264_i_frame_period 0x00990a66 (int)    : min=0 max=2147483647 step=1 default=60 value=60                                     
                   h264_level 0x00990a67 (menu)   : min=0 max=15 default=11 value=11 (4)                                                
                 h264_profile 0x00990a6b (menu)   : min=0 max=4 default=4 value=4 (High)                                                

No min_buffers_for_capture control present.

Additional context

GStreamer computes the encoder latency as min_buffers * frame_duration. The min_buffers value comes from querying
V4L2_CID_MIN_BUFFERS_FOR_CAPTURE. When missing, it falls back to 0.

A related issue has been filed on GStreamer's side: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/work_items/5030

Adding V4L2_CID_MIN_BUFFERS_FOR_CAPTURE as a volatile read-only control derived from port->minimum_buffer.num on the capture side would fix this. The same would be useful for the decoder.

[pelwell]

It sounds like you are most of the way to a PR - have you tried implementing the necessary change?

[FabienDanieau]

I will, but I have one question about the implementation: port->minimum_buffer.num is populated from the firmware when the MMAL component is created in queue_setup. If I implement V4L2_CID_MIN_BUFFERS_FOR_CAPTURE as a volatile read-only control using g_volatile_ctrl, the value may be queried before queue_setup has been called (i.e., before the component exists). Is there an earlier point in the lifecycle where minimum_buffer.num is available, or should the control just return a default value when ctx->component is NULL?

[FabienDanieau]

The patch seems to work:

$ v4l2-ctl -d /dev/video11 --list-ctrls | grep min_number_of_capture_buffers
  min_number_of_capture_buffers 0x00980927 (int)    : min=1 max=32 step=1 default=1 value=1 flags=read-only, volatile

$ GST_DEBUG=v4l2videoenc:6 timeout 5 gst-launch-1.0   libcamerasrc ! capsfilter caps=video/x-raw,width=1280,height=720,framerate=30/1,format=YUY2   ! queue ! v4l2h264enc extra-controls="controls,repeat_sequence_header=1"   ! 'video/x-h264,level=(string)4' ! fakesink 2>&1 | grep latency
...
Setting latency: 0:00:00.033333333 (1 frame at 30fps)

[6by9]

V4L2_CID_MIN_BUFFERS_FOR_CAPTURE has nothing to do with latency.
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/control.html

V4L2_CID_MIN_BUFFERS_FOR_CAPTURE (integer)

This is a read-only control that can be read by the application and used as a hint to determine the number of CAPTURE buffers to pass to REQBUFS. The value is the minimum number of CAPTURE buffers that is necessary for hardware to work. This control is required for stateful decoders.

The GStreamer code acknowledges this:

  /* FIXME This may not be entirely correct, as encoder may keep some
   * observation without delaying the encoding. Linux Media API need some
   * more work to explicitly expressed the decoder / encoder latency. This
   * value will then become max latency, and the reported driver latency would
   * become the min latency. */

The encoder hardware on the Pi has no requirements for additional buffers as it has no support for B-frames.

The encoding latency of the hardware will be dictated by the number of macroblocks in the frame. For 720p on a Pi4 it will be about 10ms. Pi0-3 will be slightly longer as the clock is lower.
1080p on Pi0-3 will take about 40ms as motion estimation and CABAC are pipelined, hence is able to achieve 30fps despite that being longer than the frame time.

Note that the V4L2 docs for V4L2_CID_MIN_BUFFERS_FOR_CAPTURE say

This control is required for stateful decoders.

So why is GStreamer looking at it for encoding? I would have expected it to be using V4L2_CID_MIN_BUFFERS_FOR_OUTPUT.

V4L2_CID_MIN_BUFFERS_FOR_OUTPUT (integer)

This is a read-only control that can be read by the application and used as a hint to determine the number of OUTPUT buffers to pass to REQBUFS. The value is the minimum number of OUTPUT buffers that is necessary for hardware to work. This control is required for stateful encoders.

V4L2_CID_MIN_BUFFERS_FOR_CAPTURE statically reporting 1 for decoders, and V4L2_CID_MIN_BUFFERS_FOR_OUTPUT reporting 1 for encoders would be reasonable. That is the minimum number of frames required by the 2 queues.
Achieving best performance does require the userspace app to NOT queue the OUTPUT buffer and then wait for the CAPTURE buffer to be produced, but treat the two queues as separate entities.

[FabienDanieau]

Thank you it's much clearer