This base class is for video decoders turning encoded data into raw video frames.
The GstVideoDecoder base class and derived subclasses should cooperate as follows:
Configuration
* Initially, GstVideoDecoder calls start when the decoder element is activated, which allows the subclass to perform any
global setup.
* GstVideoDecoder calls set_format to inform the subclass of caps describing input video data that it is about to receive,
including possibly configuration data. While unlikely, it might be called more than once, if changing input parameters require
reconfiguration.
* Incoming data buffers are processed as needed, described in Data Processing below.
* GstVideoDecoder calls stop at end of all processing.
Data processing
* The base class gathers input data, and optionally allows subclass to parse this into subsequently manageable chunks, typically
corresponding to and referred to as 'frames'.
* Each input frame is provided in turn to the subclass' handle_frame callback. * When the subclass enables the subframe
mode with `gst_video_decoder_set_subframe_mode`, the base class will provide to the subclass the same input frame with different input
buffers to the subclass handle_frame callback. During this call, the subclass needs to take ownership of the input_buffer
as GstVideoCodecFrame.input_buffer will have been changed before the next subframe buffer is received. The subclass will
call `gst_video_decoder_have_last_subframe` when a new input frame can be created by the base class. Every subframe will share the same
GstVideoCodecFrame.output_buffer to write the decoding result. The subclass is responsible to protect its access.
* If codec processing results in decoded data, the subclass should call gst_video_decoder_finish_frame to have decoded data
pushed downstream. In subframe mode the subclass should call gst_video_decoder_finish_subframe until the last subframe
where it should call gst_video_decoder_finish_frame. The subclass can detect the last subframe using
GST_VIDEO_BUFFER_FLAG_MARKER on buffers or using its own logic to collect the subframes. In case of decoding failure, the subclass must
call gst_video_decoder_drop_frame or gst_video_decoder_drop_subframe, to allow the base class to do timestamp
and offset tracking, and possibly to requeue the frame for a later attempt in the case of reverse playback.
Shutdown phase
* The GstVideoDecoder class calls stop to inform the subclass that data parsing will be stopped.
Additional Notes
* Seeking/Flushing
* When the pipeline is seeked or otherwise flushed, the subclass is informed via a call to its reset callback, with the
hard parameter set to true. This indicates the subclass should drop any internal data queues and timestamps and prepare for a fresh set
of buffers to arrive for parsing and decoding.
* End Of Stream
* At end-of-stream, the subclass parse function may be called some final times with the at_eos parameter set to true,
indicating that the element should not expect any more data to be arriving, and it should parse and remaining frames and call
have_frame if possible.
The subclass is responsible for providing pad template caps for source and sink pads. The pads need to be named "sink" and "src". It also
needs to provide information about the output caps, when they are known. This may be when the base class calls the subclass'
set_format function, though it might be during decoding, before calling gst_video_decoder_finish_frame. This is done
via gst_video_decoder_set_output_state
The subclass is also responsible for providing (presentation) timestamps (likely based on corresponding input ones). If that is not
applicable or possible, the base class provides limited framerate based interpolation.
Similarly, the base class provides some limited (legacy) seeking support if specifically requested by the subclass, as full-fledged
support should rather be left to upstream demuxer, parser or alike. This simple approach caters for seeking and duration reporting using
estimated input bitrates. To enable it, a subclass should call gst_video_decoder_set_estimate_rate to enable handling of
incoming byte-streams.
The base class provides some support for reverse playback, in particular in case incoming data is not packetized or upstream does not
provide fragments on keyframe boundaries. However, the subclass should then be prepared for the parsing and frame processing stage to
occur separately (in normal forward processing, the latter immediately follows the former), The subclass also needs to ensure the parsing
stage properly marks keyframes, unless it knows the upstream elements will do so properly for incoming data.
The bare minimum that a functional subclass needs to implement is:
* Provide pad templates * Inform the base class of output caps via gst_video_decoder_set_output_state
* Parse input data, if it is not considered packetized from upstream Data will be provided to parse which should invoke
gst_video_decoder_add_to_frame and gst_video_decoder_have_frame to separate the data belonging to each video
frame.
* Accept data in handle_frame and provide decoded results to gst_video_decoder_finish_frame, or call
gst_video_decoder_drop_frame.
- public void add_to_frame (int n_bytes)
Removes next n_bytes of input data and adds it to
currently parsed frame.
- public Buffer allocate_output_buffer ()
Helper function that allocates a buffer to hold a video frame for
this's current CodecState.
- public FlowReturn allocate_output_frame (CodecFrame frame)
Helper function that allocates a buffer to hold a video frame for
this's current CodecState.
- public FlowReturn allocate_output_frame_with_params (CodecFrame frame, BufferPoolAcquireParams @params)
- public virtual bool close ()
Optional.
- public virtual bool decide_allocation (Query query)
Optional.
- public virtual FlowReturn drain ()
Optional.
- public FlowReturn drop_frame (owned CodecFrame frame)
Similar to
finish_frame, but drops frame in any case and posts a
QoS message with the frame's details on the bus.
- public FlowReturn drop_subframe (owned CodecFrame frame)
Drops input data.
- public virtual FlowReturn finish ()
Optional.
- public FlowReturn finish_frame (owned CodecFrame frame)
frame should have a valid decoded data buffer, whose
metadata fields are then appropriately set according to frame data and pushed downstream.
- public FlowReturn finish_subframe (owned CodecFrame frame)
Indicate that a subframe has been finished to be decoded by the
subclass.
- public virtual bool flush ()
Optional.
- public void get_allocator (out Allocator allocator, out AllocationParams @params)
Lets Decoder sub-classes to know
the memory allocator used by the base class and its params.
- public BufferPool get_buffer_pool ()
- public int get_estimate_rate ()
- public CodecFrame get_frame (int frame_number)
- public List<CodecFrame> get_frames ()
- public uint get_input_subframe_index (CodecFrame frame)
Queries the number of the last subframe received by the decoder
baseclass in the frame.
- public void get_latency (out ClockTime min_latency, out ClockTime max_latency)
Query the configured decoder latency.
- public ClockTimeDiff get_max_decode_time (CodecFrame frame)
Determines maximum possible decoding time for frame that
will allow it to decode and arrive in time (as determined by QoS events).
- public int get_max_errors ()
- public bool get_needs_format ()
Queries decoder required format handling.
- public bool get_needs_sync_point ()
Queries if the decoder requires a sync point before it starts
outputting data in the beginning.
- public CodecFrame get_oldest_frame ()
- public CodecState get_output_state ()
Get the CodecState
currently describing the output stream.
- public bool get_packetized ()
Queries whether input data is considered packetized or not by the base
class.
- public size_t get_pending_frame_size ()
Returns the number of bytes previously added to the current frame by
calling add_to_frame.
- public uint get_processed_subframe_index (CodecFrame frame)
Queries the number of subframes in the frame processed by the decoder
baseclass.
- public double get_qos_proportion ()
- public bool get_subframe_mode ()
Queries whether input data is considered as subframes or not by the
base class.
- public virtual Caps getcaps (Caps filter)
Optional.
- public virtual FlowReturn handle_frame (owned CodecFrame frame)
- public virtual bool handle_missing_data (ClockTime timestamp, ClockTime duration)
- public FlowReturn have_frame ()
Gathers all data collected for currently parsed frame, gathers
corresponding metadata and passes it along for further processing, i.
- public FlowReturn have_last_subframe (CodecFrame frame)
Indicates that the last subframe has been processed by the decoder in
frame.
- public void merge_tags (TagList? tags, TagMergeMode mode)
Sets the audio decoder tags and how they should be merged with any
upstream stream tags.
- public virtual bool negotiate ()
Negotiate with downstream elements to currently configured
CodecState.
- public virtual bool open ()
Optional.
- public virtual FlowReturn parse (CodecFrame frame, Adapter adapter, bool at_eos)
Required for non-packetized input.
- public virtual bool propose_allocation (Query query)
Optional.
- public Caps proxy_getcaps (Caps? caps, Caps? filter)
Returns caps that express caps (or sink template caps if
caps == NULL) restricted to resolution/format/... combinations supported by downstream elements.
- public void release_frame (owned CodecFrame frame)
Similar to
drop_frame, but simply releases frame without any processing other than removing it from list of pending frames,
after which it is considered finished and released.
- public void request_sync_point (CodecFrame frame, DecoderRequestSyncPointFlags flags)
Allows the Decoder subclass to
request from the base class that a new sync should be requested from upstream, and that frame was the frame when the
subclass noticed that a new sync point is required.
- public virtual bool reset (bool hard)
Optional.
- public void set_estimate_rate (bool enabled)
Allows baseclass to perform byte to time estimated conversion.
- public virtual bool set_format (CodecState state)
Notifies subclass of incoming data format (caps).
- public CodecState set_interlaced_output_state (Format fmt, InterlaceMode interlace_mode, uint width, uint height, CodecState? reference)
Same as
set_output_state() but also allows you to also set the
interlacing mode.
- public void set_latency (ClockTime min_latency, ClockTime max_latency)
Lets Decoder sub-classes tell the
baseclass what the decoder latency is.
- public void set_max_errors (int num)
Sets numbers of tolerated decoder errors, where a tolerated one is
then only warned about, but more than tolerated will lead to fatal error.
- public void set_needs_format (bool enabled)
Configures decoder format needs.
- public void set_needs_sync_point (bool enabled)
Configures whether the decoder requires a sync point before it starts
outputting data in the beginning.
- public CodecState set_output_state (Format fmt, uint width, uint height, CodecState? reference)
Creates a new
CodecState with the specified fmt, width and height as the output state for the decoder.
- public void set_packetized (bool packetized)
Allows baseclass to consider input data as packetized or not.
- public void set_subframe_mode (bool subframe_mode)
If this is set to TRUE, it informs the base class that the subclass
can receive the data at a granularity lower than one frame.
- public void set_use_default_pad_acceptcaps (bool use)
Lets Decoder sub-classes decide if
they want the sink pad to use the default pad query handler to reply to accept-caps queries.
- public virtual bool sink_event (Event event)
Optional.
- public virtual bool sink_query (Query query)
Optional.
- public virtual bool src_event (Event event)
Optional.
- public virtual bool src_query (Query query)
Optional.
- public virtual bool start ()
Optional.
- public virtual bool stop ()
Optional.
- public virtual bool transform_meta (CodecFrame frame, Meta meta)
Optional.
