Encoding Procedures

There are two methods for shared memory allocation and handling in the SDK: external and internal.

External Memory

The following example shows the pseudo code of the encoding procedure with external memory (legacy mode):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
MFXVideoENCODE_QueryIOSurf(session, &init_param, &request);
allocate_pool_of_frame_surfaces(request.NumFrameSuggested);
MFXVideoENCODE_Init(session, &init_param);
sts=MFX_ERR_MORE_DATA;
for (;;) {
   if (sts==MFX_ERR_MORE_DATA && !end_of_stream()) {
      find_unlocked_surface_from_the_pool(&surface);
      fill_content_for_encoding(surface);
   }
   surface2=end_of_stream()?NULL:surface;
   sts=MFXVideoENCODE_EncodeFrameAsync(session,NULL,surface2,bits,&syncp);
   if (end_of_stream() && sts==MFX_ERR_MORE_DATA) break;
   // Skipped other error handling
   if (sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session, syncp, INFINITE);
      do_something_with_encoded_bits(bits);
   }
}
MFXVideoENCODE_Close(session);
free_pool_of_frame_surfaces();

Note the following key points about the example:

Note

It is the application’s responsibility to fill pixels outside of the crop window when it is smaller than the frame to be encoded, especially in cases when crops are not aligned to minimum coding block size (16 for AVC and 8 for HEVC and VP9).

Internal Memory

The following example shows the encoding procedure with internal memory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
MFXVideoENCODE_Init(session, &init_param);
sts=MFX_ERR_MORE_DATA;
for (;;) {
   if (sts==MFX_ERR_MORE_DATA && !end_of_stream()) {
      MFXMemory_GetSurfaceForEncode(session,&surface);
      fill_content_for_encoding(surface);
   }
   surface2=end_of_stream()?NULL:surface;
   sts=MFXVideoENCODE_EncodeFrameAsync(session,NULL,surface2,bits,&syncp);
   if (surface2) surface->FrameInterface->Release(surface2);
   if (end_of_stream() && sts==MFX_ERR_MORE_DATA) break;
   // Skipped other error handling
   if (sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session, syncp, INFINITE);
      do_something_with_encoded_bits(bits);
   }
}
MFXVideoENCODE_Close(session);

There are several key differences in this example, compared to external memory (legacy mode):

  • The application doesn’t need to call the MFXVideoENCODE_QueryIOSurf() function to obtain the number of working frame surfaces since allocation is done by the SDK.

  • The application calls the MFXMemory_GetSurfaceForEncode() function to get a free surface for the subsequent encode operation.

  • The application needs to call the FrameInterface->(\*Release) function to decrement the reference counter of the obtained surface after the call to the MFXVideoENCODE_EncodeFrameAsync() function.

Configuration Change

The application changes configuration during encoding by calling the MFXVideoENCODE_Reset() function. Depending on the difference in configuration parameters before and after the change, the SDK encoder will either continue the current sequence or start a new one. If the SDK encoder starts a new sequence, it completely resets internal state and begins a new sequence with the IDR frame.

The application controls encoder behavior during parameter change by attaching the mfxExtEncoderResetOption structure to the mfxVideoParam structure during reset. By using this structure, the application instructs the encoder to start or not start a new sequence after reset. In some cases, the request to continue the current sequence cannot be satisfied and the encoder will fail during reset. To avoid this scenario, the application may query the reset outcome before the actual reset by calling MFXVideoENCODE_Query() function with the mfxExtEncoderResetOption attached to the mfxVideoParam structure.

The application uses the following procedure to change encoding configurations:

  1. The application retrieves any cached frames in the SDK encoder by calling the MFXVideoENCODE_EncodeFrameAsync() function with a NULL input frame pointer until the function returns mfxStatus::MFX_ERR_MORE_DATA.

  2. The application calls the MFXVideoENCODE_Reset() function with the new configuration:

    • If the function successfully sets the configuration, the application can continue encoding as usual.

    • If the new configuration requires a new memory allocation, the function returns mfxStatus::MFX_ERR_INCOMPATIBLE_VIDEO_PARAM. The application must close the SDK encoder and reinitialize the encoding procedure with the new configuration.

External Bitrate Control

The application can make the encoder use the external Bitrate Control (BRC) instead of the native bitrate control. To make the encoder use the external BRC, the application should attach the mfxExtCodingOption2 structure with ExtBRC = MFX_CODINGOPTION_ON and the mfxExtBRC callback structure to the mfxVideoParam structure during encoder initialization. The Init, Reset, and Close callbacks will be invoked inside their corresponding functions: MFXVideoENCODE_Init(), MFXVideoENCODE_Reset(), and MFXVideoENCODE_Close(). The following figure shows asynchronous encoding flow with external BRC (using GetFrameCtrl and Update):

asynchronous encoding flow with external BRC

Note

IntAsyncDepth is the SDK max internal asynchronous encoding queue size. It is always less than or equal to mfxVideoParam::AsyncDepth.

The following pseudo code shows use of the external BRC:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
   #include "mfxvideo.h"
   #include "mfxbrc.h"

   typedef struct {
      mfxU32 EncodedOrder;
      mfxI32 QP;
      mfxU32 MaxSize;
      mfxU32 MinSize;
      mfxU16 Status;
      mfxU64 StartTime;
      // ... skipped
   } MyBrcFrame;

   typedef struct {
      MyBrcFrame* frame_queue;
      mfxU32 frame_queue_size;
      mfxU32 frame_queue_max_size;
      mfxI32 max_qp[3]; //I,P,B
      mfxI32 min_qp[3]; //I,P,B
      // ... skipped
   } MyBrcContext;

   void* GetExtBuffer(mfxExtBuffer** ExtParam, mfxU16 NumExtParam, mfxU32 bufferID)
   {
       int i=0;
       for(i = 0; i < NumExtParam; i++) {
           if(ExtParam[i]->BufferId == bufferID) return ExtParam[i];
       }
       return NULL;
   }

   static int IsParametersSupported(mfxVideoParam *par)
   {
       // do some checks
       return 1;
   }

   static int IsResetPossible(MyBrcContext* ctx, mfxVideoParam *par)
   {
       // do some checks
       return 1;
   }

   static MyBrcFrame* GetFrame(MyBrcFrame *frame_queue, mfxU32 frame_queue_size, mfxU32 EncodedOrder)
   {
       //do some logic
       if(frame_queue_size) return &frame_queue[0];
       return NULL;
   }

   static mfxU32 GetFrameCost(mfxU16 FrameType, mfxU16 PyramidLayer)
   {
       // calculate cost
       return 1;
   }

   static mfxU32 GetMinSize(MyBrcContext *ctx, mfxU32 cost)
   {
       // do some logic
       return 1;
   }

   static mfxU32 GetMaxSize(MyBrcContext *ctx, mfxU32 cost)
   {
       // do some logic
       return 1;
   }

   static mfxI32 GetInitQP(MyBrcContext *ctx, mfxU32 MinSize, mfxU32 MaxSize, mfxU32 cost)
   {
       // do some logic
       return 1;
   }

   static mfxU64 GetTime()
   {
       mfxU64 wallClock;
       return wallClock;
   }

   static void UpdateBRCState(mfxU32 CodedFrameSize, MyBrcContext *ctx)
   {
       return;
   }

   static void RemoveFromQueue(MyBrcFrame* frame_queue, mfxU32 frame_queue_size, MyBrcFrame* frame)
   {
       return;
   }

   static mfxU64 GetMaxFrameEncodingTime(MyBrcContext *ctx)
   {
       return 2;
   }

   mfxStatus MyBrcInit(mfxHDL pthis, mfxVideoParam* par) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;
      mfxI32 QpBdOffset;
      mfxExtCodingOption2* co2;
      mfxI32 defaultQP;

      if (!pthis || !par)
         return MFX_ERR_NULL_PTR;

      if (!IsParametersSupported(par))
         return MFX_ERR_UNSUPPORTED;

      ctx->frame_queue_max_size = par->AsyncDepth;
      ctx->frame_queue = (MyBrcFrame*)malloc(sizeof(MyBrcFrame) * ctx->frame_queue_max_size);

      if (!ctx->frame_queue)
         return MFX_ERR_MEMORY_ALLOC;

      co2 = (mfxExtCodingOption2*)GetExtBuffer(par->ExtParam, par->NumExtParam, MFX_EXTBUFF_CODING_OPTION2);
      QpBdOffset = (par->mfx.FrameInfo.BitDepthLuma > 8) ? (6 * (par->mfx.FrameInfo.BitDepthLuma - 8)) : 0;

      ctx->max_qp[0] = (co2 && co2->MaxQPI) ? (co2->MaxQPI - QpBdOffset) : defaultQP;
      ctx->min_qp[0] = (co2 && co2->MinQPI) ? (co2->MinQPI - QpBdOffset) : defaultQP;

      ctx->max_qp[1] = (co2 && co2->MaxQPP) ? (co2->MaxQPP - QpBdOffset) : defaultQP;
      ctx->min_qp[1] = (co2 && co2->MinQPP) ? (co2->MinQPP - QpBdOffset) : defaultQP;

      ctx->max_qp[2] = (co2 && co2->MaxQPB) ? (co2->MaxQPB - QpBdOffset) : defaultQP;
      ctx->min_qp[2] = (co2 && co2->MinQPB) ? (co2->MinQPB - QpBdOffset) : defaultQP;

      // skipped initialization of other other BRC parameters

      ctx->frame_queue_size = 0;

      return MFX_ERR_NONE;
   }

   mfxStatus MyBrcReset(mfxHDL pthis, mfxVideoParam* par) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;

      if (!pthis || !par)
         return MFX_ERR_NULL_PTR;

      if (!IsParametersSupported(par))
         return MFX_ERR_UNSUPPORTED;

      if (!IsResetPossible(ctx, par))
         return MFX_ERR_INCOMPATIBLE_VIDEO_PARAM;

      // reset here BRC parameters if required

      return MFX_ERR_NONE;
   }

   mfxStatus MyBrcClose(mfxHDL pthis) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;

      if (!pthis)
         return MFX_ERR_NULL_PTR;

      if (ctx->frame_queue) {
         free(ctx->frame_queue);
         ctx->frame_queue = NULL;
         ctx->frame_queue_max_size = 0;
         ctx->frame_queue_size = 0;
      }

      return MFX_ERR_NONE;
   }

   mfxStatus MyBrcGetFrameCtrl(mfxHDL pthis, mfxBRCFrameParam* par, mfxBRCFrameCtrl* ctrl) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;
      MyBrcFrame* frame = NULL;
      mfxU32 cost;

      if (!pthis || !par || !ctrl)
         return MFX_ERR_NULL_PTR;

      if (par->NumRecode > 0)
         frame = GetFrame(ctx->frame_queue, ctx->frame_queue_size, par->EncodedOrder);
      else if (ctx->frame_queue_size < ctx->frame_queue_max_size)
         frame = &ctx->frame_queue[ctx->frame_queue_size++];

      if (!frame)
         return MFX_ERR_UNDEFINED_BEHAVIOR;

      if (par->NumRecode == 0) {
         frame->EncodedOrder = par->EncodedOrder;
         cost = GetFrameCost(par->FrameType, par->PyramidLayer);
         frame->MinSize = GetMinSize(ctx, cost);
         frame->MaxSize = GetMaxSize(ctx, cost);
         frame->QP = GetInitQP(ctx, frame->MinSize, frame->MaxSize, cost); // from QP/size stat
         frame->StartTime = GetTime();
      }

      ctrl->QpY = frame->QP;

      return MFX_ERR_NONE;
   }

   #define DEFAULT_QP_INC 4
   #define DEFAULT_QP_DEC 4

   mfxStatus MyBrcUpdate(mfxHDL pthis, mfxBRCFrameParam* par, mfxBRCFrameCtrl* ctrl, mfxBRCFrameStatus* status) {
      MyBrcContext* ctx = (MyBrcContext*)pthis;
      MyBrcFrame* frame = NULL;
      mfxU32 panic = 0;

      if (!pthis || !par || !ctrl || !status)
         return MFX_ERR_NULL_PTR;

      frame = GetFrame(ctx->frame_queue, ctx->frame_queue_size, par->EncodedOrder);
      if (!frame)
         return MFX_ERR_UNDEFINED_BEHAVIOR;

      // update QP/size stat here

      if (   frame->Status == MFX_BRC_PANIC_BIG_FRAME
        || frame->Status == MFX_BRC_PANIC_SMALL_FRAME)
         panic = 1;

      if (panic || (par->CodedFrameSize >= frame->MinSize && par->CodedFrameSize <= frame->MaxSize)) {
         UpdateBRCState(par->CodedFrameSize, ctx);
         RemoveFromQueue(ctx->frame_queue, ctx->frame_queue_size, frame);
         ctx->frame_queue_size--;
         status->BRCStatus = MFX_BRC_OK;

         // Here update Min/MaxSize for all queued frames

         return MFX_ERR_NONE;
      }

      panic = ((GetTime() - frame->StartTime) >= GetMaxFrameEncodingTime(ctx));

      if (par->CodedFrameSize > frame->MaxSize) {
         if (panic || (frame->QP >= ctx->max_qp[0])) {
            frame->Status = MFX_BRC_PANIC_BIG_FRAME;
         } else {
            frame->Status = MFX_BRC_BIG_FRAME;
            frame->QP = DEFAULT_QP_INC;
         }
      }

      if (par->CodedFrameSize < frame->MinSize) {
         if (panic || (frame->QP <= ctx->min_qp[0])) {
            frame->Status = MFX_BRC_PANIC_SMALL_FRAME;
            status->MinFrameSize = frame->MinSize;
         } else {
            frame->Status = MFX_BRC_SMALL_FRAME;
            frame->QP = DEFAULT_QP_DEC;
         }
      }

      status->BRCStatus = frame->Status;

      return MFX_ERR_NONE;
   }

   void EncoderInit()
   {
        //initialize encoder
        MyBrcContext brc_ctx;
        mfxExtBRC ext_brc;
        mfxExtCodingOption2 co2;
        mfxExtBuffer* ext_buf[2] = {&co2.Header, &ext_brc.Header};
        mfxVideoParam vpar;

        memset(&brc_ctx, 0, sizeof(MyBrcContext));
        memset(&ext_brc, 0, sizeof(mfxExtBRC));
        memset(&co2, 0, sizeof(mfxExtCodingOption2));

        vpar.ExtParam = ext_buf;
        vpar.NumExtParam = sizeof(ext_buf) / sizeof(ext_buf[0]);

        co2.Header.BufferId = MFX_EXTBUFF_CODING_OPTION2;
        co2.Header.BufferSz = sizeof(mfxExtCodingOption2);
        co2.ExtBRC = MFX_CODINGOPTION_ON;

        ext_brc.Header.BufferId = MFX_EXTBUFF_BRC;
        ext_brc.Header.BufferSz = sizeof(mfxExtBRC);
        ext_brc.pthis           = &brc_ctx;
        ext_brc.Init            = MyBrcInit;
        ext_brc.Reset           = MyBrcReset;
        ext_brc.Close           = MyBrcClose;
        ext_brc.GetFrameCtrl    = MyBrcGetFrameCtrl;
        ext_brc.Update          = MyBrcUpdate;

        sts = MFXVideoENCODE_Query(session, &vpar, &vpar);
        if (sts == MFX_ERR_UNSUPPORTED || co2.ExtBRC != MFX_CODINGOPTION_ON)
            // unsupported case
            sts = sts;
        else
            sts = MFXVideoENCODE_Init(session, &vpar);
   }

JPEG

The application can use the same encoding procedures for JPEG/motion JPEG encoding, as shown in the following pseudo code:

// encoder initialization
MFXVideoENCODE_Init (...);
// single frame/picture encoding
MFXVideoENCODE_EncodeFrameAsync (...);
MFXVideoCORE_SyncOperation(...);
// close down
MFXVideoENCODE_Close(...);

ENCODE supports JPEG baseline profile encoding as follows:

  • DCT-based process

  • Source image: 8-bit samples within each component

  • Sequential

  • Huffman coding: 2 AC and 2 DC tables

  • 3 loadable quantization matrices

  • Interleaved and non-interleaved scans

  • Single and multiple scans

    • Chroma sub-sampling ratios:

      • Chroma 4:0:0 (gray image)

      • Chroma 4:1:1

      • Chroma 4:2:0

      • Chroma horizontal 4:2:2

      • Chroma vertical 4:2:2

      • Chroma 4:4:4

  • 3 channel images

The application may specify Huffman and quantization tables during encoder initialization by attaching mfxExtJPEGQuantTables and mfxExtJPEGHuffmanTables buffers to the mfxVideoParam structure. If the application does not define tables then the SDK encoder uses tables recommended in ITU-T* Recommendation T.81. If the application does not define a quantization table it must specify the Quality parameter in the mfxInfoMFX structure. In this case, the SDK encoder scales the default quantization table according to the specified Quality parameter.

The application should properly configured chroma sampling format and color format using the FourCC and ChromaFormat fields in the mfxFrameInfo structure. For example, to encode a 4:2:2 vertically sampled YCbCr picture, the application should set FourCC to MFX_FOURCC_YUY2 and ChromaFormat to MFX_CHROMAFORMAT_YUV422V. To encode a 4:4:4 sampled RGB picture, the application should set FourCC to MFX_FOURCC_RGB4 and ChromaFormat to MFX_CHROMAFORMAT_YUV444.

The SDK encoder supports different sets of chroma sampling and color formats on different platforms. The application must call the MFXVideoENCODE_Query() function to check if the required color format is supported on a given platform and then initialize the encoder with proper values of FourCC and ChromaFormat in the mfxFrameInfo structure.

The application should not define the number of scans and number of components. These numbers are derived by the SDK encoder from the Interleaved flag in the mfxInfoMFX structure and from chroma type. If interleaved coding is specified, then one scan is encoded that contains all image components. Otherwise, the number of scans is equal to number of components. The SDK encoder uses the following component IDs: “1” for luma (Y), “2” for chroma Cb (U), and “3” for chroma Cr (V).

The application should allocate a buffer that is big enough to hold the encoded picture. A rough upper limit may be calculated using the following equation where Width and Height are weight and height of the picture in pixel and BytesPerPx is the number of bytes for one pixel:

BufferSizeInKB = 4 + (Width * Height * BytesPerPx + 1023) / 1024;

The equation equals 1 for a monochrome picture, 1.5 for NV12 and YV12 color formats, 2 for YUY2 color format, and 3 for RGB32 color format (alpha channel is not encoded).

Multi-view Video Encoding

Similar to the decoding and video processing initialization procedures, the application attaches the mfxExtMVCSeqDesc structure to the mfxVideoParam structure for encoding initialization. The mfxExtMVCSeqDesc structure configures the SDK MVC encoder to work in three modes:

  • Default dependency mode: The application specifies NumView and all other fields to zero. The SDK encoder creates a single operation point with all views (view identifier 0 : NumView-1) as target views. The first view (view identifier 0) is the base view. Other views depend on the base view.

  • Explicit dependency mode: The application specifies NumView and the view dependency array, and sets all other fields to zero. The SDK encoder creates a single operation point with all views (view identifier View[0 : NumView-1].ViewId) as target views. The first view (view identifier View[0].ViewId) is the base view. View dependencies are defined as mfxMVCViewDependency structures.

  • Complete mode: The application fully specifies the views and their dependencies. The SDK encoder generates a bitstream with corresponding stream structures.

The SDK MVC encoder does not support importing sequence and picture headers via the mfxExtCodingOptionSPSPPS structure.

During encoding, the SDK encoding function MFXVideoENCODE_EncodeFrameAsync() accumulates input frames until encoding of a picture is possible. The function returns mfxStatus::MFX_ERR_MORE_DATA for more data at input or mfxStatus::MFX_ERR_NONE if it successfully accumulated enough data for encoding a picture. The generated bitstream contains the complete picture (multiple views). The application can change this behavior and instruct the encoder to output each view in a separate bitstream buffer. To do so, the application must turn on the ViewOutput flag in the mfxExtCodingOption structure. In this case, the encoder returns mfxStatus::MFX_ERR_MORE_BITSTREAM if it needs more bitstream buffers at output and mfxStatus::MFX_ERR_NONE when processing of the picture (multiple views) has been finished. It is recommended that the application provide a new input frame each time the SDK encoder requests new bitstream buffer. The application must submit view data for encoding in the order they are described in the mfxExtMVCSeqDesc structure. Particular view data can be submitted for encoding only when all views that it depends upon have already been submitted.

The following pseudo code shows the encoding procedure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
mfxExtBuffer *eb;
mfxExtMVCSeqDesc  seq_desc;
mfxVideoParam init_param;

init_param.ExtParam=(mfxExtBuffer **)&eb;
init_param.NumExtParam=1;
eb=(mfxExtBuffer *)&seq_desc;

/* init encoder */
MFXVideoENCODE_Init(session, &init_param);

/* perform encoding */
for (;;) {
    MFXVideoENCODE_EncodeFrameAsync(session, NULL, surface2, bits,
                                    &syncp);
    MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
}

/* close encoder */
MFXVideoENCODE_Close(session);