mesa之virgl video-codec

profile与entrypoint

什么是profile

profile可以理解为一种配置，代表不同的视频处理标准，比如h264标准有多种profile：

VAProfileH264Baseline, VAProfileH264Main, VAProfileH264High, 等等。可以用下图来简单理解这三种profile之间的区别：

libva/va/va.h中定义了profile和entrypoint，mesa和virglrenderer中都要和这里保持一致。

/** Currently defined profiles */
typedef enum {/** \\brief Profile ID used for video processing. */VAProfileNone                       = -1,VAProfileMPEG2Simple        = 0,VAProfileMPEG2Main          = 1,VAProfileMPEG4Simple        = 2,VAProfileMPEG4AdvancedSimple    = 3,VAProfileMPEG4Main          = 4,VAProfileH264Baseline va_deprecated_enum = 5,VAProfileH264Main           = 6,VAProfileH264High           = 7,VAProfileVC1Simple          = 8,VAProfileVC1Main            = 9,VAProfileVC1Advanced        = 10,VAProfileH263Baseline       = 11,VAProfileJPEGBaseline               = 12,VAProfileH264ConstrainedBaseline    = 13,VAProfileVP8Version0_3              = 14,VAProfileH264MultiviewHigh          = 15,VAProfileH264StereoHigh             = 16,VAProfileHEVCMain                   = 17,VAProfileHEVCMain10                 = 18,VAProfileVP9Profile0                = 19,VAProfileVP9Profile1                = 20,VAProfileVP9Profile2                = 21,VAProfileVP9Profile3                = 22,VAProfileHEVCMain12                 = 23,VAProfileHEVCMain422_10             = 24,VAProfileHEVCMain422_12             = 25,VAProfileHEVCMain444                = 26,VAProfileHEVCMain444_10             = 27,VAProfileHEVCMain444_12             = 28,VAProfileHEVCSccMain                = 29,VAProfileHEVCSccMain10              = 30,VAProfileHEVCSccMain444             = 31,VAProfileAV1Profile0                = 32,VAProfileAV1Profile1                = 33,VAProfileHEVCSccMain444_10          = 34,/** \\brief Profile ID used for protected video playback. */VAProfileProtected                  = 35,VAProfileH264High10                 = 36
} VAProfile;

VC1的profile有3个

• VAProfileVC1Simple

• VAProfileVC1Main

• VAProfileVC1Advance

VC-1标准中通过2-bits位来定义profile，一共有三种：

• The Simple profile targets low-rate internet streaming and low-complexity applications such as mobile communications, or playback of media in personal digital assistants. There are two levels in this profile.

• The Main profile targets high-rate internet applications such as streaming, movie delivery via IP, or TV/VOD over IP. This profile contains three levels.

• The Advanced profile targets broadcast applications, such as digital TV, DVD, or HDTV. It is the only profile that supports interlaced content. In addition, this profile contains the required syntax elements to transmit video bitstreams conformant to this standard into generic systems, such as MPEG-2 Transport or Program Streams (ISO/IEC 13818-2). This profile contains five levels.

va与pipe profile对应关系

/* virglrenderer/src/virgl_video.c */
static enum pipe_video_profile pipe_profile_from_va(VAProfile profile)
{switch (profile) {case VAProfileMPEG2Simple:return PIPE_VIDEO_PROFILE_MPEG2_SIMPLE;case VAProfileMPEG2Main:return PIPE_VIDEO_PROFILE_MPEG2_MAIN;case VAProfileMPEG4Simple:return PIPE_VIDEO_PROFILE_MPEG4_SIMPLE;case VAProfileMPEG4AdvancedSimple:return PIPE_VIDEO_PROFILE_MPEG4_ADVANCED_SIMPLE;case VAProfileVC1Simple:return PIPE_VIDEO_PROFILE_VC1_SIMPLE;case VAProfileVC1Main:return PIPE_VIDEO_PROFILE_VC1_MAIN;case VAProfileVC1Advanced:return PIPE_VIDEO_PROFILE_VC1_ADVANCED;case VAProfileH264ConstrainedBaseline:return PIPE_VIDEO_PROFILE_MPEG4_AVC_BASELINE;case VAProfileH264Main:return PIPE_VIDEO_PROFILE_MPEG4_AVC_MAIN;case VAProfileH264High:return PIPE_VIDEO_PROFILE_MPEG4_AVC_HIGH;case VAProfileHEVCMain:return PIPE_VIDEO_PROFILE_HEVC_MAIN;case VAProfileHEVCMain10:return PIPE_VIDEO_PROFILE_HEVC_MAIN_10;case VAProfileJPEGBaseline:return PIPE_VIDEO_PROFILE_JPEG_BASELINE;case VAProfileVP9Profile0:return PIPE_VIDEO_PROFILE_VP9_PROFILE0;case VAProfileVP9Profile2:return PIPE_VIDEO_PROFILE_VP9_PROFILE2;case VAProfileAV1Profile0:return PIPE_VIDEO_PROFILE_AV1_MAIN;case VAProfileNone:return PIPE_VIDEO_PROFILE_UNKNOWN;default:return PIPE_VIDEO_PROFILE_UNKNOWN;}
}

VC1的三个profile对应关系是：

entrypoint是什么

entrypoint是视频处理入口，应用程序先根据profile寻找到对应的配置，再根据实际要处理的操作寻找对应的entrypoint，进入编解码处理流程。

libva/va/va.h中定义的VAEntrypoint有：

typedef enum {VAEntrypointVLD     = 1,VAEntrypointIZZ     = 2,VAEntrypointIDCT        = 3,VAEntrypointMoComp      = 4,VAEntrypointDeblocking  = 5,VAEntrypointEncSlice    = 6,    /* slice level encode */VAEntrypointEncPicture  = 7,    /* pictuer encode, JPEG, etc */VAEntrypointEncSliceLP  = 8,VAEntrypointVideoProc       = 10,   /**< Video pre/post-processing. */VAEntrypointFEI         = 11,VAEntrypointStats       = 12,VAEntrypointProtectedTEEComm       = 13,VAEntrypointProtectedContent       = 14,
}

mesa中支持的entrypoint有：

enum pipe_video_entrypoint
{PIPE_VIDEO_ENTRYPOINT_UNKNOWN,PIPE_VIDEO_ENTRYPOINT_BITSTREAM,PIPE_VIDEO_ENTRYPOINT_IDCT,PIPE_VIDEO_ENTRYPOINT_MC,PIPE_VIDEO_ENTRYPOINT_ENCODE,PIPE_VIDEO_ENTRYPOINT_PROCESSING,
};

entrypoint的含义

va与pipe的entrypoint对应关系

/* virlgrenderer/src/virgl_video.c */
/* NOTE: mesa va frontend only supports VLD and EncSlice */
static enum pipe_video_entrypoint pipe_entrypoint_from_va(VAEntrypoint entrypoint)
{switch (entrypoint) {case VAEntrypointVLD:return PIPE_VIDEO_ENTRYPOINT_BITSTREAM;case VAEntrypointIDCT:return PIPE_VIDEO_ENTRYPOINT_IDCT;case VAEntrypointMoComp:return PIPE_VIDEO_ENTRYPOINT_MC;case VAEntrypointEncSlice: /* fall through */case VAEntrypointEncSliceLP:return PIPE_VIDEO_ENTRYPOINT_ENCODE;default:return PIPE_VIDEO_ENTRYPOINT_UNKNOWN;}
}

对应关系

我们可以先不考虑VAEntrypointMoComp运动补偿，暂时只考虑VLD和Slice两种entrypoint。

vainfo工作原理

当输入vainfo命令时，会输出系统支持的profile和entrypoints，以profile为例，其代码执行流：

mesa中support profile/entrypoint/config的代码在src/gallium/frontends/va/config.c中：

修改config.c中的vlVaQueryConfigProfiles(),vlVaQueryConfigEntrypoints()等来支持VC1。

整体原理图如下：

所以需要支持vainfo中显示VC1，核心只要修改virglrenderer中video_caps的初始化（增加VC1支持），mesa的virtio-gpu中支持VC1的virgl_video_get_param()操作即可。

结果：来看一个我修改后的vainfo的显示结果（增加了VC1相关的profile和entrypoint）

video codec初始化

技巧：mesa代码中开启virgl debug模式：export VIRGL_DEBUG=all(全开), video(开启和video相关), tgsi(开启tgsi相关), tgsi,video(开启tgsi和video)。主要有：

verbose, tgsi, use_tgsi, noemubgra, nobgraswz, sync, xfer, r8srgb-readback, nocoheren, video这些个选项

驱动初始化vtable

vtable

mesa中vtable是个回调函数table汇总，在src/gallium/frontends/va/context.c中定义：

static struct VADriverVTable vtable =
{&vlVaTerminate,&vlVaQueryConfigProfiles,&vlVaQueryConfigEntrypoints,&vlVaGetConfigAttributes,&vlVaCreateConfig,&vlVaDestroyConfig,&vlVaQueryConfigAttributes,&vlVaCreateSurfaces,&vlVaDestroySurfaces,&vlVaCreateContext,&vlVaDestroyContext,&vlVaCreateBuffer,&vlVaBufferSetNumElements,&vlVaMapBuffer,&vlVaUnmapBuffer,&vlVaDestroyBuffer,&vlVaBeginPicture,&vlVaRenderPicture,&vlVaEndPicture,&vlVaSyncSurface,&vlVaQuerySurfaceStatus,&vlVaQuerySurfaceError,&vlVaPutSurface,&vlVaQueryImageFormats,&vlVaCreateImage,&vlVaDeriveImage,&vlVaDestroyImage,&vlVaSetImagePalette,&vlVaGetImage,&vlVaPutImage,&vlVaQuerySubpictureFormats,&vlVaCreateSubpicture,&vlVaDestroySubpicture,&vlVaSubpictureImage,&vlVaSetSubpictureChromakey,&vlVaSetSubpictureGlobalAlpha,&vlVaAssociateSubpicture,&vlVaDeassociateSubpicture,&vlVaQueryDisplayAttributes,&vlVaGetDisplayAttributes,&vlVaSetDisplayAttributes,&vlVaBufferInfo,&vlVaLockSurface,&vlVaUnlockSurface,NULL, /* DEPRECATED VaGetSurfaceAttributes */&vlVaCreateSurfaces2,&vlVaQuerySurfaceAttributes,&vlVaAcquireBufferHandle,&vlVaReleaseBufferHandle,
#if VA_CHECK_VERSION(1, 1, 0)NULL, /* vaCreateMFContext */NULL, /* vaMFAddContext */NULL, /* vaMFReleaseContext */NULL, /* vaMFSubmit */NULL, /* vaCreateBuffer2 */NULL, /* vaQueryProcessingRate */&vlVaExportSurfaceHandle,
#endif
#if VA_CHECK_VERSION(1, 15, 0)NULL, /* vaSyncSurface2 */&vlVaSyncBuffer,
#endif
}

vtable_vpp

vtable_vpp也是在src/gallium/frontends/va/context.c中定义：

static struct VADriverVTableVPP vtable_vpp =
{1,&vlVaQueryVideoProcFilters,&vlVaQueryVideoProcFilterCaps,&vlVaQueryVideoProcPipelineCaps
};

创建screen资源

src/gallium/drivers/virgl/virgl_scree.c中virgl_create_screen()函数。

pipe_screen钩子函数

创建screen时，初始化screen->base(struct pipe_screen)的钩子函数。

virgl_get_param

static int
virgl_get_param(struct pipe_screen *screen, enum pipe_cap param){}

该函数主要是针对PIPE_CAP的属性进行处理，应该不涉及VC1具体的东西。

/*** Implementation capabilities/limits which are queried through* pipe_screen::get_param()*/
enum pipe_cap
{PIPE_CAP_GRAPHICS,PIPE_CAP_NPOT_TEXTURES,PIPE_CAP_MAX_DUAL_SOURCE_RENDER_TARGETS,PIPE_CAP_ANISOTROPIC_FILTER,...
};

virgl_get_shader_param

static int
virgl_get_shader_param(struct pipe_screen *screen,enum pipe_shader_type shader,enum pipe_shader_cap param){}

该函数主要是针对pipe shader type的处理：应该不涉及VC1相关标准

/*** Shader stages.** The order must match how shaders are ordered in the pipeline.* The GLSL linker assumes that if i<j, then the j-th shader is* executed later than the i-th shader.*/
typedef enum pipe_shader_type
{MESA_SHADER_NONE = -1,MESA_SHADER_VERTEX = 0,PIPE_SHADER_VERTEX = MESA_SHADER_VERTEX,...
} gl_shader_stage;

virgl_get_video_param

该函数处理video的参数性质，profile、entrypoint，pipe_video_cap等，涉及VC1的具体处理。

static int
virgl_get_video_param(struct pipe_screen *screen,enum pipe_video_profile profile,enum pipe_video_entrypoint entrypoint,enum pipe_video_cap param){}

需要增加处理PIPE_VIDEO_FORMAT_VC1的情况。

创建context资源

由src/gallium/drivers/virgl/virgl_context.c中virgl_context_create()函数实现。

创建codec资源

src/gallium/drivers/virgl/virgl_video.c中定义

数据结构关系总结

下图展示了从screen资源到video的picture desc的数据结构关系图：

video处理

通过下发cmd命令，由virgl驱动继续处理。

/* context cmds to be encoded in the command stream */
enum virgl_context_cmd {VIRGL_CCMD_NOP = 0,.../* video codec */VIRGL_CCMD_CREATE_VIDEO_CODEC,VIRGL_CCMD_DESTROY_VIDEO_CODEC,VIRGL_CCMD_CREATE_VIDEO_BUFFER,VIRGL_CCMD_DESTROY_VIDEO_BUFFER,VIRGL_CCMD_BEGIN_FRAME,VIRGL_CCMD_DECODE_MACROBLOCK,VIRGL_CCMD_DECODE_BITSTREAM,VIRGL_CCMD_ENCODE_BITSTREAM,VIRGL_CCMD_END_FRAME,VIRGL_MAX_COMMANDS
};