Name: ChromiumGStreamerBackend
Owner: Samsung
Description: Use GStreamer to play media streams in chromium. We implemented a Media Process which is own by the Browser Process and creates players on-demand. Any Video tag will be backed by a GStreamer pipeline that lives in the Media Process.
Created: 2015-06-17 02:00:00.0
Updated: 2017-12-27 04:30:30.0
Pushed: 2016-12-01 09:41:18.0
Homepage: null
Size: 1726819
Language: null
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Chromium, GStreamer, MediaProcess, Sandbox, MSE, EME, Zero-Copy, GstPlayer, GstGL, GstChromiumHttpSrc, Build, Tips, Maintenance, UnitTests, Upstream, Issues, GstConf2015
80537668dfbb447d0ba4889ba51cb48db9bbaad3 (Sun Nov 27 2016) - tag 57.0.2935.2
This is an experimental project that aims to have GStreamer as media player in Chromium browser. We introduced a dedicated Media Process that maintains GStreamer pipelines. The Media Process is sandboxed with same level as GPU Process. GstPlayer is used to construct and to maintain GStreamer pipelines. Each HTML5 video tag is backed by a GStreamer pipeline that lives in the Media Process.
Same as official chromium/src source code: here.
06/09/2016: (just to give an idea of the delta from official chromium/src)
89 files changed, 905 insertions(+), 25 deletions(-)
70 files changed, 11039 insertions(+)
git diff –diff-filter=AM –stat sha-1 HEAD
Start from a working build of official Chromium Browser. Then refer to this section to build the Chromium GStreamer Backend.
There is only one Media Process no matter the number of video tags in the same pages or the number of pages (~ tabulations).
GStreamer is almost only used through the a high level API GstPlayer (ex: gst_player_new. gst_player_play, gst_player_pause, gst_player_seek)
It reduces GStreamer code lines a lot and it avoids doing mistakes when using more low level GStreamer API.
Exception for the video rendering part because GstGL needs to be setup to use GPU Process. In short we pass an external get_process_addr to GstGL.
Indeed the Media Process does not load OpenGL libraries; it uses chromium API to forward OpenGL calls to GPU Process which is the only sandboxed
process that is allowed to load GL libraries.
Exception also for the GstChromiumHttpSrc. It is a GStreamer element that wraps chromium API to load an url.
__
GstGLContextGPUProcess: A new backend that allows to build the GstGL?s vtable from a given get_proc_addr, i.e. chromium::gles2::GetGLFunctionPointer. See gpu-accelerated-compositing-in-chrome ?From the client's perspective, an application has the option to either write commands directly into the command buffer or use the GL ES 2.0 API via a client side library that we provide which handles the serialization behind the scenes. Both the compositor and WebGL currently use the GL ES client side library for convenience. On the server side, commands received via the command buffer are converted to calls into either desktop OpenGL or Direct3D via ANGLE.? The Media Process has its own connection to the GPU Process and it uses this second mechanism to forward gl calls it. In the Media Process all gl calls happen in the GLThread.
MediaPlayerGStreamer: Created in the Media Process for each video tag. It receives player commands (load, play, pause, seek) from the WebMediaPlayerGStreamer that lives a Renderer Process. It uses the gst-player library to play a stream. In the handler of glimagesink ?client-draw? signal, the current gltexture is wrapped using glGenMailboxCHROMIUM/glProduceTextureDirectCHROMIUM (see mailbox in gpu-accelerated-compositing-in-chrome). The mailbox?s name for each gltexture is forwarded to WebMediaPlayerGStreamer through IPC (MediaChannel). But this can be avoided by making MediaPlayerGStreamer inherits from cc::VideoFrameProvider in order to avoid sending the gltexture id to Renderer Process (to be sync with the web compositor). Indeed according to oop-iframes-rendering and oop-iframes-rendering it is theoretically possible to make our Media Process be a kind of SubFrame renderer. By registering a cc::VideoLayer in the MediaProcess, the ubercompositor will do the synchronisation with the Frame in the RendererProcess. We suspect it is all about having a cc::VideoFrameProvider::Client in MediaProcess.
WebMediaPlayerGStreamer: Created in a Renderer Process for each video tag. It inherits from blink::WebMediaPlayer and cc::VideoFrameProvider. The gltexture is retrieved from a mailbox name and wrapped into a media::VideoFrame using media::VideoFrame::WrapNativeTexture. At any time the compositing engine is free to call GetCurrentFrame(). This late part can be avoided, see MediaPlayerGStreamer description.
GstChromiumHttpSrc: GStreamer element that wraps the chromium media::BufferedDataSource and instantiate a content::WebURLLoader . It was not possible to re-use WebKitWebSrc because some interfaces are missing or has been removed like PlatformMediaResourceLoader. It uses some parts of WebKitWebSrc though but not sure if it is necessary. Also It does not use appsrc comparing to WebKitWebSrc. It is more similar to filesrc due to media::BufferedDataSource design. That?s clearly an area to improve, maybe we can get rid of media::BufferedDataSource and implement missing interface in order to re-use WebKitWebSrc. Indeed internally it uses ResourceDispatcher, see multi-process-resource-loading. Or maybe we can design a complete new element and interfaces that could be shared between blink and WebKit.
__
It has limited access to resources. For example it cannot create network sockets. It has to use chromium API in order to ask the Browser Process to create a socket. This is similar to Renderer Process. Also all system calls are filters using linux kernel feature Seccomp-BPF (link).
We opted for similar design as GPU Process sandbox, using Seccomp-BPF, see chromium sandbox doc. All the following sequence is existing sequence for GPU Process. In some sort the Media Process just defines its own policy for the system calls, see bpf_media_policy_linux.cc. A main difference is that the Media Process allow loading GStreamer plugins dynamically. But the list can be restricted and open is done in the brocker process.
In Media Process main the sandbox is setup before any thread creation. Also there is a preSansbox step where it is possible to dlopen some particular resources. A routine is executed in one go for each system call. In this loop it is decided to allow the system call number, to refuse it or to add a handler for emulation. All of this is setup through Seccomp-BPF. A trap handler (SECCOMP_RET_TRAP), is installed for open and access. Then when starting the sandbox the process forks itself (see chromium/src/sandbox/linux/syscall_broker/broker_process ::BrokerProcess::Init). The child becomes the so called Media Broker Process which instantiates a BrokerHost. The Media Process instantiates a BrokerProcessClient. When a open or access happens it sends a cachable SIGSYS. In this trap handler the BrokerClient emulates the open/access call. It actually asks the BrokerProcessHost, through IPC, to do the real call. Depending on the policy that has been setup the path is allowed or rejected. The return value is passed back to the BrokerClient, i.e. the Media Process, for usage if the call has been accepted.
__
IPC between media process and render process is maintained via browser process. Browser process MediaProcessHost is initialized with a connection to media process during the browser start up. Later, WebMediaPlayerDispatcher is used when the render process decides to create a media player for html media element. A first message is sent to media process. It establishes a direct IPC channel from render process to media process is created.
Establish media channel request comes from WebMediaPlayerGStreamer to RenderThreadImpl (main render thread), which creates MediaChannelHost for render process and sends establish channel message to MediaMessageFilter in browser process, which redirects the message through MediaProcessHost to media process.
Media process, upon receiving the request, establishes the channel (MediaChannel) with the help of MediaChannelFilter and sends back the channel handler which is passed then by browser process to the render process media player and used to further IPC communication between render process and media process. Any messages from media process to render process get filtered by MediaChannelHost and get dispatched by WebMediaPlayerMessageDispatcher to the right WebMediaPlayergStreamer instance.
For any messages from WebMediaPlayerGStreamer WebMediaPlayerMessageDispatcher dispatches messages via MediaChannelHost to MediaChannel in media process. MediaChannel maps the message to the associated media player and call the callback.
__
Media Source Extensions specification.
When load_type param of blink::WebMediaPlayer::load is equal to LoadTypeMediaSource the url is formated to start with mediasourceblob://.
It allows GstPlayer's pipeline to select GstChromiumMediaSrc. This GstBin adds and removes appsrc elements when receiving AddSourceBuffer and RemoveSourceBuffer.
Encoded data is currently sent through IPC message. We can later consider using shared memory but it seems efficient enough like. And it is also what Android does.
Currently seeking is not implemented but WebKitGTK does not support it too. Also it exists on-going work somewhere. And it should be easier to handle it with future multiappsrc element.
__
Encrypted Media Extensions specification.
ChromiumCommonEncryptionDecrypt GStreamer element factory is registered in MediaPlayerGStreamer to decrypt encrypted streams inside mp4 and webm containers.
When MediaPlayerGStreamer receives a need key asynchronous event from ChromiumCommonEncryptionDecrypt (it is triggered by GStreamer protection event) it passes an initial data from encrypted stream pssh box to to the render process.
On key needed event WebMediaPlayerGStreamer passes the initial data from the media process to the default CDM. It registers OnCdmKeysReady callback in order to receive a parsed key data from the web application. Then new keys are sent back to the media process.
On getting add key IPC message from the render process MediaPlayerGStreamer passes the key down to ChromiumCommonEncryptionDecrypt element, which unblocks and starts decrypting of the encrypted frame in place.
__
The main idea is to keep decoded frames in GPU memory all the time. In other words, any round trip to CPU memory has to be avoided. As the rendering is done through OpenGL, the decoded surface has to be converted to a GL texture.
2 solutions depending on the HW:
In our experimentation we have selected the first solution because EGL_EXT_image_dma_buf_import is in EXT.
Whereas EGL_MESA_image_dma_buf_export is still under MESA specific extension. Though it should move to EXT as some point.
To experiment that on desktop openmax backend provided by Gallium3D needs to support Tizonia first instead of Bellagio.
It will be similar improvements we made for vaapi backend provided by Gallium3D.
For additional information see slides 17, 18, 19, 20 and the Demo2 from live or slides
The following diagram exposes the call stack using Gallium drivers (“nouveau” in this case) but we also support intel vaapi driver.
In theory it should work on AMD gpu because our solution is generic in a sense that we support all Gallium drivers.
ogressive streaming:
://www.w3schools.com/html/mov_bbb.ogg
://www.w3schools.com/html/html5_video.asp
eo width="560" height="320" controls>
<source src="http://video.webmfiles.org/big-buck-bunny_trailer.webm" type="video/webm">
deo>
aptive streaming
HLS -->
eo width="480" height="360" controls>
<source src="http://devimages.apple.com/iphone/samples/bipbop/bipbopall.m3u8" type="application/x-mpegURL">
deo>
treamer
eamer >= 1.11 is required.
one official chromium repositories
clone https://chromium.googlesource.com/chromium/tools/depot_tools.git (then add it in front of your PATH)
clone https://chromium.googlesource.com/chromium/chromium
tch and sync all sub repos. (run all of this from chromium directory, not chromium/src)
h --nohooks chromium # only the very first time
to chromium/src and build everything.
rc
ent sync # see proxy section if needed
d/install-build-deps.sh
GN build instructions below
ild Chromium GStreamer Backend
rc
remote add github_gstbackend https://github.com/Samsung/ChromiumGStreamerBackend
fetch github_gstbackend
checkout -b gstbackend --track github_gstbackend/master
replace c30e05c2db2fba181e29437ea94ecea47e8e5b2e 80537668dfbb447d0ba4889ba51cb48db9bbaad3
ent sync # see proxy section
GN build instructions below
ing GN to generate ninja build files
lean out/mybuild/
rgs out/mybuild --list
rgs out/mybuild
hould open a file then put the following flags inside:
_debug = false
_clang = true # false if want to use gcc
e_sysroot = false
oprietary_codecs = true
mpeg_branding = "Chrome"
_component_build = true
able_nacl = false
dia_use_ffmpeg = false
dia_use_libvpx = false
e_debug_fission = false # if using icecc
nux_use_bundled_binutils = false # if using icecc
ang_use_chrome_plugins = false # if clang+icecc
ke_clang_dir=/opt/icecream/ # if clang+icecc
a -C out/mybuild chrome chrome_sandbox -jN # if icecc -j60
n without any sandbox
t/mybuild/chrome --no-sandbox http://www.w3schools.com/html/mov_bbb.ogg
n while disabling setuid and media sandbox
t/mybuild/chrome --disable-media-sandbox --disable-setuid-sandbox http://www.w3schools.com/html/mov_bbb.ogg
n with all sandbox
ME_DEVEL_SANDBOX=out/mybuild/chrome_sandbox ./out/mybuild/chrome http://www.w3schools.com/html/mov_bbb.ogg
n with EME enabled:
t/mybuild/chrome --no-sandbox http://yt-dash-mse-test.commondatastorage.googleapis.com/unit-tests/2016.html
n with zero-copy decoding/rendering:
inspect-1.0 vaapi
A_DRIVER_NAME=nouveau vainfo # or LIBVA_DRIVER_NAME=i965 if you have an intel gpu
t/mybuild/chrome --no-sandbox http://www.w3schools.com/html/html5_video.asp
uch ~/.boto and copy the 3 following lines inside ~/.boto
o]
y = _proxy_ip
y_port = _proxy_port
n
UTH_BOTO_CONFIG=~/.boto gclient sync
UTH_BOTO_CONFIG=~/.boto gclient runhooks
have not managed to push original history of chromium because:
emote: error: File chrome_frame/tools/test/reference_build/chrome_frame/chrome_dll.pdb is
124.32 MB; this exceeds GitHub's file size limit of 100.00 MB"
is file is not there anymore but it was there in the past.
en using tools like "bfg --strip-blobs-bigger-than 50M" it will change all the next SHA
om the point it strips a file.
so there is no official chromium/src repo on github that we could fork.
lution: truncate history to a point where there is not a file bigger than 100.00 MB in order
push to github.
use "git replace" to allow rebasing between our truncated branch and original branch.
t reset $(git commit-tree HEAD^{tree} -m "This commit contains all previous commits to reduce history")
ke our branch point because we truncated the history
replace c30e05c2db2fba181e29437ea94ecea47e8e5b2e 80537668dfbb447d0ba4889ba51cb48db9bbaad3
play chromium original upstream commits on top of our branch with truncated history
checkout NEW_ORIGIN_SHA
checkout -b master_new
rebase c30e05c2db2fba181e29437ea94ecea47e8e5b2e
replace $(git rev-parse HEAD) NEW_ORIGIN_SHA
play gst backend
checkout -b gstbackend --track github_gstbackend/master
rebase master_new
push github_gstbackend gstbackend:master --force
place are located in .git/refs/replace/
replace -l
replace -d SHA
sable nacl to reduce build time
le_nacl = false
ve_webcore_debug_symbols = true
ol_level = 1
mmand lines options for ./out/Release/chrome
st of all command line switches is available here:
://peter.sh/experiments/chromium-command-line-switches/
sable gpu process and particular sandbox at runtime
ME_SANDBOX_DEBUGGING=1 CHROME_DEVEL_SANDBOX=out/Release/chrome_sandbox ./out/Release/chrome --allow-sandbox-debugging --disable-render-sandbox --disable-hang-monitor http://localhost/test/mov_bbb.ogg
ME_DEVEL_SANDBOX=out/Release/chrome_sandbox ./out/Release/chrome --disable-render-sandbox --disable-gpu --disable-hang-monitor about:blank
n chromium in debug mode
ME_DEVEL_SANDBOX=out/Debug/chrome_sandbox ./out/Debug/chrome http://www.w3schools.com/html/mov_bbb.ogg
ME_DEVEL_SANDBOX=out/Debug/chrome_sandbox ./out/Debug/chrome --media-startup-dialog --allow-sandbox-debugging about:blank
b
t/Release/chrome --disable-media-sandbox --disable-setuid-sandbox --allow-sandbox-debugging --media-launcher='xterm -title renderer -e gdb --args'
ME_DEVEL_SANDBOX=out/Release/chrome_sandbox ./out/Release/chrome --disable-render-sandbox --disable-gpu --disable-hang-monitor --media-startup-dialog --allow-sandbox-debugging about:blank
able / disable gpu workarounds
t/Release/chrome --no-sandbox --use_virtualized_gl_contexts=0
workarounds are listed in chromium/src/gpu/config/gpu_driver_bug_workaround_type.h
sable all processes, i.e. make all processes running as in-process mode in browser process.
ngle-process
n media process as in-process mode, so it will be thread in Browser Process
-media-process
gs in debug mode
G(level) in debug mode, DVLOG(1)
(level) in release mode, VLOG(1)
able-logging=stderr --v=N (level)
able-logging=stderr --v=1
tach debugger
--media-startup-dialog to pause the media process at startup and print the pid
attach a debugger: gdb -p the_pid
db type: signal SIGUSR1 to resume pause and type c to continue
dent code
run: git cl format, to indent latest commit.
ild and run "content" unit tests
a -C out/Release content_unittest
t/Release/content_unittests
ild everything
a -C all
ild more group of unit tests at a time
a -C out/Release media_blink_unittests content_unittests media_unittests base_unittests gl_unittests gpu_unittests
nd other group of unit tests and run one
chromium/src -name "*unittests.isolate" : ./media/blink/media_blink_unittests.isolate
a -C out/Release media_blink_unittests
st all tests within "gpu" unit tests group
t/Release/gpu_unittests --gtest_list_tests
n all tests in "gpu" unit tests group that contains "TexSubImage2DFloatDoesClearOnGLES3"
t/Release/gpu_unittests --gtest_filter=*TexSubImage2DFloatDoesClearOnGLES3* --single-process-tests
n all tests in "gpu" unit tests group that contains "GPUConfig"
t/Release/gpu_unittests --gtest_filter=*GPUConfig* --single-process-tests
adless like none gpu try bots
on testing/xvfb.py ./out/Release/ ./out/Release/views_unittests --gtest_filter=*WidgetTest.Transparency* --single-process
st
ntent/test/gpu/run_gpu_test.py list
n
ME_DEVEL_SANDBOX=out/Release/chrome_sandbox ./content/test/gpu/run_unittests.py
ME_DEVEL_SANDBOX=out/Release/chrome_sandbox ./content/test/gpu/run_gpu_test.py gpu_process
n browser tests with visible stdout
ME_DEVEL_SANDBOX=out/Release/chrome_sandbox content/test/gpu/run_gpu_test.py --browser=release --show-stdout webgl_conformance
thout sandbox
ntent/test/gpu/run_gpu_test.py --show-stdout --browser=release --extra-browser-args="--no-sandbox" gpu_process
ild
a -C out/Debug content_shell
n
t/Debug/chrome --run-layout-test --no-sandbox third_party/WebKit/LayoutTests/inspector/network/network-domain-filter.html
t/Release/chrome --run-layout-test --no-sandbox third_party/WebKit/LayoutTests/compositing/animation/animated-composited-inside-hidden.html
her way
a -C out/Release blink_tests
ME_DEVEL_SANDBOX=out/Release/chrome_sandbox python third_party/WebKit/Tools/Scripts/run-webkit-tests compositing
ME_DEVEL_SANDBOX=out/Release/chrome_sandbox python third_party/WebKit/Tools/Scripts/run-webkit-tests --skipped=ignore --no-retry-failures plugins/can-create-without-renderer.html --driver-logging
ficial steps
s://www.chromium.org/developers/testing/webkit-layout-tests
If you signed the CLA with a non gmail account then create a google account from that external email: SignUpWithoutGmail
Sign in to codereview.chromium.org using the email account which one you used to sign the CLA.
Before uploading the patch run “depot-tools-auth login https://codereview.chromium.org” to authentificate using “OAuth2”.
It should open a new tab in your browser starting by “localhost:8090” and after logged in it should be written: “The authentication flow has completed.”
Then you are ready to upload your patch by running “git cl upload”.
To submit to a patch to an existing CL just type “git cl issue 1415793003” before uploading.
To make sure it creates a new CL just type “git cl issue 0”.
To apply locally any CL just run git cl patch 1415793003.