[Wine] RFC: New dependency on FFmpeg?

[Andrew Eikum]

I have a working implementation of the new xaudio2 API in my tree.
Microsoft is treating it as a replacement for dsound[1], and as a
result, this new API is used by lots of recent games; see Bug
26808[2] for some examples.

Most games that use the xaudio2 API use a version of Microsoft's WMA
codec. Wine doesn't currently have the ability to decode this audio
for playback. In order for the xaudio2 implementation to be useful for
most games, we'll need some way to convert it. I chose to use the
FFmpeg library to decode WMA audio.

This means for most games, Wine's built-in xaudio2 will require that
the 32-bit FFmpeg library be available on the system to successfully
play audio.

The new code only uses the audio decoding features of FFmpeg,
specifically linking only against libavcodec and libavutil. The newest
APIs that we use were introduced in 2013. Most of the rest date from
2011 and a few from 2002. FFmpeg's API isn't stable, but I don't
expect much maintenance burden from the small piece that we're using.

An additional complication is the libav fork. The APIs that we're
using are currently unchanged between FFmpeg and libav, so libav
should function as a drop-in replacement. Further, distros seem to be
migrating back to FFmpeg, if they ever changed. So again, I don't
expect this to be a barrier.

Wine packagers and developers, do you have any thoughts or objections
to depending on FFmpeg/libav's libavutils and libavcodec for xaudio2
support?

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ee415813%28v=vs.85%29.aspx

[2] https://bugs.winehq.org/show_bug.cgi?id=26808

Thanks,
Andrew

[Andrew Udvare]

Post by Andrew Eikum
An additional complication is the libav fork. The APIs that we're
using are currently unchanged between FFmpeg and libav, so libav
should function as a drop-in replacement. Further, distros seem to be
migrating back to FFmpeg, if they ever changed. So again, I don't
expect this to be a barrier.
Wine packagers and developers, do you have any thoughts or objections
to depending on FFmpeg/libav's libavutils and libavcodec for xaudio2
support?

As unfortunate as this is, most applications that uses those libraries
are packaging their own copy of ffmpeg for a few reasons:

- ffmpeg APIs break (on point releases) too much
- Depending on the distro's copy of ffmpeg is very unreliable because of
said point releases
- libav (another project)/ffmpeg incompatibilities (I know you mentioned
this but numerous amount of people are probably still using libav by
default)

So basically I think you would have to propose inclusion of a version of
ffmpeg into Wine's source to get the green-light (and update as
necessary). Otherwise there will be a lot of new bug reports of crashes
landing in a probably stripped copy of the system's ffmpeg or libav,
with users not knowing how to give good information for debugging.

Just my 2 cents.

Andrew

[Florian Pelz]

Post by Andrew Udvare
So basically I think you would have to propose inclusion of a version of
ffmpeg into Wine's source to get the green-light (and update as
necessary). Otherwise there will be a lot of new bug reports of crashes
landing in a probably stripped copy of the system's ffmpeg or libav,
with users not knowing how to give good information for debugging.

ffmpeg/libav sometimes has security issues. Including a vulnerable
version with Wine would make it impossible for the user to update if
Wine were stuck on an old ffmpeg version.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://www.winehq.org/pipermail/wine-users/attachments/20150821/ad436474/attachment.sig>

[g***@bege.email]

Hi

I'd be curious, why wouldnt it be possible to dynamicly load ffmpeg
library if existing in the system.
(via search path, ld path etc.)?
I mean on a minor revision the API of ffmpeg should more or less stay
the same,
I have not that much experience with ffmpeg - I used it once
(programming wise).

Post by Florian Pelz

Post by Andrew Udvare
So basically I think you would have to propose inclusion of a version of
ffmpeg into Wine's source to get the green-light (and update as
necessary). Otherwise there will be a lot of new bug reports of crashes
landing in a probably stripped copy of the system's ffmpeg or libav,
with users not knowing how to give good information for debugging.

ffmpeg/libav sometimes has security issues. Including a vulnerable
version with Wine would make it impossible for the user to update if
Wine were stuck on an old ffmpeg version.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
<http://www.winehq.org/pipermail/wine-users/attachments/20150821/ad436474/attachment.sig>

[Martin Gregorie]

Post by g***@bege.email
Hi
I'd be curious, why wouldnt it be possible to dynamicly load ffmpeg
library if existing in the system.
(via search path, ld path etc.)?

I haven't tried to use ffmpeg, but I have written dynamically loaded
libraries as part of a plugin system for an emulator. Both main program
and libraries are C code for Linux. The program will compile and link
easily enough if your code is compatible with the library function's
prototype. However, if the function's interface doesn't match the
prototype (which could be the case if the library was replaced with a
later version and the caller was not recompiled) or there's a name
clash between extern variable names, i.e. the same variable is defined
in both library and calling program, then the library will load OK but
you'll get a linkage failure at run time.


HTH

Martin

[Georg Bege]

Hello

Of course, I know this - but you can react on those errors on runtime.
Eg. disabling the decoding/encoding - inform the user etc.
It could be feasable, if static linking cannot be done due various
distribution conflicts... however as we also read in the mailing list
already.

Maybe its better to just implement the encoder then...

Post by Martin Gregorie

Post by g***@bege.email
Hi
I'd be curious, why wouldnt it be possible to dynamicly load ffmpeg
library if existing in the system.
(via search path, ld path etc.)?

I haven't tried to use ffmpeg, but I have written dynamically loaded
libraries as part of a plugin system for an emulator. Both main program
and libraries are C code for Linux. The program will compile and link
easily enough if your code is compatible with the library function's
prototype. However, if the function's interface doesn't match the
prototype (which could be the case if the library was replaced with a
later version and the caller was not recompiled) or there's a name
clash between extern variable names, i.e. the same variable is defined
in both library and calling program, then the library will load OK but
you'll get a linkage failure at run time.
HTH
Martin

--
Georg Bege
Mail: ***@bege.email
XMPP: ***@ninth-art.de
IRC: megaTherion @ Freenode

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://www.winehq.org/pipermail/wine-users/at

[Andrew Eikum]

FFmpeg doesn't have a particularly stable API or ABI. Dynamically
linking to it isn't that great of an idea because the .so number
version increments fairly often requiring at least a relink. The
audio decode routine is up to avcodec_decode_audio4 by now, and who
knows when avcodec_decode_audio5 will happen. The deprecated
functions do eventually get removed and distros can have them
compiled out.
It's a really good lib if you can require specific versions, and
either static-link it or ship with the version you build with, but I
imagine the constantly-breaking ABI would be more of a headache for
a codebase like Wine unless you directly copy over the code (but
then, why not just take the decoder routines ffmpeg has and wrap
them into whatever Windows audio decoder API?).

Hmm, okay, thanks. We really only need the WMA decoder. That probably
wouldn't be too hard to pull into Wine, but I was hoping to avoid it
for all the obvious reasons. If it's our best option, though, then
maybe that's what we should do.

Andrew

[Pau Garcia i Quiles]

Hello,

Most Linux distributions using libav have voted to move to ffmpeg in the
future, Debian (and thus Ubuntu) being one of the most prominent ones.

Even with ffmpeg API changes, remaining source-compatible should not be a
problem for wine. But I guess the problem is not source-compatibility for
wine but binary-compatibility for CrossOver. In that case, I'd suggest
dlopen'ing and checking for the symbol you want. Maybe even have different
code paths if avcodec_decode_audio5 is available (that means a later
version of ffmpeg is available on that system).

Or you could forget about everything and just take libwmapro from RockBox:

https://github.com/mguentner/rockbox/tree/master/apps/codecs/libwmapro

They took ffmpeg, isolated the WMA code and made it a standalone library.
They even further optimized the code. It's taken from an old version of
ffmpeg (2010) but it works :-)

Post by Andrew Eikum
I have a working implementation of the new xaudio2 API in my tree.
Microsoft is treating it as a replacement for dsound[1], and as a
result, this new API is used by lots of recent games; see Bug
26808[2] for some examples.
Most games that use the xaudio2 API use a version of Microsoft's WMA
codec. Wine doesn't currently have the ability to decode this audio
for playback. In order for the xaudio2 implementation to be useful for
most games, we'll need some way to convert it. I chose to use the
FFmpeg library to decode WMA audio.
This means for most games, Wine's built-in xaudio2 will require that
the 32-bit FFmpeg library be available on the system to successfully
play audio.
The new code only uses the audio decoding features of FFmpeg,
specifically linking only against libavcodec and libavutil. The newest
APIs that we use were introduced in 2013. Most of the rest date from
2011 and a few from 2002. FFmpeg's API isn't stable, but I don't
expect much maintenance burden from the small piece that we're using.
An additional complication is the libav fork. The APIs that we're
using are currently unchanged between FFmpeg and libav, so libav
should function as a drop-in replacement. Further, distros seem to be
migrating back to FFmpeg, if they ever changed. So again, I don't
expect this to be a barrier.
Wine packagers and developers, do you have any thoughts or objections
to depending on FFmpeg/libav's libavutils and libavcodec for xaudio2
support?
[1]
https://msdn.microsoft.com/en-us/library/windows/desktop/ee415813%28v=vs.85%29.aspx
[2] https://bugs.winehq.org/show_bug.cgi?id=26808
Thanks,
Andrew

[Chris Robinson]

Post by Pau Garcia i Quiles
But I guess the problem is not source-compatibility for
wine but binary-compatibility for CrossOver. In that case, I'd suggest
dlopen'ing and checking for the symbol you want.

There's also issues with structure changes which won't be caught that
way. libwmapro sounds like the sanest solution, short of doing the same
thing with Wine.

[Andrew Eikum]

Post by Chris Robinson

Post by Pau Garcia i Quiles
But I guess the problem is not source-compatibility for
wine but binary-compatibility for CrossOver. In that case, I'd suggest
dlopen'ing and checking for the symbol you want.

There's also issues with structure changes which won't be caught
that way. libwmapro sounds like the sanest solution, short of doing
the same thing with Wine.

Cool, I didn't know about libwmapro. It does look like exactly what we
need, but I'm concerned about it being so out of date. ffmpeg's wma
files alone have over 400 commits since 2011.

Andrew

[Pau Garcia i Quiles]

On Mon, Aug 24, 2015 at 4:09 PM, Andrew Eikum <***@codeweavers.com>
wrote:

Cool, I didn't know about libwmapro. It does look like exactly what we

Post by Andrew Eikum
need, but I'm concerned about it being so out of date. ffmpeg's wma
files alone have over 400 commits since 2011.

I don't know what was added later in ffmpeg but libwmapro definitely plays
WMA files on my Sansa Clip+ running RockBox 3.13.

This is what I would do:

- Take libwmapro as-is to replace the ffmpeg dependency in your xinput2
implementation

- If something is missing, or does not work as expected, contact the
libwmapro maintainers and ask them why they haven't kept up to date with
ffmpeg. Maybe those changes are only adaptations to new ffmpeg APIs, or
unimportant changes. And maybe they will even say "it's a good idea, we'll
look into updating libwmapro"

- If everything else fails, take ffmpeg and spend a non-negligible
amount of time isolating a "libwinewma", which you'll have to maintain
yourself and everything in the future.