The cause is the following code in SoundEngineEffect::PlaybackProc()

  if (THIS->HasAttackBuffer())
  {
    ALint numBuffersProcessed = 0;
    while (numBuffersProcessed < 1)
    {
       alGetSourcei(THIS->GetEffectID(), AL_BUFFERS_PROCESSED, &numBuffersProcessed);
         AssertNoOALError("Error getting processed buffer number", end)
    }

While the attack buffer is playing, you’re in a tight polling loop waiting for it to complete.  This is no problem if the preamble is really short, but for OpenGL games, it’s going to cause problems for any sound longer than a fraction of a frame.

The fix, fortunately, is simple.  Sleep this thread for a few milliseconds each iteration of the polling loop in order to give the rest of your application a chance to run.

The modified code is:

  if (THIS->HasAttackBuffer())
  {
    ALint numBuffersProcessed = 0;
    while (numBuffersProcessed < 1)
    {
       alGetSourcei(THIS->GetEffectID(), AL_BUFFERS_PROCESSED, &numBuffersProcessed);
         AssertNoOALError("Error getting processed buffer number", end)
       usleep(5000);  // Sleep for 5 mS each polling loop
    }

This little change makes all the difference for this bit of code and makes it suitable for long (> 1second) preambles.

I’ve found SoundEngine to work pretty well, although I have had to make a number of modifications to it for my specific application.  However, I think I’m at the point where I need to write my own OpenAL sound processing class library.  I’d be interested to hear from some of you to see what you’re using for game sound.

One of the questions that I’ve faced myself, and that I’ve now tried to answer many times for others, is, “Which method should I use to play sounds for my app?”

Here’s the overview, right from Apple’s Getting Started with Audio & Video document:

Playing Audio and Invoking Vibration
Depending on your needs, you play audio in iPhone OS using System Sound Services, the AVAudioPlayer class, Audio Queue Services, OpenAL, or the I/O audio unit.

• To play alerts and user-interface sound effects, or to invoke vibration, use System Audio Services. This technology can play .caf, .wav, and .aif files containing sounds with durations of 30 seconds or less. Refer to “Using Sound in iPhone OS” in iPhone Application Programming Guide and to System Sound Services Reference.
• To play sounds of any duration, to play multiple sounds simultaneously [dgs- see comments on this below], or to play sounds with level control, use the AVAudioPlayer class. This technology can play any audio format supported in iPhone OS including MP3, AAC, ALAC (Apple Lossless), IMA4, linear PCM, and others. See iPhone Application Programming Guide and AVAudioPlayer Class Reference.
• To play sounds with precise control—such as for synchronization—or to play audio captured from an Internet stream, use Audio Queue Services. See Audio Queue Services Programming Guide and Audio Queue Services Reference. For sample code, see SpeakHere.
• To play positional audio, especially if your application is a game, use OpenAL, an open source technology documented at http://openal.org. See OpenAL FAQ for iPhone OS for important information on using this technology in iPhone OS. For sample code, see oalTouch.
• To play sounds with lowest I/O latency, or to provide simultaneous audio input and output, use the I/O audio unit. See the aurioTouch sample.

Some details that are left out of this overview, including:

Compressed Formats on the iPhone — The bottom line here is that the higher level frameworks on the iPhone will only play one highly compressed sound at a time.  This is due to limitations in the processing available to decompress sounds in the audio pipeline.  You might be able to get around this by implementing your own decoder and feeding the resulting uncompressed audio to either OpenAL, or AudioQueue, but that’s beyond my interests at this point.  Apple’s statement above regarding AVAudioPlayer could be a bit misleading, because AVAudioPlayer will not play multiple MP3 sounds simultaneously.  Here’s the section from Coding How-Tos: Audio and Video:

The following limitations pertain for simultaneous sounds in iPhone OS, depending on the audio data format:

• AAC, MP3, and ALAC (Apple Lossless) audio: no simultaneous playback; one sound at a time.

Can I Play an MP3 with SystemAlerts? — No.  SystemAudioServices only plays sounds in linear PCM or IMA4 (IMA/ADPCM) format, packaged as a .caf, .aif, or .wav file.  AVAudioPlayer, on the other hand, supports the playback of any audio format available in the iPhone OS.  This is documented in the iPhone Application Programming Guide: Multimedia Support

Why not just stick with AVAudioPlayer? — AVAudioPlayer is fine, if you don’t need positional audio and if you’re not worried about some significant delays in starting up each player.  This latter bit can be especially important if you’re trying to fire off sounds in a tight loop, or if you need to synchronize effects in a multi-threaded environment.

What I you recommend? — Use SystemAudioServices if your sounds are short and available in the proper formats.  It’s by far the simplest player and, within its constraints, it pretty much handles everything for you.  Use AVAudioPlayer for operations like playing songs or recorded speech.  Unfortunately, Apple doesn’t sanction SoundEngine anymore, but I’ve become a big fan of using SoundEngine on top of OpenAL for games.  This combination hides some of the complexities of OpenAL while still giving you access to most of the standard stuff.  Also, it’s easy to extend SoundEngine to get at pretty much any OpenAL functionality you need.  We downloaded the source for SoundEngine from the Oolong site, but it might be a bit easier to grab it directly from coocos2d.

A final word.  With pretty much everything beyond SystemAudioServices, you’ll want to seriously consider managing the Audio Session yourself.  As always, Apple provides excellent documentation.  Even Apple encourages you to take care of this detail:

Each iPhone OS application gets an audio session object, usually called an audio session. This object comes with some default behavior that you can use to get started in development. Except for certain special cases, however, the default behavior is unsuitable for a shipping application.

You don’t need to instantiate an AudioSession object.  The iPhone OS provides every app with a singleton AudioSession when the app launches.  However, you do have to initialize the AudioSession.  You may also need to set properties, and you’ll probably want to Activate/Deactivate the session.

Happy sounds!

——–
Technorati Tags:  iPhone, iPod Touch, Apple, mobile

Before I go any further, I should share some of the details of the sounds for this app.  They are all relatively short (roughly 2 seconds, or less).  Their intensity peaks early, tapers quickly to a relatively low level, then trails off very gently for most of their duration.  Their time domain plot looks something like a question mark rotated -90 degrees, then mirrored across the x-axis.

Since, as previously discussed, I couldn’t use AudioServices, I had to figure out a way to make either SoundEngine/OpenAL, or AVAudioPlayer (which was later rejected for a different reason), work.  Some of my less embarrassing attempts to solve this included:

The first tactic that I tried was to simply stop the currently playing effect/player (for the sake of simplicity, I’ll just refer to both as players for the rest of this post) and start it again.  Unfortunately, this introduced a distinct, and very unpleasant, click when the stop/start happened.  If your app is a shooter, with lots of sharp explosions and weapon reports, this might not be a problem, but it was a showstopper for us.

The next thing that I tried was gradually reducing the volume of the current player before stopping it.  That didn’t help, much.

With AVAudioPlayer, I also tried just setting the player back to the beginning, instead of stopping it. [myPlayer setCurrentTime:0.0] – the results were actually worse than a stop/restart.

At that point, I decided to try a combination approach.  If the subsequent attempt to play came early in the original player’s span, I just skipped the later attempt.  If the attempt came well into the original, I gradually reduced the volume of the original, stopped it and then started the player again.  This was better, because there were fewer stop/starts, but it still wasn’t perfect.  Which leads us to the solution that I ended up implementing…

In the end, I went with a backup player technique.  I have two players set up to play each sound (I used players here to be consistent.  They’re really effects, since I’m using SoundEngine/OpenAL).  If the first one is busy, I use the second one.  If both are busy, I skip playing the sound.  I was all set to go even deeper, to three, or even four, players for each sound.  Fortunately, two gave good results, and the code was straightforward, so I stuck with a single level of backup.

I probably should have tried this first, but I was worried about both the memory and the execution overhead of having multiple players for each sound.  My concern was not warranted.  With the current implementation, the overhead is negligible, and the results are great.

——–

Technorati Tags:  iPhone, iPod Touch, Apple, mobile, SoundEngine, AVAudio, AudioServices

The problem with that approach quickly became apparent.  There was a noticeable distortion when more than 2-3 sounds were playing at the same time.  The unpleasant effect was kind of a stuttering static, sort of like one might hear if one over-drives speakers.  This, unfortunately, pretty much ruined the effect of the app, so Paul suggested lowering the volume of the sounds.  Sadly, one can’t do that for an AudioServices system sound.

At that point, I switched the app over to using AVAudioPlayer.  This seemed like it was going to be a good solution, so I wrote a utility app to help me test the individual sound volume settings and spent quite a bit of time experimenting to find the “right” level for each possible combination of sounds.  I really thought that I was in the home stretch until I fired up the app and found that the motion went from butter smooth to unacceptably jerky.  Since actual playback of AVAudioPlayer is asynchronous with regards to the main thread, the last thing that I looked at (after wasting more time than I should have on optimization) was the actual call to [myAVAudioPlayer play].  Big mistake.

After eliminating everything else, I bracketed the [myAVAudioPlayer play] calls with a timer and was amazed to discover that the calls were taking 0.04 – 0.05 seconds, on the calling thread, just to get started.  After that, the player went along asynchronously, as expected.  Since my simulation loop was running in increments of 0.0167 seconds, having each call to [myAVAudioPlayer play] block for 3x that amount made for a very unhappy user experience.  My next thought was to bounce right to OpenAL, but I decided that the (apparently) no longer sanctioned (at least it’s not available from Apple anymore) SoundEngine might give me most of what I needed, without my having to worry about managing buffers and sources.

Unfortunately, SoundEngine lacked some OpenAL functionality that I really needed (being able to find out if a given effect is currently playing and also to find out how far along the effect is).  I ended up implementing extensions to SoundEngine to get at the OpenAL stuff, and, so far, everything is working.  The most recent discovery was that the SoundEngine/OpenAL combination does not initialize and properly configure an AudioSession.  This means that our app stopped when the device went into Auto-Lock.  Not necessarily a good thing for a “lifestyle” app.  Once I took care of that, I’m back to calibrating the sound levels and doing more testing on the functionality.  I constantly have my fingers crossed that I don’t encounter any more problems that will force me to abandon the extended SoundEngine.

I may do a longer post on the perils of using compressed audio formats in the future, but let’s wrap this bit up with a warning.  While compression can really help with your download times, you may want to be careful about choosing when to employ compressed formats.  There are a couple of reasons for this.  First, some of the iPhone SDK playback options support only a limited subset of the compressed formats.  Second, the device won’t play the highly compressed formats simultaneously.  If you plan to play multiple sounds at the same time, highly compressed formats might not be an option.

Most of the points covered here are well documented in the Getting Started with Audio & Video document.  If I’d had the discipline to read the whole thing before I started down this path, the trip would have been much less frustrating.

Technorati Tags:  iPhone, iPod Touch, Apple, mobile