iOS SDK: SpriteKit sound

Xcode_Icon

Problem

Recently, I come across a SpriteKit sound bug (iOS 7.0.4) described on stackoverflow.com: Sprite Kit & playing sound leads to app termination. My further investigation and developed solutions answer to this and more issues:

  1. SpriteKit app using [SKAction playSoundFileNamed:] cause crash on app exit or going into background.
  2. Deactivating AVAudioSession in AppDelegate applicationDidEnterBackground: fails with error (no deactivation in effect): Error Domain=NSOSStatusErrorDomain Code=560030580 "The operation couldn’t be completed. (OSStatus error 560030580.)
  3. SpriteKit app using [SKAction playSoundFileNamed:] on AVAudioSession interruption (e.g. call arrived) cannot be reactivated (silence).
  4. How to implement AVAudioPlayer in place of SpriteKit [SKAction playSoundFileNamed:]?

Solution

Sooner or later bugs above should be fixed but, anyway, template presented here is divided into 2 sections that:

  1. Should be used to proper handling AVAudioSession
    or/and
  2. can be used to implement AVAudioPlayer in SpriteKit.

1. AVAudioSession management

In short, all comes down to:

  • Activating and deactivating AVAudioSession throughout app lifecycle in AppDelegate.
  • Handling audio interruptions from other apps.

I’m presenting implementation in AppDelegate but it can be done anywhere else (e.g in your own audio handler singleton).

YourAppDelegate.h file

#import <UIKit/UIKit.h>

@interface YourAppDelegate : UIResponder

@property (strong, nonatomic) UIWindow *window;

+ (BOOL)isAudioSessionActive; // Informs if SpriteKit should play sounds (SpriteKit BUG)
+ (BOOL)isOtherAudioPlaying; // Informs if other app makes sounds

@end
YourAppDelegate.m file

#import "YourAppDelegate.h"
#import <AVFoundation/AVFoundation.h>

@implementation YourAppDelegate

#pragma mark - AppDelegate methods

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
    NSLog(@"%s", __FUNCTION__);
    [self startAudio];
    return YES;
}

- (void)applicationWillResignActive:(UIApplication *)application {
    NSLog(@"%s", __FUNCTION__);    
}

- (void)applicationDidEnterBackground:(UIApplication *)application {
    // SpriteKit uses AVAudioSession for [SKAction playSoundFileNamed:]
    // AVAudioSession cannot be active while the application is in the background,
    // so we have to stop it when going in to background
    // and reactivate it when entering foreground.
    NSLog(@"%s", __FUNCTION__);    
    [self deactivateAudioSession];
}

- (void)applicationWillEnterForeground:(UIApplication *)application {
    // Reactivate AVAudioSession
    NSLog(@"%s", __FUNCTION__);    
    [self activateAudioSession];
}

- (void)applicationDidBecomeActive:(UIApplication *)application {
    NSLog(@"%s", __FUNCTION__);    
}

- (void)applicationWillTerminate:(UIApplication *)application {
    NSLog(@"%s", __FUNCTION__);
    // Close AVAudioSession
    [self stopAudio];
}

#pragma mark - AVAudioSession methods

// Flag that informs if SpriteKit should play sounds
static BOOL isAudioSessionActive = NO;

- (void)startAudio {
    AVAudioSession *audioSession = [AVAudioSession sharedInstance];
    NSError *error = nil;

    NSLog(@"%s isOtherAudioPlaying: %d, oldCategory: %@ withOptions: %d", __FUNCTION__, audioSession.otherAudioPlaying, audioSession.category, audioSession.categoryOptions);

    [audioSession setCategory:AVAudioSessionCategoryAmbient error:&error];

    if (!error) {
        [self activateAudioSession];
    } else {
        NSLog(@"%s setCategory Error: %@", __FUNCTION__, error);
    }

    if (isAudioSessionActive) {
        [self observeAudioSessionNotifications:YES];
    }
}

// Class method that informs if other app(s) makes sounds
+ (BOOL)isOtherAudioPlaying {
    AVAudioSession *audioSession = [AVAudioSession sharedInstance];
    return audioSession.otherAudioPlaying;
}

- (void)activateAudioSession {
    AVAudioSession *audioSession = [AVAudioSession sharedInstance];
    NSError *error = nil;

    [audioSession setActive:YES error:&error];

    NSLog(@"%s [Main:%d] isActive: %d, isOtherAudioPlaying: %d, AVAudioSession Error: %@", __FUNCTION__, [NSThread isMainThread], isAudioSessionActive, audioSession.isOtherAudioPlaying, error);

    if (error) {
        // It's not enough to setActive:YES
        // We have to deactivate it effectively (without that error),
        // so try again (and again... until success).
        isAudioSessionActive = NO;
        [self activateAudioSession];
        return;
    }

    if (!error) {
        // We have to set this flag at the end of activation attempt to avoid playing any sound before.
        isAudioSessionActive = YES;
    } else {
        // Activation failure
        isAudioSessionActive = NO;
    }

    NSLog(@"%s isActive: %d, AVAudioSession Activated with category: %@ Error: %@", __FUNCTION__, isAudioSessionActive, [audioSession category], error);
}

// Informs if SpriteKit should play sounds (SpriteKit BUG)
+ (BOOL)isAudioSessionActive {
    return isAudioSessionActive;
}

- (void)stopAudio {
    if (!isAudioSessionActive) {
        // Prevent background apps from duplicate entering if terminating an app.
        return;
    }

    // Start deactivation process
    [self deactivateAudioSession];

    // Remove observers
    [self observeAudioSessionNotifications:NO];
}

- (void)deactivateAudioSession {
    if (isAudioSessionActive) {
        // We have to set this flag before any deactivation attempt to avoid trying playing any sound underway.
        isAudioSessionActive = NO;
    }

    AVAudioSession *audioSession = [AVAudioSession sharedInstance];
    NSError *error = nil;

    //[audioSession setActive:NO error:&error];
    [audioSession setActive:NO withOptions:AVAudioSessionSetActiveOptionNotifyOthersOnDeactivation error:&error];

    NSLog(@"%s isActive: %d, AVAudioSession Error: %@", __FUNCTION__, isAudioSessionActive, error);

    if (error) {
        // It's not enough to setActive:NO
        // We have to deactivate it effectively (without that error),
        // so try again (and again... until success).
        [self deactivateAudioSession];
        return;
    } else {
        // Success
    }
}

- (void)observeAudioSessionNotifications:(BOOL)observe {
    NSLog(@"%s YES: %d", __FUNCTION__, observe);

    AVAudioSession *audioSession = [AVAudioSession sharedInstance];
    NSNotificationCenter *center = [NSNotificationCenter defaultCenter];

    if (observe) {
        [center addObserver:self selector:@selector(handleAudioSessionInterruption:) name:AVAudioSessionInterruptionNotification object:audioSession];
        [center addObserver:self selector:@selector(handleAudioSessionRouteChange:) name:AVAudioSessionRouteChangeNotification object:audioSession];
        [center addObserver:self selector:@selector(handleAudioSessionMediaServicesWereLost:) name:AVAudioSessionMediaServicesWereLostNotification object:audioSession];
        [center addObserver:self selector:@selector(handleAudioSessionMediaServicesWereReset:) name:AVAudioSessionMediaServicesWereResetNotification object:audioSession];
    } else {
        [center removeObserver:self name:AVAudioSessionInterruptionNotification object:audioSession];
        [center removeObserver:self name:AVAudioSessionRouteChangeNotification object:audioSession];
        [center removeObserver:self name:AVAudioSessionMediaServicesWereLostNotification object:audioSession];
        [center removeObserver:self name:AVAudioSessionMediaServicesWereResetNotification object:audioSession];
    }
}

- (void)handleAudioSessionInterruption:(NSNotification *)notification {
    AVAudioSession *audioSession = (AVAudioSession *)notification.object;

    AVAudioSessionInterruptionType interruptionType =
        (AVAudioSessionInterruptionType)[[notification.userInfo objectForKey:AVAudioSessionInterruptionTypeKey] unsignedIntegerValue];

    AVAudioSessionInterruptionOptions interruptionOption =
        (AVAudioSessionInterruptionOptions)[[notification.userInfo objectForKey:AVAudioSessionInterruptionOptionKey] unsignedIntegerValue];

    BOOL isAppActive = ([[UIApplication sharedApplication] applicationState] == UIApplicationStateActive)?YES:NO;

    switch (interruptionType) {
        case AVAudioSessionInterruptionTypeBegan: {
            [self deactivateAudioSession];
            break;
        }

        case AVAudioSessionInterruptionTypeEnded: {
            [self activateAudioSession];        
            if (interruptionOption == AVAudioSessionInterruptionOptionShouldResume) {
                // Do your resume routine
            }
            break;
        }

        default:
            break;
    }

    NSLog(@"%s [Main:%d] [Active: %d] AVAudioSession Interruption: %@ withInfo: %@", __FUNCTION__, [NSThread isMainThread], isAppActive, notification.object, notification.userInfo);
}

- (void)handleAudioSessionRouteChange:(NSNotification*)notification {

    AVAudioSessionRouteChangeReason routeChangeReason =
    (AVAudioSessionRouteChangeReason)[[notification.userInfo objectForKey:AVAudioSessionRouteChangeReasonKey] unsignedIntegerValue];

    AVAudioSessionRouteDescription *routeChangePreviousRoute =
    (AVAudioSessionRouteDescription *)[notification.userInfo objectForKey:AVAudioSessionRouteChangePreviousRouteKey];

    NSLog(@"%s routeChangePreviousRoute: %@", __FUNCTION__, routeChangePreviousRoute);

    switch (routeChangeReason) {
        case AVAudioSessionRouteChangeReasonUnknown:
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonUnknown", __FUNCTION__);
            break;

        case AVAudioSessionRouteChangeReasonNewDeviceAvailable:
            // e.g. a headset was added or removed
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonNewDeviceAvailable", __FUNCTION__);
            break;

        case AVAudioSessionRouteChangeReasonOldDeviceUnavailable:
            // e.g. a headset was added or removed
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonOldDeviceUnavailable", __FUNCTION__);
            break;

        case AVAudioSessionRouteChangeReasonCategoryChange:
            // called at start - also when other audio wants to play
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonCategoryChange", __FUNCTION__);
            break;

        case AVAudioSessionRouteChangeReasonOverride:
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonOverride", __FUNCTION__);
            break;

        case AVAudioSessionRouteChangeReasonWakeFromSleep:
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonWakeFromSleep", __FUNCTION__);
            break;

        case AVAudioSessionRouteChangeReasonNoSuitableRouteForCategory:
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonNoSuitableRouteForCategory", __FUNCTION__);
            break;

        case AVAudioSessionRouteChangeReasonRouteConfigurationChange:
            NSLog(@"%s routeChangeReason: AVAudioSessionRouteChangeReasonRouteConfigurationChange", __FUNCTION__);
            break;

        default:
            break;
    }
}

-(void)handleAudioSessionMediaServicesWereReset:(NSNotification *)notification {
    NSLog(@"%s [Main:%d] Object: %@ withInfo: %@", __FUNCTION__, [NSThread isMainThread], notification.object, notification.userInfo);
}

-(void)handleAudioSessionMediaServicesWereLost:(NSNotification *)notification {
    NSLog(@"%s [Main:%d] Object: %@ withInfo: %@", __FUNCTION__, [NSThread isMainThread], notification.object, notification.userInfo);
}

@end

2. AVAudioPlayer in SpriteKit

Typical sound handling in SpriteKit

Usually we would handle sounds in SpriteKit in two simple steps:
1. Preloading sounds for instant access – most convenient way seems to create [SKAction playSoundFileNamed:] instances at the beginning like this:

// Somewhere in your SKNode subclass implementation of your sprite that makes sounds...

#pragma mark - Shared Assets

// Method for preloading assets concurrently to speedup start
+ (void)loadSharedAssets {
    [super loadSharedAssets];

    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
         sSharedSomeSoundAction = [SKAction playSoundFileNamed:@"yoursoundfile.aif" waitForCompletion:NO];
   });
}

static SKAction *sSharedsomeSoundAction = nil;
- (SKAction *)someSoundAction {
    return sSharedSomeSoundAction;
}

2. Then play sound actions with message [self runAction:[self someSoundAction] like this:

// Somewhere in the same SKNode subclass implementation of your sprite that makes sounds...

#pragma mark - Physics Methods

- (void)collidedWith:(SKPhysicsBody *)other atPoint:(CGPoint)contactPoint {
    if (other.categoryBitMask & DefinedColliderType1 || other.categoryBitMask & DefinedColliderType2) {
        //NSLog(@"Sprite collided with sprite of DefinedColliderType1 or DefinedColliderType2");
        [self runAction:[self someSoundAction]];
    }
}
An AVAudioPlayer way

As mentioned at the beginning I was forced by current bug to make workaround, which is in fact just another way of playing sounds with in SpriteKit (that’s why it’s here). I wanted to have exact same pattern of use as that above, so I decided to make SKAction category with following assumptions:

  • sound will be played with the same [self runAction:[self someSoundAction]] message,
  • instead of SKAction, an NSData was chosen as sound container in memory (it’s faster creation of AVAudioPlayer with NSData than with file URL),
  • each sound action will have its own temporary AVAudioPlayer instance for sound lifetime only to support multiple sounds at once of the same sound file.
  • skipped waitForCompletion: flag for simplicity sake

The use of this solution can look like this:

YourSprite.h file

#import <SpriteKit/SpriteKit.h>
#import "SKAction+ATWSound.h"

// Bitmask for the different entities with physics bodies.
typedef enum : uint8_t {
    DefinedColliderType1            = 1,
    DefinedColliderType2            = 2,
    DefinedColliderType3            = 4,
    DefinedTypeEmpty1               = 8,
    DefinedTypeEmpty2               = 16
} DefinedColliderType;

@interface YourSprite : SKSpriteNode

// Preload shared animation frames, emitters, etc.
+ (void)loadSharedAssets;

// Overridden methods.
- (void)collidedWith:(SKPhysicsBody *)other atPoint:(CGPoint)contactPoint;

// Any other methods...

@end
YourSprite.m file

#import "YourSprite.h"

// Somewhere in your SKNode subclass implementation of your sprite that makes sounds...

#pragma mark - Shared Assets

// Method for preloading assets concurrently to speedup start
+ (void)loadSharedAssets {
    [super loadSharedAssets];

    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
         sSharedSomeSoundData = [SKAction atwDataFromSoundFileNamed:@"yoursoundfile.aif"];
   });
}

static NSData __strong *sSharedSomeSoundData = nil;
- (SKAction *)someSoundAction {
    return [SKAction atwPlaySoundWithData:sSharedSomeSoundData];
}

// Somewhere else in the same SKNode subclass implementation of your sprite that makes sounds...

#pragma mark - Physics Methods

- (void)collidedWith:(SKPhysicsBody *)other atPoint:(CGPoint)contactPoint {
    if (other.categoryBitMask & DefinedColliderType1 || other.categoryBitMask & DefinedColliderType2) {
        //NSLog(@"Sprite collided with sprite of DefinedColliderType1 or DefinedColliderType2");
        [self runAction:[self someSoundAction]];
    }
}

The SKAction+ATWSound category behind the scene:

SKAction+ATWSound.h file

#import <SpriteKit/SpriteKit.h>
#import <AVFoundation/AVFoundation.h>

@interface SKAction (ATWSound)

// Helper method to create NSData from sound file
+ (NSData *)atwDataFromSoundFileNamed:(NSString *)fileNameWithExtention;

// Method of creating sound action
+ (SKAction *)atwPlaySoundWithData:(NSData *)soundData;

@end
SKAction+ATWSound.m file

#import "SKAction+ATWSound.h"
#import "YourAppDelegate.h"

@implementation SKAction (ATWSound)

+ (NSData *)atwDataFromSoundFileNamed:(NSString *)fileNameWithExtention {
    if (fileNameWithExtention == nil) return nil;

    NSData *soundData = nil;
    NSString *soundFile = [[NSBundle mainBundle] pathForResource:fileNameWithExtention ofType:nil];

    NSAssert(soundFile, @"No such file in mainBundle: %@", fileNameWithExtention);
    soundData = [[NSData alloc] initWithContentsOfFile:soundFile];

    return soundData;
}

static NSMutableSet *sATWAudioPlayersCache; // It's our cache
static dispatch_queue_t sATWAudioPlayersCacheQueue; // It's our concurrent queue of audioPlayers that are playing

+ (void)atwCacheAudioPlayer:(AVAudioPlayer *)audioPlayer {

    // Init cache for the first time
    if (sATWAudioPlayersCache == nil) {
        sATWAudioPlayersCache = [[NSMutableSet alloc] initWithCapacity:1];
        sATWAudioPlayersCacheQueue = dispatch_queue_create("com.AppThatWorks.SKAction+ATWSound.sATWAudioPlayersCacheQueue", DISPATCH_QUEUE_SERIAL);
    }
    
    if (audioPlayer == nil) return;
    
    // Write audioPlayer to concurrently modified cache with barrier to ensure exclusive access to the cache while the block runs.
    // Not only does it exclude all other writes to the cache while it runs, but it also excludes all other reads, making the modification safe.
    // We can skip barriers if our queue is serial (dispatch_queue_create("queue.name", DISPATCH_QUEUE_SERIAL)).
    // More about barriers in a concurrent env: https://www.mikeash.com/pyblog/friday-qa-2011-10-14-whats-new-in-gcd.html
    dispatch_barrier_async(sATWAudioPlayersCacheQueue, ^{
        [sATWAudioPlayersCache addObject:audioPlayer];
    });

    // Set delayed cache clear event for that audioPlayer (with barrier also to ensure exclusive access)
    double delay = [audioPlayer duration]+0.1; // +0.1 second buffer for safe sound end; adjust it/decrease if it takes too much memory for you
    dispatch_time_t popTime = dispatch_time(DISPATCH_TIME_NOW, (int64_t)(delay * NSEC_PER_SEC));
    dispatch_after(popTime, sATWAudioPlayersCacheQueue, ^(void){
        dispatch_barrier_async(sATWAudioPlayersCacheQueue, ^{
            // For safety sake, stop audioPlayer to avoid potential bad access while removing/autoreleasing.
            // It could happen if delay was nearly equal to [audioPlayer duration] (i.e. released while stopping).
            [audioPlayer stop];
            [sATWAudioPlayersCache removeObject:audioPlayer];
            NSLog(@"%s Removed: %@ , Cache: %lu", __PRETTY_FUNCTION__, audioPlayer, (unsigned long)[sATWAudioPlayersCache count]);
        });
    });
}

+ (SKAction *)atwPlaySoundWithData:(NSData *)soundData {
    // If AVAudioSession is inactive do not make any sound
    if ([BAPAppDelegate isAudioSessionActive] == NO) {
        return [SKAction runBlock:^{}];
    } else {
        // Create playSoundAction which starts on concurrent queue to gain speed
        return [SKAction runBlock:^{
            NSError *error;
            AVAudioPlayer *audioPlayer = [[AVAudioPlayer alloc] initWithData:soundData error:&error];
            
            if (!error) {
                audioPlayer.numberOfLoops = 0;
                [audioPlayer play];

                // Cache audioPlayer to avoid autoreleasing before play end
                [SKAction atwCacheAudioPlayer:audioPlayer];
            }
           NSLog(@"%s  Added: %@ withError: %@, Cache: %lu", __PRETTY_FUNCTION__, audioPlayer, error, (unsigned long)[sATWAudioPlayersCache count]);
        } queue:sATWAudioPlayersCacheQueue];
    }
}

@end

Discussion 4 komentarze

  1. Hi,
    I’ve tested this solution but it seems to produce some interface lag each time the sound is played….
    I think that we should keep a cache of ready to use avplayer for each sound that needs to be repeated to prevent this…

    • I have never experienced UI lags (everything is done aside main thread). I found that restarting audioplayer takes almost the same time that instantiating new one from NSData. Additionally, we don’t know when audioplayer will be needed next time and it will consume precious mem for nothing. Take also a look that sound (NSData) should be shared (loaded only once) outside of this category – maybe you are reading it every time from file insted of sharing sound between audioplayers?

  2. A huge thanks! for sharing this. I spent a while testing all kinds of solutions for an alternative to SKAction playSoundFileNamed due to the bug where sound is disabled for the game after a phone call or alarm. Also for the lack of volume control.
    The problem was that everything I tried besides the normal SKAction way caused performance lags. But
    your beautiful and well written category solution gives me awesome performance, every bit as good as the highly optimised stock SK playSoundFileNamed and I did not have to change the calls to play sounds throughout the code.
    Thank again

Give your feedback