Making a Survivors-like with Latios Framework Part 5 : SFX

Goals:

Let’s add some footsteps SFX for the Player as a learning experience.
Refactor the SFX system to be more efficient (or not).
Add some SFX to the axe prefab.
Add some SFX to the enemies when they are hit.
Add some VFX to the axe.
Add some Shader Graphs effects to the Skeletons.

Adding footsteps SFX to the Player

Picking the SFXs

I’ve looked around on FreeSound and didn’t find any simple footsteps SFXs, only very specific-ones.

Then, I remembered that Kenney had some SFX packs! I picked the RPG Audio Pack which contains a set of standard footsteps SFXs and Impact Sounds that contains some more footsteps and other sounds.

Adding One-Shot SFXs to the game

Latios Framework has an ECS Audio module, Myri Audio, that is pretty simple to use. It is lacking built-in common effects like “pitch shift” or “reverb”.

As Myri’s documentation states, it uses Unity’s DSP Graph but:

The underlying DSPGraph is not exposed in any way. A solution to this problem is coming soon in the form of Effect Stacks, which will overhaul Myri and add a much more customizable and controllable API surface.

I’m starting with the KISS route, making a prefab for each footstep SFX.

Six similar prefabs for the footsteps SFX

Next comes the boring part where I’m adding events to the animations, painfully trying to put events at the right spot.

Animation events for the footsteps SFX

I’m setting the event’s Int property to 1 as I’m going to use a quick and dirty enum to identify event types.

public enum ESfxEventType
{
    Footstep = 1
    // TODO: Add more event types
}

Then, I’m creating a new Authoring MonoBehaviour that will create a DynamicBuffer<FootstepBufferElement> from a list of footstep SFX prefabs.

namespace Survivors.Play.Authoring.Player.SFX
{
    public class PlayerFootstepsAuthoring : MonoBehaviour
    {
        [SerializeField] List<GameObject> footstepPrefabs;
        [SerializeField] ESfxEventType eventType;

        class PlayerFootstepsAuthoringBaker : Baker<PlayerFootstepsAuthoring>
        {
            public override void Bake(PlayerFootstepsAuthoring authoring)
            {
                var entity = GetEntity(TransformUsageFlags.Dynamic);
                var buffer = AddBuffer<FootstepBufferElement>(entity);

                foreach (var footstepPrefab in authoring.footstepPrefabs)
                {
                    var prefab = GetEntity(footstepPrefab, TransformUsageFlags.Dynamic);
                    buffer.Add(new FootstepBufferElement
                    {
                        FootstepPrefab = prefab
                    });
                }
            }
        }
    }


    public struct FootstepBufferElement : IBufferElementData
    {
        public EntityWith<Prefab> FootstepPrefab;
    }
}

Simple

Sending the SFX events

In a similar fashion to the Part 3, I’ll send the events to an Sfx Spawn Queue.

Step one : Modifying ClipState to be able to track animation clips’ weights

    /// <summary>
    ///     The state of a clip.
    /// </summary>
    public struct ClipState
    {
        public float PreviousTime;
        public float Time;
        public float SpeedMultiplier;
        public int EventHash;
        
        public float CurrentWeight;                                 

        public void Update(float deltaTime)
        {
            PreviousTime =  Time;
            Time         += deltaTime;
        }
    }

Step two : Actually keep track of the weights in FourDirectionsAnimationSystem

    void UpdateClipState(ref ClipState state,
        ref SkeletonClip clip,
        float deltaTime,
        float weight)
    {
        state.CurrentWeight = weight;                               
        if (weight > math.EPSILON)
        {
            state.Update(deltaTime * state.SpeedMultiplier);
            state.Time = clip.LoopToClipTime(state.Time);
        }
    }

Step three : Create a new system that runs after the FourDirectionsAnimationSystem and sends the events to the Sfx Spawn Queue (to avoid cluttering the FourDirectionsAnimationSystem)

AnimationEventsJob

    [BurstCompile]
    partial struct AnimationEventsJob : IJobEntity, IJobEntityChunkBeginEnd
    {
        [NativeDisableParallelForRestriction] public NativeQueue<SfxSpawnQueue.SfxSpawnData> SfxQueue;
        public SystemRng Rng;

        void Execute(
            in FourDirectionClipStates clipStates,
            in Clips clips,
            in WorldTransform worldTransform,
            DynamicBuffer<FootstepBufferElement> footstepsPrefabs)
        {

            var clipStatesArray = new NativeArray<ClipState>(5, Allocator.Temp);
            clipStatesArray[(int)EDirections.Center] = clipStates.Center;
            clipStatesArray[(int)EDirections.Up]     = clipStates.Up;
            clipStatesArray[(int)EDirections.Down]   = clipStates.Down;
            clipStatesArray[(int)EDirections.Left]   = clipStates.Left;
            clipStatesArray[(int)EDirections.Right]  = clipStates.Right;

            // Find the heaviest clip state

            var heaviestIndex = 0;

            for (var i = 0; i < clipStatesArray.Length; i++)
            {
                var state = clipStatesArray[i];

                if (state.CurrentWeight > clipStatesArray[heaviestIndex].CurrentWeight)
                    heaviestIndex = i;
            }

            var heaviestState = clipStatesArray[heaviestIndex];


            ref var clip = ref clips.ClipSet.Value.clips[heaviestIndex];

            // Check if the clip has any events in the range of PreviousTime and Time
            if (!clip.events.TryGetEventsRange(heaviestState.PreviousTime, heaviestState.Time, out var index,
                    out var count)) return;

            // -1 means no events in the range
            if (index == -1) return;

            for (var j = index; j < index + count; j++)
            {
                var evt = clip.events.parameters[j];

                // Check if the event is a footstep event
                if (evt != (int)ESfxEventType.Footstep) continue;

                var prefabIndex = Rng.NextInt(0, footstepsPrefabs.Length);

                // Queue the event to be spawned
                SfxQueue.Enqueue(new SfxSpawnQueue.SfxSpawnData
                {
                    EventHash = evt, // unused for now
                    Position  = worldTransform.position,
                    SfxPrefab = footstepsPrefabs[prefabIndex].FootstepPrefab
                });
            }
        }
    // [...]
    }

Step four : Create a new system SfxSpawnQueueSystem that will spawn the SFXs from the queue.

SfxSpawnQueueSystem.cs

namespace Survivors.Play.Systems.SFX
{
    [BurstCompile]
    public partial struct SfxSpawnQueueSystem : ISystem
    {
        LatiosWorldUnmanaged m_worldUnmanaged;

        [BurstCompile]
        public void OnCreate(ref SystemState state)
        {
            m_worldUnmanaged = state.GetLatiosWorldUnmanaged();
        }

        [BurstCompile]
        public void OnUpdate(ref SystemState state)
        {
            var sfxQueue = m_worldUnmanaged.sceneBlackboardEntity.GetCollectionComponent<SfxSpawnQueue>().SfxQueue;

            if (sfxQueue.IsEmpty()) return;

            var icb = m_worldUnmanaged.syncPoint.CreateInstantiateCommandBuffer<WorldTransform>();

            state.Dependency = new SfxSpawnJob
            {
                SfxQueue         = sfxQueue,
                SpawnQueueWriter = icb.AsParallelWriter()
            }.Schedule(state.Dependency);
        }


        [BurstCompile]
        struct SfxSpawnJob : IJob
        {
            public NativeQueue<SfxSpawnQueue.SfxSpawnData> SfxQueue;
            public InstantiateCommandBuffer<WorldTransform>.ParallelWriter SpawnQueueWriter;

            public void Execute()
            {
                var sortKey = 0;

                while (!SfxQueue.IsEmpty())
                    if (SfxQueue.TryDequeue(out var data))
                    {
                        var transform = new WorldTransform
                        {
                            worldTransform = TransformQvvs.identity
                        };

                        transform.worldTransform.position = data.Position;

                        SpawnQueueWriter.Add(
                            data.SfxPrefab, transform, ++sortKey);
                    }
            }
        }
    }
}

Well, that’s a pretty generic system for SFX events that spawns at a position. Nice.

Result

Intuition

While adding an SFX list to the axe prefab, I had the intuition that it wouldn’t be very efficient.

Let’s say we have a super-skill that spawns a thousand axes, each axe will allocate a new Buffer but we need only one SFX played per axe.

I asked about my intuition on the Latios Discord. Since, sometimes, English is hard, I came up with this quick image to ask if I was right.

My intuition was right. Even better, Dreaming gave refactoring work to a lot of people in the community by telling us that “this (spawning entities with dynamic buffers) was one of the few use case where having InternalBufferCapacitylarger than 0 can make a lot of sense”.

Indeed, the default size for each buffer on an entity is 128 bytes!

WTF

Refactoring

At first, I went down the road of having entities reference some SFX prefabs that are holding a buffer of SFX prefabs. Then, I found that this setup was not well suited for entities that spawned different kinds of SFXs “events”. It became quite cumbersome to pass a bunch of ComponentLookups and BufferLookups to jobs.

It started to look like a brain fart and I started to get really confused when triggering an event from an entity that had a reference to an entity that had a reference to an entity that had a list of references…

So, I’ll try again, keep it simple, set the InternalBufferCapacity to a low number and see if it is an issue (performance-wise) later.

Axe Swoosh SFX

Most entities that will send an SFX event to the queue will have a similar authoring behaviour.

namespace Survivors.Play.Authoring.Player.Weapons
{
    [DisallowMultipleComponent]
    [AddComponentMenu("Survivors/Player/Weapon/AxeSwoosh")]
    public class AxeSwooshAuthoring : MonoBehaviour
    {
        [SerializeField] List<GameObject> swooshPrefabs;
        [SerializeField] ESfxEventType    eventType;

        class AxeSwooshAuthoringBaker : Baker<AxeSwooshAuthoring>
        {
            public override void Bake(AxeSwooshAuthoring authoring)
            {
                var entity = GetEntity(TransformUsageFlags.Dynamic);
                var buffer = AddBuffer<AxeSwooshBufferElement>(entity);

                foreach (var swooshPrefab in authoring.swooshPrefabs)
                {
                    var prefab = GetEntity(swooshPrefab, TransformUsageFlags.Dynamic);
                    buffer.Add(new AxeSwooshBufferElement
                    {
                        SwooshPrefab = prefab
                    });
                }
            }
        }
    }

    [InternalBufferCapacity(8)]
    public struct AxeSwooshBufferElement : IBufferElementData
    {
        public EntityWith<Prefab> SwooshPrefab;
    }
}

The AxeSwooshBufferElement is intended to be used when the axe is spawned. So I’m modifying the WeaponThrowTriggerSystem and proceed to pick an SFX in a similar fashion than in FourDirectionsAnimationEventsSystem.

Enemy Hit SFX

When the axe collides with an enemy, it adds a HitInfos component to this enemy and a DeadTag.

The DeadTag is temporary. It’s function is to make the dead entity “pass through” the DisableDeadCollidersSystem and, then, the SkeletonDeathSystem. It will be added elsewhere once the enemies will have and health system implemented.

Since I want the enemies to be able to play some SFX when being hit, I can already create a new system that will handle enemies with the HitInfos component and play the “egg crackling” SFX.

SkeletonHitInfosUpdateSystem.cs

namespace Survivors.Play.Systems.Enemies
{
    public partial struct SkeletonHitInfosUpdateSystem : ISystem, ISystemNewScene
    {
        EntityQuery          m_query;
        LatiosWorldUnmanaged m_world;


        [BurstCompile]
        public void OnCreate(ref SystemState state)
        {
            m_query = state.Fluent()
                .With<WorldTransform>()
                .With<EnemyTag>()
                .WithEnabled<HitInfos>()
                .With<SkeletonHitSfxBufferElement>()
                .Build();

            m_world = state.GetLatiosWorldUnmanaged();
        }

        [BurstCompile]
        public void OnUpdate(ref SystemState state)
        {
            var sfxQueue = m_world.sceneBlackboardEntity.GetCollectionComponent<SfxSpawnQueue>()
                .SfxQueue;

            state.Dependency = new HitInfosUpdateJob
            {
                Rng            = state.GetJobRng(),
                SfxQueue       = sfxQueue.AsParallelWriter(),
                HitInfosLookup = SystemAPI.GetComponentLookup<HitInfos>()
            }.Schedule(m_query, state.Dependency);
        }


        public void OnNewScene(ref SystemState state)
        {
            state.InitSystemRng("SkeletonHitInfosUpdateSystem");
        }

        [BurstCompile]
        partial struct HitInfosUpdateJob : IJobEntity, IJobEntityChunkBeginEnd
        {
            public SystemRng Rng;
            public NativeQueue<SfxSpawnQueue.SfxSpawnData>.ParallelWriter SfxQueue;
            [NativeDisableParallelForRestriction] public ComponentLookup<HitInfos> HitInfosLookup;

            void Execute(
                Entity entity,
                [EntityIndexInQuery] int index,
                in WorldTransform worldTransform,
                in DynamicBuffer<SkeletonHitSfxBufferElement> sfxBuffer)
            {
                var prefabIndex = Rng.NextInt(0, sfxBuffer.Length);
                SfxQueue.Enqueue(new SfxSpawnQueue.SfxSpawnData
                {
                    SfxPrefab = sfxBuffer[prefabIndex].SfxPrefab,
                    Position  = worldTransform.position
                });

                // TODO: Add logic to handle HitInfos component
                
                HitInfosLookup.SetComponentEnabled(entity, false);
            }

            public bool OnChunkBegin(in ArchetypeChunk chunk, int unfilteredChunkIndex, bool useEnabledMask,
                in v128 chunkEnabledMask)
            {
                Rng.BeginChunk(unfilteredChunkIndex);
                return true;
            }

            public void OnChunkEnd(in ArchetypeChunk chunk, int unfilteredChunkIndex, bool useEnabledMask,
                in v128 chunkEnabledMask,
                bool chunkWasExecuted) { }
        }
    }
}

Result

Here’s a video showcasing the footsteps, axe swoosh and enemy hit SFXs in action.

Aside

Did you notice some color differences on the ground tiles? I’m retopologizing the assets as I noticed that the level alone had around 400k vertices. While baking textures, it gave me some slight color differences. The lighter-ones are the original ones.

Adding VFXs

LifeFX

LifeFX is a visual effects and graphics programming module built on top of Kinemation. Currently, it mainly serves as a bridge to Unity VFX Graph. But this bridge can also be leveraged for many other graphics operations.

LifeFX uses a postal system to send events to the GPU.
it broadcasts graphics buffer data generated from the ECS world to the GameObject world.
It comes with out-of-the-box support for Unity VFX Graph.

Adding an Hit VFX to the axe

I wanted to add a VFX when the axe hits and enemy and I’ve found this nice YouTube video from by Gabriel Aguiar.

LifeFX’s documentation give’s a pretty good example of how to setup a simple VFX event : here. It works very well for a VFX Graph containing a single spawn event.

Sample

To integrate the VFX from the video, I had to use trial and error (and my brain, a bit) to make it work since the VFX Graph I ended up with has three spawn events.

I made it work with these steps:

Create 3 sub-graphs (one for each spawn event) and put theme in a main VFX Graph.

For each-subgraph, expose their “spawn count”.

Plug something similar to the next screenshot into each sub-graph.

The code side of things is pretty straightforward. This VFX only needs a position and is a “one-shot” event.

namespace Survivors.VfxTunnels
{
    [CreateAssetMenu(fileName = "New PositionGraphicsEventTunnel",
        menuName = "Survivors/VfxTunnels/PositionGraphicsEventTunnel")]
    public class PositionGraphicsEventTunnel : GraphicsEventTunnel<float3> { }
}

The EventTunnel is a simple float3

public struct OneShotPositionEventSpawner : IComponentData
{
    public UnityObjectRef<PositionGraphicsEventTunnel> PositionGraphicsEventTunnel;
}

The component is added to a prefab and a reference to this prefab is added to the axe

I’ll skip the details on the additional spawn queue system since it is similar to the other spawn queue systems.

Spawning the VFX events inside an IJobEntity works very well!

    [BurstCompile]
    internal partial struct VfxPositionEventSpawnerJob : IJobEntity
    {
        public GraphicsEventPostal                 MailBox;
        public DestroyCommandBuffer.ParallelWriter Dcb;

        void Execute(Entity entity, [EntityIndexInQuery] int idx, in WorldTransform transform,
            ref OneShotPositionEventSpawner spawner)
        {
            var position = transform.worldTransform.position;
            MailBox.Send(position, spawner.PositionGraphicsEventTunnel);
            Dcb.Add(entity, idx);
        }
    }

Feeling under-stimulated

Implementing VFXs and SFXs are not my favorite part of game dev. Those simple examples are nice but boring to implement (since there may be a lot more of them to come).

Don’t be mistaken, though! LifeFX and Myri are super extensible and powerful modules.

I could spend some time learning how to make custom filters for Myri or more complex VFXs (like those that are spawning from a SkinnedMesh’s surface…) but it probably would be way over my current skill level and I’d like to focus more on the game mechanics.

Adding Shader Graphs effects to the Skeletons

Scrolling through the aforementioned YouTube channel, I’ve found this video about making a disintegrate and dissolve effect.

I thought it would be a nice effect to add to the skeletons when they die.

I started by making a copy of the base shader graph I’m using for skinned meshes and followed the video. It gave me a material with an exposed value that can control the amount of dissolve and that looks like this:

Authoring the Dissolve effect

That was surprisingly easy to implement!

Here’s the authoring script and components:

namespace Survivors.Play.Authoring.Materials
{
    public class MaterialDissolveAuthoring : MonoBehaviour
    {
        [SerializeField] float dissolveSpeedMultiplier = .5f;

        class MaterialDissolveAuthoringBaker : Baker<MaterialDissolveAuthoring>
        {
            public override void Bake(MaterialDissolveAuthoring authoring)
            {
                var entity = GetEntity(TransformUsageFlags.Renderable);
                var dissolveAmount = new MaterialDissolveAmount
                {
                    Value = 0.0f
                };

                AddComponent(entity, dissolveAmount);

                AddComponent(entity, new MaterialDissolveSpeed
                {
                    Value = authoring.dissolveSpeedMultiplier
                });
            }
        }
    }


    [MaterialProperty("_Dissolve_Amount")]
    public struct MaterialDissolveAmount : IComponentData
    {
        public float Value;
    }

    public struct MaterialDissolveSpeed : IComponentData
    {
        public float Value;
    }
}

Updating the Skeletons

To keep it simple, I modified the SkeletonDeathSystem to start the dissolve effect when the skeleton’s death animation is finished playing. Additionally, I added a line to destroy the skeleton entity when the dissolve effect is over.

        void Execute(
            Entity entity,
            [EntityIndexInQuery] int idx,
            OptimizedSkeletonAspect skeleton,
            in DeathClips clips,
            ref DeathClipsStates clipStates,
            ref DynamicBuffer<LinkedEntityGroup> linkedEntityGroup)
        {

            // [...]


            // One shot clips
            if (state.Time >= clips.ClipSet.Value.clips[clipStates.ChosenState].duration)
            {

                // Animation is done, start the dissolve effect

                // Clamp to duration
                state.Time = clips.ClipSet.Value.clips[clipStates.ChosenState].duration;
                
                var dissolved = false;

                // apply dissolve to linked entities' materials
                foreach (var entityGroup in linkedEntityGroup)
                {
                    if (!DissolveAmountLookup.HasComponent(entityGroup.Value)) continue;

                    var dissolve = DissolveAmountLookup.GetRefRW(entityGroup.Value);
                    var dissolveSpeed = DissolveSpeedLookup.GetRefRO(entityGroup.Value);
                    dissolve.ValueRW.Value =
                        math.min(dissolve.ValueRW.Value + DeltaTime * dissolveSpeed.ValueRO.Value, 1f);

                    dissolved = (dissolve.ValueRW.Value >= 1f);
                }

                if (dissolved)
                    // Destroy the skeleton entity
                    Dcb.Add(entity, idx);

                return;
            }

            // [...]
        }

Result

Skeletons dissolving

Conclusion

This post took me a while to write. From April 17 to April 29!

Time Flies

I’ve got sidetracked, got some life-related stuff to handle and I had to refactor the SFX system a few times… to end up with a super simple-stupid system 😭.

I hope you enjoyed this post and learned something new about Latios Framework.

Next post will be about… I don’t know yet but it will be about gameplay mechanics because “we’re not game yet”.

Source Code : GitHub