If you look up “Spectral Audio Processing” it’s possible to feel overwhelmed by mathematical formulas, lot of technical jargon, and deep dives into something called the Fourier Transformation or its speed-optimised cousin, the FFT. While that is all very relevant for a DSP developer, for artists and sound designers who want to create something a high level description of how it all works is sufficient in order to have a proper understanding of what Spectral Audio Processing can do for you. In order to achieve that in this blog post I will explain the following:

  • What is Spectral Analysis / Resynthesis about?
  • Which parameters are available in Spectral Audio Processing?
  • What kind of effects can I create using Spectral Audio Processing? (3 examples)

This is where the Magic happens

There are two steps involved in every spectral audio effect: Analysis and Resynthesis. While these are important steps which control the overall detail and quality, we are not getting into that. Instead, we are going to replace those with a black box - or in our case a magic hat.

The relevant part for you as a creator is what kind of information you get from an analysis, not how it works. So, let’s start by stuffing some audio into our magic hat, say “Abracadabra” and observe what we pull out:

Magic Hat Transformation

You might have expected a white rabbit, but we got something even better! The analysis transforms our audio signal into information about its spectral content. It had split the audio signal into its components and the components are sine waves with a specific amplitude and frequency. If you add up all those sine waves you end up with the original audio signal again! I.e. resynthesis!

In fact, any signal can be decomposed into its sinusoidal components. If you don’t believe it you can read about Fourier Series on Wikipedia , but the diagram below should give you a visual intuition of how this works:

Original

The Input signal is split into its components (analysis). In this case 3 sine waves. The 3 sine waves are summed up for the Output (resynthesis). Note that the Output is identical to the Input.

But hold on, doesn’t audio change over time? How can I get the same signal back by mixing fixed sine waves? The truth is we simplified things a bit in the picture. The number of sine waves as well as their amplitudes and frequencies change over time as well. So in reality for each point in time we get a set of sine waves which if added together would result in the value of the audio signal at that specific point of time. Mathematicians call this sort of operation a transformation. Which is, simply put, a different representation of the same thing. But we can ignore that detail for now.

The Magic Chef

Let’s create a metaphor: When you eat something you are able to tell which ingredients were used. At least you can tell if it’s fish, meat, or vegetables, and hopefully some spices. Your tongue is also an analyser. A pretty good one actually! Now imagine you could put any dish into a magic hat and you get out all the ingredients that were used to make it. The dream of every chef who wants to copy a recipe has come true!

Sound works differently from food of course and copying an audio file never has never been simpler using a computer. We need no magic for that. Also, artists and sound designers don’t want to copy other people’s work. They want to come up with something new and unique! So how does the analysis help you? Coming back to the metaphor above, you could change something in the ingredients to create a new variation or even a completely different dish. By having access to the ingredients you could change the balance between them, completely remove one of them, or even add ingredients that weren’t there in the original dish. You’d still need to cook it, but that’s just as easy as getting the ingredients: just put them into the magic hat, say “Abracadabra” again and pull out your new dish! In a similar way you can treat your audio signal: Analyse it, do something with the components, and finally resynthesise the altered components to produce a new audio signal!

So what can we actually do with the components? We have information about amplitude and frequency over time. These are the parameters we can change. Let’s say you make a rule: decrease the amplitude if the frequency is high but leave it unchanged if the frequency is low. This would create a low-pass filter:

Low-Pass

Low-Pass Filter

Or how about we multiply all the components' frequencies by 2, leaving everything else unchanged. This will shift the pitch up by 1 Octave but won’t change the tempo (remember, you didn’t change time):

Pitch Shift

Pitch Shifting

In the case of a spectral delay, such as our MagicDelay , it’s about delaying components depending on their frequency. That’s what happens when you’re altering the curve in the MagicDelay user interface, you are setting a delay per frequency - like a magic chef!

Delay

Spectral Delay

Conclusion

You should now have a better understanding of what Spectral Audio Processing can do (and what it probably can’t). You might already be thinking about new ways to alter components which no one ever thought of! At least I hope you now have a better conceptual model for what is going on in Spectral Audio Processing.

When working with tools like our plugins these high-level concepts are much more important than understanding mathematical concepts like the Fourier Transformation. Let us do the magic for you so you can focus on interesting new sounds!

3Blue1Brown Fourier Transformation on Youtube

Spectral Audio Signal Processing by Julius O. Smith III