This article outlines a methodology for algorithmic design of repeat patterns through synthesis and processing of audio signals. Vector graphic forms may be synthesized by mapping audio waveforms to coordinates in space. Using a modular architecture of synthesizers and synchronized symmetry operations, this work describes how audio tools can be used to arrange such forms into repeat patterns with emergent visual resonances. Audio effects are inserted between the devices that organize levels of pattern structure, such that the visual results may be explored. While there is a long history of artistic visualization of audio waveforms, they have seldom been used in pattern design. This work aims to highlight the underexplored potential of audio as a pattern generation and manipulation framework.
Visual imagery may be produced with audio by mapping signal values to the x- and y-coordinates of vector graphic display technologies such as oscilloscopes, lasers, plotters, and SVG paths. This practice of synthesizing signals with the goal of producing vector imagery, sometimes called vector synthesis[1] or oscillographics[2], has a rich history in visual music, early computer and video art, and contemporary audiovisual composition and performance. Examples span from the oscilloscope-based pattern designs of C. E. Burnett[3] in the 1930s to Jerobeam Fenderson’s wildly popular “How To Draw Mushrooms On An Oscilloscope With Sound”[4] on YouTube. From the perspective of visual design, techniques of audio synthesis and signal processing provide a wide range of forms and transformations not commonly found in the visual arts, even those with a focus on algorithmic methods. This paper will explore some of these techniques and the process behind their implementation. This work aims to be accessible to practitioners focused on either image, sound, or both, to make available the unique potential of these audiovisual processes.
All vector graphic forms that follow have been synthesized and processed as audio in Max and Ableton Live, then written to SVG in MATLAB. Code examples come from the current implement in Max’s gen~ environment, using the codebox operator.
This work investigates audio as a means of producing repeat patterns as outlined by Paul Jackson[5]. A repeat pattern can be described as an arrangement of visual elements repeated through one or more symmetry operations, which include rotation, translation, reflection, and glide reflection. These operations can be applied at various levels of organization within the pattern. At the lowest level, an element is a singular visual form that serves as the pattern’s basic building block. Symmetry operations can be applied to arrange elements into motifs, which can then be arranged into meta-motifs, and so on to form progressively more complex patterns. A repeat pattern can then be produced by organizing the final arrangement of elements into a tiling, which defines the pattern’s repetition scheme over a plane, independent of the tiles’ content. The following sections present one of many approaches to repeat pattern organization and how it may be implemented algorithmically through audio synthesis and signal processing.
At the heart of the pattern organization processes in this work is a system of hard-synchronized phasors. Phasors are ramp functions that repetitively increment from zero to one over a defined period of time. They can be said to “drive” oscillator functions by supplying them with cyclic input values to produce cyclic output. When multiplied by 2π, the phasor value may represent the phase or angle of a trigonometric function.
Here, phasors and step functions for each level of patterning are synchronized through an approach called hard-sync, where one phasor resets the phase of another every time it completes its cycle. In this implementation, one phasor acts as the master clock, with its frequency dictating what could be considered the frame rate of the pattern. By multiplying this phasor by some number and wrapping between zero and one with a modulo operation, a phasor at a new rate is produced, hard-synchronized to the first. The phasor that drives each level of the pattern process is synchronized to the phasor of the level above it, as shown in the code below.
clock = phasor(freq, sync); phasorTY = fract(clock * tilesY); phasorTX = fract(phasorTY * tilesX); phasorMM = fract(phasorTX * metamotifs); phasorM = fract(phasorMM * motifs); phasorE = fract(phasorM * elements); phasorC = fract(phasorE * cycles); stepTY = floor(clock * tilesY) / tilesY; stepTX = floor(phasorTY * tilesX) / tilesX; stepMM = floor(phasorTX * metamotifs) / metamotifs; stepM = floor(phasorMM * motifs) / motifs; stepE = floor(phasorM * elements) / elements; stepC = floor(phasorE * cycles) / cycles;
This cascade approach is valuable because it maintains stability throughout all levels of the pattern if there are any fractional quantities of pattern elements, motifs, and so on. Such fractional values are useful for animating changes in the pattern, so a motif can gradually sprout new elements, for instance, rather than elements appearing instantaneously. Without hard-sync, forms would cycle throughout the pattern, typically at very chaotic rates due to the typically non-harmonic frequency ratios between pattern levels.
The code above shows a pattern level called “cycles,” which refers to the number of times each element is drawn before progressing to the next. This is particularly useful for producing concentric shapes and spirals through amplitude modulation. The “cycle phasor” is the phasor actually responsible here for synthesizing the forms that are manipulated throughout the successive patterning operations.
The four symmetry operations touched upon here are rotation, translation, reflection, and glide reflection. In the current implementation, they are implemented by using step functions or scaled phasors, sometimes called ramp functions below, as inputs to functions that carry out the symmetry operations themselves.
In rotation symmetry, an element is duplicated through rotation around a central point. For instance, if there are six elements in a motif, then this duplication can occur by synthesizing an element using a phasor cycling six times faster than the step function that rotates it. To produce a circle for illustration, the “cycle phasor” discussed above is multiplied by 2π and sent into cosine and sine functions, which produce the circle’s x- and y-coordinates, respectively. The circle is then translated along the x-axis and rotated to produce the figures below. Rotation is driven by a step function in the first figure and a ramp function in the second, producing an epitrochoidal curve.
To differentiate between the various symmetry operations, asymmetry will be introduced into the pattern element by summing the existing circle with its second harmonic and phase shifting. The previous rotation symmetry operations are illustrated below using this new element.
In translation symmetry, an element is duplicated and shifted in space, with distance and angle remaining constant between iterations. Shown below are radial motifs implemented through translation by the same step function and ramp as before.
In reflection symmetry, one element is a mirror image of another. Since it’s clear what that looks like with two elements, reflection integrated into the previous radial motifs.
In glide reflection symmetry, an element is translated then reflected over a line parallel to its translation. Incorporating this operation into the current implementation is a matter of future work.
Meta-motifs can be produced by carrying out any of these symmetry operations on a motif rather than an individual element. Shown below are meta-motifs produced by step and ramp rotation. While this iterative process could continue indefinitely, we will stop here to explore how these forms can be arranged and tiled into repeat patterns.
Tiling involves organizing geometric shapes such that they cover a surface without overlapping or leaving space between them. In this sense, the current approach adheres to a simple grid system of tiles that can be repeated as many times as desired over the x- and y-axes. To produce repeat patterns that loop at the edges of these tiles, meta-motifs are arranged using ramp and step functions in conjunction with wrapping operations. The code below shows how ramps are generated to distribute the meta-motifs, where phasorTX represents the phasor that cycles each tile.
phasorRamp = phasorTX * metamotifs;
phasorStep = floor(phasorRamp);
phasorMix = mix(phasorStep, phasorRamp, interp);
phasorNorm = phasorMix / metamotifs;
rampX = (phasorScale * scalarX) + phasor(animateX, sync);
rampY = (phasorScale * scalarY) + phasor(animateY, sync);
stepTY = floor(clock * tilesY);
stepTX = floor(phasorTY * tilesX);
x = rampX + offsetX + (stepTX * tileW);
y = rampY - offsetY + (stepTY * tileH);
tileDimX = tilesX * tileW;
tileDimY = tilesY * tileH;
x = wrap(x, 0, tileDimX);
y = wrap(y, 0, tileDimY);
At the end of this process, x and y represent the coordinates of interpolations between ramp and step functions for each tile in the repeat pattern. These functions could be compared to the scanlines used to produce imagery on CRT displays, but with variable slopes, wrapping freely over the x- or y-axes. The scalarX and scalarY variables control both the slope of the functions, their length, and the spacing between meta-motifs distributed along them. Shown below are ramp functions wrapped across a 2 x 2 grids of tiles. Though they only wrap at the edges of the full array, the visual output is identical to wrapping at the edge of each tile due to the nature of this modulo process. The width and height of the tiles are set to 1 and 0.5625, or a 16:9 aspect ratio, which changes the wrap patterning. Before the meta-motifs are even introduced, patterns arise based on alignments in where the functions wrap and end.
Meta-motifs can then be arranged on these paths by summing the scanline coordinates with the input pattern waveforms. Illustrated below are curved meta-motifs summed with step and ramp functions from the images above. Using the atan2 function, meta-motifs are rotated based on the slope of the scanline on which they fall, providing more visual coherence to the image than if only translation occurs.
Different edge conditions can be produced in the repeat pattern by summing the meta-motif waveforms with the scanlines before or after wrapping. Summation after wrapping, as in the previous examples, means that jumps from one edge of a pattern to the other take place only when the scanline function wraps. If scanning is treated as a step function, then the edges of the pattern will include only whole meta-motifs. Summation before wrapping yields fractional meta-motifs with hard edges at the boundaries of the pattern. This may be the desired look in many applications, implying that the pattern continues beyond the frame, however it poses additional complications in practice. For display on an oscilloscope, there is significantly more high-amplitude, high-frequency activity as coordinates jump between opposite edges of the frame within individual meta-motifs. When writing to SVG, new paths need to be initiated when these jumps take place.
It is worth noting that this approach of distributing forms along scaled and wrapped step functions aligns very closely with John Whitney’s concept of differential dynamics[6]. He proposes that patterns of harmonic resonance arise between nodes at fractional intervals along any ascending progression of periodic phenomena. These resonant patterns stem from hierarchies between nodes mapped to polar or Cartesian coordinate systems in a periodic way. These patterns can be very difficult to predict, forming new resonances often with minute changes in parameters, and are particularly compelling when set in motion, continuously aligning and realigning. In the current work, these processes occur because changes in scale of the scanline function cause each element to move at a different rate based on its step in the progression. By wrapping this function within the pattern tiling, its mapping becomes periodic to allow such patterns to emerge. Shown below are two event patterns separated by only a small change in scanline scale. The scanline and meta-motif signals have been summed before wrapping to show the hard-edge boundary condition.
What follows is a brief exploration of four audio signal processing techniques that may be used to transform these visual patterns. In the current modular approach, arbitrary processing can take place between each patterning level. This means there are infinite possibilities in how such audiovisual patterns can be manipulated. This discussion will be developed further to provide more detail on the processes used to produce the following figures.
Filters can be used to alter the spectral magnitude and phase characteristics of signals. While they are often thought of in terms of frequency, they are typically implemented as time-domain processes that involve summing a signal with a time-shifted version of itself, either through feed-back or feed-forward processes. This is worth considering to understand how filtering can be used to alter vector imagery, that the coordinates of points can be influenced in different ways by those of the points around them. In practice, this allows for altering the curvature of vector paths based on filter type.
Low pass filters attenuate high frequencies, meaning sharp angles and discontinuities are smoothed out. High pass filters have an opposite effect, attenuating low frequencies, making sharp curves more dramatic relative to the whole. Using Ableton Live’s EQ Eight, these filter types can be combined with bell curve, notch, and shelf filters for complex alterations of shape based on the frequency characteristics of pattern components. Such changes are particularly defined when filtering is applied to motifs and meta-motifs, where multiple elements are translated by step functions. These discontinuities involve a great deal of high frequency harmonic content, similar to square waves in traditional synthesis, so they become the most salient features affected by filtering. Low pass filters will tend to connect element and motif paths while high pass filters accentuate jumps between them into spikes. Some examples are shown below.
Reverberation is an approach to simulating the behavior of sound in a space or system. In application here, it serves as a useful way to add jitter and deviation to patterning, sometimes with an almost sketch-like quality.
Saturation is a nonlinear distortion process, historically derived from the behavior of magnetic and electrical systems responding to high amplitude signals. Incrementally increasing signal strength yields marginally lower increases in output, such that values of the signal plateau and peaks are rounded off. Such behavior can be emulated digitally using exponential functions, such as below:
invX = ((x > 0.0) * 2.0) - 1.0;
x *= invX;
x = invX * (x - ((1.0 / exp) * pow(x, exp))) * (exp / (exp - 1.0));
Here, invX equals 1 or -1 based on whether the current value of the input signal is positive or negative. This inversion factor is required since the signal input to the function must be positive. The exponent is intended to take on any value greater than one. Higher exponent values yield more linear behavior at lower amplitudes and more abrupt limiting at peaks.
Saturation can be applied to stereo signals independently, to produce a squaring effect, or to their shared polar magnitude, resulting in circular forms. The circles below have been amplitude modulated by a step function to produce concentric shapes that illustrate saturation’s nonlinear behavior.
Wavefolding involves applying a triangle wave transfer function to a signal, causing it to fold between upper and lower bounds. The result is visually intuitive with obvious applications in pattern design.
Discussions of the signal processing aspects of this work are to be developed further. Symmetry operations will also be explored at the level of meta-motif arrangement so pattern units may differ in orientation and perhaps other parameters such as scale and phase shift. Convolution as an audiovisual process deserves further investigation as well. It may be described as a reverb, filter, or spectral multiplication, leaving many angles from which to approach custom impulse response design as a tool for vector manipulation.