# Music Visualized by Nonlinear Time Series Analysis

Music evokes emotion. Many musical factors, such as timbre, melody, harmony, meter, and beat, constitute musical structure and produce an emotional response in the listener. The relationship between emotion and the physical properties of music has been studied but is not fully understood. Previous research on the relation between music signal processing and physical properties has proposed the extraction of various musical features by assuming linear models. Multiple feature vectors are typically used to characterize musical factors and the structure of musical compositions. These methods perform very well when applied to extract physical and/or objective properties of the music, for example, chord estimation and beat analysis. However, such methods are not suitable for analyzing music’s emotional effect since they mainly focus on amplitude information, discarding phase information among the features. We proposed a new method based on nonlinear time series analysis to solve this problem [2].

### Recurrence Plots

A recurrence plot is an important analysis tool for nonlinear time series data. Eckmann et al. [1] originally proposed the plot to visualize a time series (for a review, see [7]). It is a two-dimensional plot whose axes both correspond to the same time axis. If two states at a pair of times are close to each other, we plot a point at the corresponding place; otherwise we do not plot a point there. Mathematically, letting \(\{x_i \in R^m | i = 1,2, ..., I\}\) be a time series and \(d:R^m \times \: R^m \rightarrow \{0\} \cup R_+\) be a distance function, we can define a recurrence plot \(R\) as follows:

\[ \begin{equation}

R(i,j)=\begin{cases}

1, & \text{if} \quad d(x_i,x_j) < \varepsilon, \qquad (1) \\

0, & \text{otherwise},

\end{cases}

\end{equation} \]

for \(1 \le i, j \le I.\) Here, \(R (i, j) = 1\) means that a point is plotted at \((i, j)\), and \(R (i, j) = 0\) means that a point is not plotted at \((i, j)\). The symbol \(\varepsilon\) is the threshold for obtaining a recurrence plot.

**Figure 1.**A recurrence plot of a sine wave. Both the bottom (xi)(xi) and the left (xj)(xj) are waveforms along the same timeline. We used the Euclidean distance for dd and set ε=0.1

Given a multivariate time series with a fixed sampling frequency, one may use the Euclidean distance for the distance function. By replacing the distance function with another distance function, we can analyze various exotic data—time series of networks [6] and marked point processes [5, 9], for example—which are time series of events accompanied by supplementary information.

### Recurrence Plot of Recurrence Plots

Conventional recurrence plots are usually applied to a time series of length up to 10,000, if we consider the plot’s visibility. The length of musical data is too large to directly apply conventional recurrence plots: a musical piece of five minutes with 44.1 kHz 16-bit monaural linear pulse code modulation (PCM) has 13,230,000 time points. One cannot grasp global characteristics by such a large recurrence plot. On the other hand, our method [2] enables us to represent both local and global characteristics simultaneously by using recurrence plots hierarchically. In this method, we first divided a long time series into 4-second windows and calculated multiple short-term recurrence plots (short-term RPs), which represented local characteristics. Second, using distances between the short-term RPs, we calculated a recurrence plot—representing global characteristics—of the entire piece. We call this long-term recurrence plot a recurrence plot of recurrence plots (RPofRPs) [2]. Given an original time series \(\{x_i \in R^m | i = 1,2, ..., IK\},\) where \(I\) and \(K\) denote the size of RPofRPs and short-term RPs respectively, we can write its definition [2] as

\[\begin{equation}

R(i,j)=\begin{cases}

1, & \text{if} \quad d_r(X_i,X_j) < \varepsilon, \qquad (2) \\

0, & \text{otherwise},

\end{cases}

\end{equation}\]

for \(1 \le i, j \le I.\) Here, \(d_r(X_i,X_j)\) presents a distance function between \(X_i\) and \(X_j\), and \(X_i\) presents the \(i\)th unthresholded short-term RP whose \(k, l\) component is defined by

Roughly speaking, a short-term RP represents a local relation and a RPofRPs represents similarities between the local relations. This simple method can retain most of the information contained in the musical pieces, including the information of phases, which forms the very core of music.

### Visualization

**Figure 2.**An example of the thresholded RPofRPs. We calculated an amateur live pop tune that was played by drums, a chopper base, an electric guitar, an electric piano, two synthesizers, and a female vocal. We can find many diagonal lines parallel to the main diagonal line. Red squares show the intersections of the chorus section. Similar patterns of the figure represent similar sections of the music. We used modified Canberra distance for drdr and set ε=0.2.

Subsequent research will further analyze music and develop methods for nonlinear time series analysis. We hope that our methods will be used widely not only by mathematicians but also by non-mathematicians for analysis of complicated real big data.

**References**

[1] Eckmann, J.P., Kamphorst S.O., & Ruelle, D. (1987). Recurrence plots of dynamical systems. *Europhys. Lett., 4*(9), 973-977.

[2] Fukino, M., Hirata, Y., & Aihara, K. (2016). Coarse-graining time series data: Recurrence plot of recurrence plots and its application to music. *Chaos, 26*(2), 023116.

[3] Hirata, Y., & Aihara, K. (2010). Devaney’s chaos on recurrence plots. *Phys. Rev. E, 82*(3), 036209.

[4] Hirata, Y., & Aihara, K. (2011). Statistical tests for serial dependence and laminarity on recurrence plots. *Int. J. Bifurcat. Chaos, 21(*4), 1077-1084.

[5] Hirata, Y., & Aihara, K. (2012). Timing matters in foreign exchange markets. *Physica A, 391*(3), 760-766.

[6] Iwayama, K., Hirata, Y., Takahashi, K., Watanabe, K., Aihara, K., & Suzuki, H. (2012). Characterizing global evolutions of complex systems via intermediate network representations. *Sci. Rep., 2*, 423.

[7] Marwan, N., Romano, M.C., Thiel, M., & Kurths, J. (2007). Recurrence plots for the analysis of complex systems. *Phys. Rep., 438*(5-7), 237-329.

[8] Robinson, G., & Thiel, M. (2009). Recurrences determine the dynamics. *Chaos, 19(*2), 023104.

[9] Suzuki, S., Hirata, Y., & Aihara, K. (2010). Definition of distance for marked point process data and its application to recurrence plot-based analysis of exchange tick data of foreign currencies. Int. J. Bifurcat. *Chaos, 20*(11), 3699-3708.