By Miwa Fukino, Yoshito Hirata, Kazuyuki Aihara
Music evokes emotion. Many musical factors, such as timbre, melody, harmony, meter, and beat, constitute musical structure and produce an emotional response in the listener. The relationship between emotion and the physical properties of music has been studied but is not fully understood. Previous research on the relation between music signal processing and physical properties has proposed the extraction of various musical features by assuming linear models. Multiple feature vectors are typically used to characterize musical factors and the structure of musical compositions. These methods perform very well when applied to extract physical and/or objective properties of the music, for example, chord estimation and beat analysis. However, such methods are not suitable for analyzing music’s emotional effect since they mainly focus on amplitude information, discarding phase information among the features. We proposed a new method based on nonlinear time series analysis to solve this problem [2].
Recurrence Plots
A recurrence plot is an important analysis tool for nonlinear time series data. Eckmann et al. [1] originally proposed the plot to visualize a time series (for a review, see [7]). It is a two-dimensional plot whose axes both correspond to the same time axis. If two states at a pair of times are close to each other, we plot a point at the corresponding place; otherwise we do not plot a point there. Mathematically, letting \(\{x_i \in R^m | i = 1,2, ..., I\}\) be a time series and \(d:R^m \times \: R^m \rightarrow \{0\} \cup R_+\) be a distance function, we can define a recurrence plot \(R\) as follows:
\[ \begin{equation}
R(i,j)=\begin{cases}
1, & \text{if} \quad d(x_i,x_j) < \varepsilon, \qquad (1) \\
0, & \text{otherwise},
\end{cases}
\end{equation} \]
for \(1 \le i, j \le I.\) Here, \(R (i, j) = 1\) means that a point is plotted at \((i, j)\), and \(R (i, j) = 0\) means that a point is not plotted at \((i, j)\). The symbol \(\varepsilon\) is the threshold for obtaining a recurrence plot.
Figure 1. A recurrence plot of a sine wave. Both the bottom (xi)(xi) and the
left (xj)(xj) are waveforms along the same timeline. We used the Euclidean
distance for dd and set ε=0.1
This simple graph can reveal a lot of things about the underlying dynamics of the data. For instance, a time series of white noise produces a recurrence plot where points are plotted almost randomly. A periodic time series produces a periodic pattern in the recurrence plot, as shown in Figure 1. A time series generated from deterministic chaos creates a recurrence plot containing many short diagonal segments. If two time series generate the same recurrence plot, their underlying dynamical rules are equivalent [8]. Therefore, by looking at a recurrence plot, one can discern the properties of the underlying dynamics, such as serial dependence [4] and consistency with Devaney’s mathematical definition of deterministic chaos [3].
Given a multivariate time series with a fixed sampling frequency, one may use the Euclidean distance for the distance function. By replacing the distance function with another distance function, we can analyze various exotic data—time series of networks [6] and marked point processes [5, 9], for example—which are time series of events accompanied by supplementary information.
Recurrence Plot of Recurrence Plots
Conventional recurrence plots are usually applied to a time series of length up to 10,000, if we consider the plot’s visibility. The length of musical data is too large to directly apply conventional recurrence plots: a musical piece of five minutes with 44.1 kHz 16-bit monaural linear pulse code modulation (PCM) has 13,230,000 time points. One cannot grasp global characteristics by such a large recurrence plot. On the other hand, our method [2] enables us to represent both local and global characteristics simultaneously by using recurrence plots hierarchically. In this method, we first divided a long time series into 4-second windows and calculated multiple short-term recurrence plots (short-term RPs), which represented local characteristics. Second, using distances between the short-term RPs, we calculated a recurrence plot—representing global characteristics—of the entire piece. We call this long-term recurrence plot a recurrence plot of recurrence plots (RPofRPs) [2]. Given an original time series \(\{x_i \in R^m | i = 1,2, ..., IK\},\) where \(I\) and \(K\) denote the size of RPofRPs and short-term RPs respectively, we can write its definition [2] as
\[\begin{equation}
R(i,j)=\begin{cases}
1, & \text{if} \quad d_r(X_i,X_j) < \varepsilon, \qquad (2) \\
0, & \text{otherwise},
\end{cases}
\end{equation}\]
for \(1 \le i, j \le I.\) Here, \(d_r(X_i,X_j)\) presents a distance function between \(X_i\) and \(X_j\), and \(X_i\) presents the \(i\)th unthresholded short-term RP whose \(k, l\) component is defined by
\[X_i (k,l) = d_s(x_{K(i-1)+k}, x_{K(i-1)+l}) \qquad (3)\]
for \(1 \le k, l \le K.\) Here, \(d_s(x_{K(i-1)+k}, x_{K(i-1)+l})\) shows a distance function between \(x_{K(i-1)+k}\) and \(x_{K(i-1)+l}\).
Roughly speaking, a short-term RP represents a local relation and a RPofRPs represents similarities between the local relations. This simple method can retain most of the information contained in the musical pieces, including the information of phases, which forms the very core of music.
Visualization
Figure 2. An example of the thresholded RPofRPs. We calculated an amateur live pop tune that was played by drums, a chopper base, an electric guitar, an electric piano, two synthesizers, and a female vocal. We can find many diagonal lines parallel to the main diagonal line. Red squares show the intersections of the chorus section. Similar patterns of the figure represent similar sections of the music. We used modified Canberra distance for drdr and set ε=0.2.
Visualization through the RPofRPs reveals fundamental physical aspects of musical pieces, as shown in Figure 2 (see also Figure 10 of [2]). For example, many nearly diagonal lines parallel to the main diagonal show nearly periodic regularity of phrases, and narrowing or widening the width of the two lines indicates gradual tempo transition. A boundary of graphical motifs is equivalent to a boundary of musical phrases. The RPofRPs can also reveal a similarity between phrases played in different key scales. Here, the diagonal lines of the RPofRPs show a succession of the same regularity, while such lines of conventional recurrence plots show a regularity. Thus, through the RPofRPs, many users can easily access and grasp the abstract image of the complicated nonlinear information in musical pieces. Therefore, while obtaining a RPofRPs may not be a goal in itself, it could be a keystone for researchers in other fields. For example, psychologists could explore how people are touched and/or healed by music.
Subsequent research will further analyze music and develop methods for nonlinear time series analysis. We hope that our methods will be used widely not only by mathematicians but also by non-mathematicians for analysis of complicated real big data.
References
[1] Eckmann, J.P., Kamphorst S.O., & Ruelle, D. (1987). Recurrence plots of dynamical systems. Europhys. Lett., 4(9), 973-977.
[2] Fukino, M., Hirata, Y., & Aihara, K. (2016). Coarse-graining time series data: Recurrence plot of recurrence plots and its application to music. Chaos, 26(2), 023116.
[3] Hirata, Y., & Aihara, K. (2010). Devaney’s chaos on recurrence plots. Phys. Rev. E, 82(3), 036209.
[4] Hirata, Y., & Aihara, K. (2011). Statistical tests for serial dependence and laminarity on recurrence plots. Int. J. Bifurcat. Chaos, 21(4), 1077-1084.
[5] Hirata, Y., & Aihara, K. (2012). Timing matters in foreign exchange markets. Physica A, 391(3), 760-766.
[6] Iwayama, K., Hirata, Y., Takahashi, K., Watanabe, K., Aihara, K., & Suzuki, H. (2012). Characterizing global evolutions of complex systems via intermediate network representations. Sci. Rep., 2, 423.
[7] Marwan, N., Romano, M.C., Thiel, M., & Kurths, J. (2007). Recurrence plots for the analysis of complex systems. Phys. Rep., 438(5-7), 237-329.
[8] Robinson, G., & Thiel, M. (2009). Recurrences determine the dynamics. Chaos, 19(2), 023104.
[9] Suzuki, S., Hirata, Y., & Aihara, K. (2010). Definition of distance for marked point process data and its application to recurrence plot-based analysis of exchange tick data of foreign currencies. Int. J. Bifurcat. Chaos, 20(11), 3699-3708.
Miwa Fukino is a doctoral student of mathematical informatics at the University of Tokyo. Yoshito Hirata is a project associate professor at the Institute of Industrial Science, the University of Tokyo. Kazuyuki Aihara is a professor at the Institute of Industrial Science, the University of Tokyo.