OPEN A self-organizing, living library of time-series data

Time-series data, representing repeated measurements over time, are ubiquitous in scientific research, spanning diverse fields like biology, economics, and astrophysics. The sheer volume and variety of applications, from medical diagnostics to financial modeling, have led to a multitude of analysis methods. However, cross-disciplinary comparison and collaboration remain limited due to challenges in identifying commonalities between datasets from different domains. This lack of interdisciplinary interaction hinders the discovery of shared patterns and mechanisms underlying similar dynamics in diverse systems. The paper introduces CompEngine, a self-organizing library aimed at addressing this challenge by automatically highlighting meaningful connections between time series from various fields, ultimately promoting collaboration and a deeper understanding of time-varying systems.

Existing research on time-series analysis emphasizes the development of numerous algorithms and methods tailored to specific domains. However, a lack of systematic comparison and a common framework for integrating data from diverse sources have hindered the exploration of broader patterns and the potential for cross-disciplinary insights. Previous work on feature-based representations of time series has demonstrated their efficacy in classification, clustering, and forecasting. The 'highly comparative time-series analysis' approach, implemented in the hctsa software package, provides a foundation for comparing time series irrespective of their origin or sampling rates. This approach involves extracting numerous features to represent the time series' statistical properties, allowing for meaningful comparisons in a high-dimensional feature space.

CompEngine leverages a feature-based representation to overcome the challenges of comparing time series with different characteristics. The core concept is to map each time series into a common feature space, using a set of features that capture various statistical properties, including autocorrelation, stationarity, complexity, and distribution of values. The hctsa software package, with its extensive feature library, was initially considered. However, for computational efficiency, a smaller, yet comprehensive subset of 22 features, called catch22, was chosen. These features capture a wide range of dynamics while being computationally efficient. The distance between time series is calculated using the Euclidean distance between their corresponding feature vectors. The platform's self-organizing nature is achieved by continuously updating the connections and similarities between time series as new data is uploaded. The user interface is designed to facilitate interactive exploration and visualization of the database, allowing users to discover similar time series across disciplines, and to analyze their features and characteristics. The system allows for easy upload of data in various formats (txt, csv, xlsx, mp3, wav), along with associated metadata (name, sampling rate, description, source, category, tags). This metadata, combined with the computed features, enables both automatic organization and interactive exploration.

The paper demonstrates the efficacy of the feature-based approach in organizing a diverse dataset of over 24,000 time series from various sources, including empirical data (birdsong, population dynamics, ECG, gait) and synthetic data (simulated differential equations, iterative maps). A t-SNE projection of the high-dimensional feature space reveals meaningful clusters of time series with similar dynamical properties. This visualization highlights unexpected connections between seemingly disparate systems, showcasing the potential for cross-disciplinary insights. CompEngine successfully combines automatic organization based on computed features and user-provided metadata to facilitate interactive data exploration. The visualization tools within CompEngine enable users to view the nearest neighbors to a target time series as an interactive network. Each node represents a time series, colored by category, and connected to other nodes based on feature-vector similarity. This visualization, along with a list view, allows users to explore the relationships between their time series and similar datasets in the library. Users can also download individual time series, subsets of matching data, or the entire database, supporting both individual exploration and programmatic access through a public API.

CompEngine successfully addresses the challenges of interdisciplinary collaboration in time-series analysis by providing a self-organizing platform that connects researchers based on the empirical structure of their data, rather than solely on user-assigned metadata. The platform's ability to highlight unexpected connections between real-world and simulated data, and between disparate empirical systems, facilitates the identification of shared patterns and mechanisms. The availability of a large and diverse dataset also enables more robust evaluations of time-series analysis algorithms, reducing biases associated with manually selected datasets. The design of CompEngine as a living library, constantly evolving with community contributions, ensures that it remains a valuable resource for the scientific community.

CompEngine represents a novel approach to data organization and sharing, offering a powerful tool for interdisciplinary collaboration in time-series analysis. Its self-organizing nature, based on a feature-based representation, enables the automatic discovery of meaningful connections between diverse time series. The platform's extensive library, combined with interactive visualization and data download capabilities, facilitates both individual exploration and large-scale analysis. Future work could focus on expanding the feature set, incorporating advanced search functionalities, and developing more sophisticated algorithms for identifying meaningful relationships within the growing dataset. The overall concept and design of CompEngine could serve as a template for creating self-organizing libraries for other types of complex data.

The computational cost of feature extraction might limit the scalability for extremely large datasets. The reliance on a specific set of features could limit the discovery of relationships based on other characteristics not captured by catch22. The quality of the database depends on the accuracy and completeness of the user-provided metadata. While the platform supports several data formats, some highly specialized formats might not be directly compatible.