Modern touchscreen-based IVISs are increasingly complex, making user experience (UX) evaluation challenging. Current methods, relying on qualitative feedback and small-scale studies, lack the scalability to handle the growing number of features and the need for context-aware evaluations in driving situations. This paper addresses this gap by proposing ICEBOAT, a data-driven solution. The design process for IVISs lacks effective tools for visualizing and analyzing user interaction data. While cars collect large amounts of usage data, UX experts often lack access or the appropriate tools to analyze it effectively. ICEBOAT aims to empower UX professionals by providing an interactive visualization tool tailored to their needs, automating data processing, and enabling efficient analysis of driver interactions with the IVIS center stack touchscreen. The tool visualizes user interaction data, driving data, and glance data collected from production vehicles, allowing experts to define tasks either manually or using an interactive IVIS emulator. ICEBOAT then generates various visualizations and statistics based on expert interviews and prior research. The contributions of this research are: (1) identifying the information and interaction needs of automotive UX experts; (2) extending existing visualizations to meet those needs; (3) developing ICEBOAT to automate data processing and visualization; and (4) evaluating the tool's effectiveness with industry experts.

Existing methods for designing and evaluating IVISs, such as questionnaires and user studies, are often slow and expensive. This paper highlights the growing need for data-driven design processes in the automotive industry, similar to the practices in web and app development. However, the automotive domain lags in adapting to data-driven decision-making due to cultural, technological, and organizational challenges. While substantial user interaction data is collected, UX experts often lack direct access and the tools needed for effective analysis. The paper reviews existing big data visualization tools, noting that general-purpose tools often fall short of meeting domain-specific needs. The existing academic research on IVIS evaluation often focuses on small-scale studies or controlled experiments, and lacks the capability to handle the large datasets collected in real-world conditions. The paper specifically addresses this gap by building upon prior work to create a more comprehensive, user-focused tool for evaluating IVIS usability and distraction.

The researchers employed a mixed-methods UCD approach, combining qualitative and quantitative data collection. Study 1 involved semi-structured interviews with four experienced automotive UX experts (5-9 years of experience, at least 5 years in UI design at Mercedes-Benz). The interviews explored information and interaction needs in analyzing large datasets. The interviews focused on usability and distraction metrics, feature usage, usage patterns, system information (considering car model variations), contextual information (driving situation, passenger count), input modalities, and user demographics. The analysis revealed 39 information needs across seven categories and 14 interaction needs across four categories. The existing visualizations from prior work by Ebel et al. were evaluated to determine their adequacy for meeting the newly discovered requirements. These interviews revealed that existing visualizations partially met the information needs of the users but lacked the interface and interaction capabilities to fully support UX experts in their workflow. Study 1 also confirmed the usefulness of a multi-level user behavior framework presented in prior work. Following Study 1, a participatory design approach was adopted to co-design prototypes with the same four UX experts. This co-design involved four iterative sessions to refine the visualizations and develop the interactive tool, ICEBOAT. Study 2 evaluated ICEBOAT with 12 participants from Mercedes-Benz and MBition (4 designers, 4 UX researchers, 4 data scientists). Usability testing, interviews, and a Context of Use questionnaire were used to assess the tool's effectiveness. Participants performed seven evaluation tasks centered around analyzing a navigation feature's destination entry task. The tasks involved using the IVIS emulator to define a task and then analyzing the data for bottlenecks, driver distraction, and outliers in glance behavior. The System Usability Scale (SUS) questionnaire was used to measure usability, and the Context of Use questionnaire assessed the tool's value in the participants' workflows. Quantitative data from the SUS and Context of Use questionnaires were collected, along with qualitative feedback from interviews and observations.

Study 1 revealed that automotive UX experts need tools that support task-specific analysis, provide various metrics (usability, distraction), visualize usage patterns, account for system and contextual variations, and incorporate different input modalities (touch, voice). Experts also prioritized the ability to define tasks interactively and drill down for detailed analysis. ICEBOAT addresses these needs by providing an interactive web application with an IVIS emulator for task definition. The tool allows users to define tasks in two ways: (1) selecting UI elements, (2) using the IVIS emulator to record a task flow. Key Performance Indicators (KPIs) are visualized through a dashboard page and a user flow analysis page, which consists of four panels that gradually appear as the user progresses through the analysis (drill-down mechanism). The analysis page incorporates Sankey diagrams, tables, and box plots for data exploration. The Sequence Details Panel allows the user to examine glance data, driving data, and interactions within a specific sequence. Study 2's quantitative results revealed a mean SUS score of 68.125 (average), indicating acceptable usability. Data scientists rated the tool higher than UX researchers, possibly due to their familiarity with data analysis tools. The Context of Use questionnaire had a mean score of 4.2 out of 5, indicating that the tool aligns well with participants' workflows. Qualitative feedback was predominantly positive, with participants emphasizing the tool's value in improving efficiency and providing data-driven insights. Participants readily identified bottlenecks, driver distraction, and outliers in glance behavior. The ability to define tasks through the IVIS emulator was well-received, simplifying the analysis process. The Sankey diagrams were easily understood, and the ability to drill down to sequence-level details proved valuable. While some minor errors occurred during usability testing, these were primarily related to the initial use of the IVIS emulator.

ICEBOAT effectively addresses the challenges of utilizing big data for automotive UI evaluation. It empowers UX experts by providing direct access to telematics data, handling automotive-specific analytical needs, and preventing information overload through its interactive drill-down design. The findings highlight the value of a user-centered approach in developing big data analytics tools. The differences in SUS scores between data scientists and UX experts suggest a need for further design improvements to cater to users with less data analysis expertise. The successful identification of bottlenecks, driver distractions, and outliers using ICEBOAT showcases its potential to inform design decisions and improve IVIS usability and safety. The integration of the IVIS emulator simplifies the task definition process, lowering the barrier to entry for less technically inclined users.

ICEBOAT successfully democratizes access to large-scale user interaction data within the automotive UX design process. It facilitates data-driven decision-making, addresses automotive-specific analysis challenges, and mitigates information overload. Future work includes expanding the tool to accommodate additional interaction modalities (voice, hardkeys), extending its application across different OEMs, addressing privacy concerns related to user data, and improving the robustness of remote control functionality. The research demonstrates the potential of user-centered design in creating effective data visualization and analysis tools for complex domains.

The study's limitations include a potential bias toward Mercedes-Benz's practices and the limited representation of interaction modalities (focus on touch interactions). The remote evaluation methodology introduced minor limitations, particularly with the remote control functionality. The higher usability rating by data scientists versus UX experts might suggest that further improvements could be made to enhance the tool's accessibility and intuitive nature for users with limited data analysis experience. The study was limited by privacy concerns that restricted the collection of detailed personal data. Future work should address these limitations to increase the generalizability and applicability of the findings.