Analyzing User Engagement with TikTok's Short Format Video Recommendations using Data Donations

The paper examines how users engage with short-format videos on TikTok and the role/effectiveness of its recommendation algorithm. Motivated by a lack of empirical studies using authentic user data and concerns about potential algorithmic steering toward problematic content, the authors design a data donation system to collect real user viewing histories and engagement signals. They define effectiveness constructs of recommendations through user engagement: increasing time spent, number of videos watched, attention (watching till the end), and interactions (likes) over time. The study aims to quantify these engagement signals longitudinally, including differences between content from followed accounts (social graph) versus non-followed accounts (algorithmic discovery).

The related work covers: 1) TikTok's recommendation algorithm and audits: user interactions, video info, and device/account info affect the For You feed; prior sock-puppet audits highlight language, location, follows/likes, view length, and posting time as influential; investigations raise concerns about rabbit holes and harmful content. 2) Content analyses on TikTok: sentiment towards topics, virality/memes, platform differences, engagement correlates, misinformation/harmful content prevalence, and challenges of multimodal misinformation detection. 3) User studies: motivations (escapism, entertainment, learning, self-expression, novelty), folk theories of algorithm behavior, and mixed findings about well-being impacts; marginalized communities' experiences. 4) User engagement frameworks emphasize attributes like attention and feedback; prior YouTube/TikTok studies link video characteristics, view duration, and sentiment to engagement, often focusing on specific content types. 5) Data donation: GDPR-enabled user data access provides rich behavioral traces; benefits over scraping or synthetic accounts; prior successes in uncovering ad targeting mechanisms and adolescent Instagram use; motivations include overcoming self-report unreliability and authenticity limitations of researcher-created accounts. The authors note their study is the first to use user-donated TikTok data to audit engagement with short-format video recommendations.

The authors built Social Media Donator (SMD), a data donation platform enabling TikTok users to request and donate their GDPR data packages. TikTok data includes video viewing history, like history, search/share history, login info, app settings, comments, favorites, followers/following, ads info, profile info, DMs, uploads, purchases, and account status. Client-side anonymization removes sensitive fields by default (profile info, DMs, uploads, IP/device info, purchase info, account status), and users can further customize donations (mandatory: viewing history URLs and timestamps; optional fields with clear descriptions and sensitive data warnings). Compensation ranges from $5 (viewing history) plus $1 per optional field (comments: $2 for content+timestamps, $1 for timestamps only), up to $16 total; survey completion adds $4. Recruitment used a Twitter post and Facebook Ads (Jan–Feb 2022), yielding 347 participants and $6.9K in compensation. Donations were assessed for quality: duplicate or near-duplicate detection via pairwise Jaccard similarity on video URLs/timestamps (flagged >0.2; 31 near-duplicates ~0.9 similarity were rejected); fabricated data checks by measuring obtainable video metadata coverage per donation (≥70% for all donations, median 90%). Video metadata was collected using an unofficial Python API/Selenium scraper (Jan 17–Mar 12, 2022) for 4,122,038 of 4,938,805 unique videos (83.4%), including creation date/time, description/title, uploader info, duration/format, and platform-wide engagement stats (views/shares/comments). Viewing duration inference used inter-arrival times of consecutive view timestamps and a two-component Gaussian mixture model (per Halfaker et al. 2015) to separate real views from long breaks; a threshold of 105 seconds was selected (98.5% inferred viewing durations ≤105s). Attention per view was computed as inferred viewing duration divided by video duration ×100%; attention ≥100% indicates watched till end (capped at 100%), <100% indicates skipped. Ethics approvals were obtained from Saarland University and University of Washington IRBs; consent via SMD; data deletions promised within 36 months; results reported in aggregate; metadata scraping limited to publicly available videos.

Sample and dataset: 347 users; 9,212,100 video views (Jul 26, 2020–Feb 21, 2022); 1,120,716 likes (328 participants; a 2-month gap missing like data); 13,282 searches; 24,944 shares (absent until Jul 28, 2021); 52,436 comments (no linked video IDs); 84,654 followings; 43,642 followers. Demographics: 96% completed survey; locations primarily Africa (52%), North/Central America (32%), South America (6.6%), Europe (3%); ages mostly ≤34 (91%); gender roughly balanced; 54% bachelor’s or higher. Activity levels: median daily time spent 1,622s (27 min; Q1 834s; Q3 2,891s), with average starting ~1,730s and exceeding 3,000s (~50 min) by day 120; median videos/day 89.9 (Q1 40.7; Q3 170.3; σ 128.9); average viewed videos rose from ~107/day to consistently >200/day by ~day 80 (peak 233). Over time trends: daily time and volume of videos increased (≈2x by ~80 days), confirmed in a subset active all 120 days (26%). Following vs non-following: only 10.3% of views from accounts users already followed; number of following accounts grew (avg ~40 on day 1 to ~350 by day 120), yet share of views from following accounts remained stable ~10% over time. Attention: median attention per view 82%; 45% of views watched till end; 55% not watched till end; 24% skipped before 20% of duration; 40% skipped before 50%. Per-user distribution: no user watched >65% of videos till end; ~70% of users watched till end for 30–50% of their views. Attention over time remained stable; users watched till end a higher fraction of videos from non-followings (≈44–46%) than from followings (≈38–42%). Interaction (likes): percentage of liked videos increased over time; from non-followings ≈6%→12% by day 120; from followings ≈12%→18% by day 120; videos from followings received more likes overall. Video characteristics: non-following videos were significantly more popular (median plays 755K vs 85K for followings), slightly shorter (Q1 10s, Q3 30s vs followings Q1 11s, Q3 35s), and older (median age 3.29 days for followings vs 1.21 days for non-followings at time of viewing); differences significant (t-tests p<0.05). Regional comparison: increasing trends in time and volume across regions; North/Central America participants watched more and spent more time than Africa/others; proportion of views from followings similar across regions (~10–11% N/C America; ~10.7% Africa; ~8.3% others). Attention stable across regions; likes increased more in N/C America (≈6%→16%), while Africa showed modest increase then stabilization (≈6–8%).

Findings indicate TikTok’s recommendation system effectively increases overall platform engagement (time spent and volume of videos) and interactions (likes) over users’ tenure, but does not increase attention (watching to the end) over time. The stable attention, coupled with a majority of views not watched till end, suggests either inherent difficulty in predicting short-video completion or an algorithmic strategy that mixes highly engaging with less engaging content to sustain long-term retention. Users watch a higher fraction of non-following videos to completion, likely due to their greater platform-wide popularity, aligning with the platform’s emphasis on the For You feed over the social graph. However, users are more inclined to like content from followed accounts, reflecting social ties and explicit interest signals. These insights address the research question by showing that recommendation effectiveness manifests more in increasing exposure and interactive engagement than in end-to-end attention. Implications include potential addictive dynamics and the need for platform nudges and transparency about recommendation objectives and trade-offs, as well as broader audits of algorithmic systems powering short-format video feeds.

The study provides the first large-scale empirical audit of TikTok short-format video engagement using authentic user-donated data. Contributions include: 1) a data donation system design and deployment (SMD) demonstrating the feasibility of collecting rich behavioral traces ethically; 2) evidence that time spent and video consumption increase over time, while attention remains stable, and likes increase—particularly for followed accounts; 3) characterization of differences between non-following vs following content (popularity, duration, age) informing why non-following videos garner more completion. Future work should analyze whether attention prediction for short videos is intrinsically difficult or intentionally mixed to maximize long-term engagement, conduct compliance audits of platform-provided data integrity, expand data donation to other platforms (e.g., YouTube Shorts), and investigate the effects of well-being nudges, retention/dropping-off behaviors, and pricing/trust mechanisms in data donation infrastructures.

Data and recruitment: limited to 347 participants with non-representative global demographics; lack of public baseline on TikTok user demographics prevents representativeness assessment; metadata missing for ~17% of videos (private/deleted); recruitment methods may bias toward participants willing to donate data; only active users (≥3 months) were recruited, excluding lost users. Analysis: alignment across varying user start times may not reflect a steady state; engagement analysis focuses on likes (comments/shares limited or lacking video linkage); attention analysis may be affected by ceiling effects (limited viewing time) not modeled; potential confounders beyond recommendations not fully accounted for; inferred viewing durations rely on thresholding (≥105s) and exclude last video in sessions, making time estimates lower bounds; like data has a 2-month gap; share data absent before July 28, 2021. Broader limitations reflect challenges inherent to auditing closed platforms via data donations and inference-based measures.