Development of an Evaluation Index System for Health Recommender Systems Based on the Health Technology Assessment Framework: Cross-Sectional Delphi Study
Background: Health Recommender Systems (HRSs) are digital platforms designed to deliver personalized health information, resources, and interventions tailored to users’ specific needs. However, existing evaluations of HRSs largely focus on algorithmic performance, with limited scientific evidence supporting user-centered assessment approaches and insufficiently defined evaluation metrics. Moreover, no unified or scientifically validated framework currently exists for evaluating these systems, resulting in limited cross-study comparability and constraining regulatory and implementation decision-making. Objective: This study aimed to develop a comprehensive, consensus-based evaluation index system for HRSs grounded in the Health Technology Assessment (HTA) framework. Methods: A two-round Delphi study was conducted with 18 experts, comprising clinicians, digital health researchers, and policymakers who possessed relevant professional experience and domain knowledge in HRSs. The age range of the experts was between 30 and 58 years, with 66.67% of them possessing over 10 years of professional experience. Based on literature analysis and HTA principles, a preliminary indicator set comprising 5 primary and 16 secondary indicators was constructed. Experts rated the importance of each indicator using a 5-point Likert scale and provided qualitative suggestions for refinement. After the Delphi process, the Analytic Hierarchy Process (AHP) was applied to determine indicator weights and assess consistency. Results: he Delphi survey reached full participation in the first round (18/18, 100%) and maintained an 88.9% response rate in the second round (16/18). The experts’ ages ranged from 30 to 58 years, and 66.67% had more than 10 years of professional experience. The final evaluation index system of HRS contained 5 first-level indicators (performance, effectiveness, safety, economy and social appropriateness), and 18 second-level indicators. The mean importance scores of the second-level indicators ranged from 4.25 ± 0.45 to 5.00 ± 0.00, with coefficients of variation (CVs) between 0.000 and 0.220. Among the first-level indicators, safety received the highest weight (0.289), followed by social appropriateness (0.251), effectiveness (0.193), performance (0.136), and economy (0.132). Conclusions: This study presents the evaluation index system for HRSs grounded in the HTA framework and validated through expert consensus. The resulting framework not only provides actionable guidance for the design, optimization, and implementation of HRSs but also fills a methodological gap in the field by offering quantifiable, hierarchical evaluation indicators with validated weighting. Future research will involve iterative refinement and empirical validation of the system in real-world deployment settings, thereby enabling continuous improvement and facilitating the establishment of unified evaluation standards for HRS research and practice.
