AN11--Kaiser Permanente R&D Statistical Expertise EHR Evaluation

2 Notice of Sources Sought The purpose of this Sources Sought Announcement is for market research to make appropriate acquisition decisions and to gain knowledge of potential qualified Service-Disabled Veteran Owned Small Businesses (SDVOSB), Veteran Owned Small Businesses (VOSB), Women Owned Small Businesses (WOSB) and Small Businesses interested and capable of providing the services requested, as well as any large businesses. The results from this source sought notice will be used to determine the appropriate set-aside. The intended contract is a firm-fixed price R&D service contract. The SBA Non-Manufacturer Rule is not applicable as the intended contract is for services. The Department of Veterans Affairs, NCO 20 is looking for Kaiser Permanente s authorized service providers that can provide below R&D services. Statement of Work: Contract Title Expertise in Measurement Error to Support VHA Digital Health Office Immersive Program Evaluation Project: VHA Digital Health Office Immersive Program Evaluation Background Virtual reality (VR) is an emerging technology that has been used as a treatment in diverse pain populations and for diverse indications. VA's Digital Health Office is exploring offering Veterans a variety of VR therapies through conducting multiple pilot initiatives. To date over 8,000 Veterans have engaged in VR activities through engaging in headset technologies in clinics and at home. Quantifying the potential benefit of disseminating VR technologies requires careful assessment of the VR activities, how they are being used in combination with other therapies such as mental health treatment and physical therapy, and monitoring of outcomes including patient reported changes in health, pain and well-being, healthcare utilization changes and overall costs. The Digital Health Office is providing tools to measure VR engagement and has requested evaluation of the VHA Immersive Program and to generate insights about how to best support further XR implementation. The evaluation activities will leverage data collected as part of VHA Immersive Program activities, data about participants linked to VHA administrative data, and primary data collected by the evaluation team. The evaluation will span three fiscal years, and will be comprised of retrospective analyses, mixed-methods evaluation design, and prospective data collection. To understand how population-level coding and documentation of VR engagement reflect true exposure levels, fidelity, and overall VR dose we require a statistical expert. We are requesting a contract with Kaiser Permanente Washington Health Research Institute (KPWHRI) in order for Pamela Shaw, PhD to work with this project and provide expertise in statistical methods and study design. Dr. Shaw is a Senior Investigator in the Division of Biostatistics at KPWHRI and also Affiliate Professor in the Department of Biostatistics at the University of Washington School of Public Health. Dr. Shaw s research focuses on the design and analysis of studies leveraging electronic health records (EHR) data to support rigorous, cost-effective, clinical and epidemiologic research to improve the health and guide treatment decisions for US citizens. Dr. Shaw brings expertise in complex survey design, statistical methods and study design to address measurement error and outcome misclassification, and the evaluations of interventions using clinical trial and observational data across a variety of health care settings and populations. Her work integrates methodological rigor with practical relevance, advancing the use of electronic health records data to public health and clinical decision making. Selected research projects done under Dr. Shaw s leadership: Williamson B, Krakauer C, Johnson E, Gruber S, Shepherd BE, van der Laan MJ, Lumley T, Lee H, Hernandez Munoz JJ, Zhao F, Dutcher SK, Desai, R, Simon GE, Shortreed SM, Nelson JC, Shaw PA. Assessing treatment effects in observational data with missing confounders: A comparative study of practical doubly-robust and traditional missing data methods. Statistics in Medicine, 2026 Feb;45(3-5):e70366 Shepherd BE, Han K, Chen T, Bian A, Pugh S, Duda SN, Lumley T, Heerman WJ, Shaw PA. Multiwave validation sampling for error-prone electronic health records. Biometrics. 2023 Sep;79(3):2649-2663. PubMed Central PMCID: PMC10525037. Shaw PA, Yang JB, Mowery DL, Schriver ER, Mahoney KB, Bar KJ, Ellenberg SS. Determinants of hospital outcomes for COVID-19 infections in a large Pennsylvania Health System. PLoS ONE, 2022 May 19; 17(5): e0268528. Shaw PA, Gustafson P, Carroll RJ, Deffner V, Dodd KW, Keogh RH, Kipnis V, Tooze JA, Wallace MP, Kuchenhoff H, Freedman LS. STRATOS guidance document on measurement error and misclassification of variables in observational epidemiology: Part II -more complex methods of adjustment and advanced topics. Statistics in Medicine 2020 Jul 20;39(16):2232-2263. Han K, Lumley T, Shepherd BE, Shaw PA. Two-phase analysis and study design for survival models with error-prone exposures. Stat Methods Med Res. 2020 Dec 16; PubMed Central PMCID: PMC8715910. Shaw PA. Use of composite outcomes to assess risk-benefit in clinical trials. Clin Trials. 2018 Aug;15(4):352-358. PubMed PMID: 30021496. Scope Dr. Shaw will provide expertise in measurement error and missing data in studies of longitudinal EHR outcomes following the rigorous frameworks for designing and analyzing observations studies as outlined in the international STRATOS guidance about statistical methods to minimize measurement error and her previous work on two-phase analyses and study designs to address error-prone electronic health records data. This will include reviewing study protocols and statistical analysis plans that rely on EHR data that are subject to missingness and measurement error, especially about use of VR/Immersive technologies that are utilized through a combination of in-clinic and at-home sessions. Dr. Shaw will also provide specific expertise in designing future prospective study protocols and data collection strategies for future evaluations of VR/Immerstive technologies that address potential residual sources of bias associated with measurement error and missing data. Dr. Shaw will work with the analytics team to review available covariate information for patients engaging in VR therapies. Based on observed patterns, she will support two efforts: 1) Provide guidance for prospective data collection efforts to improve precision in specific sub-populations about factors most likely to influence the causal quantities of interest. 2) Provide guidance regarding robust and efficient analytic approaches that leverage auxiliary EHR data with outcome data only available on a subset (e.g. patent reported outcomes). This contract will strengthen analytic accuracy, efficiency, and the credibility of evaluation outcomes. Tasks and Deliverables Task 1: Collaborate with VA Puget Sound study team to develop study protocols and statistical analysis plans. Ensure development of any necessary statistical analysis plans are in line with STRATOS/STROBE guidelines. Task 2: Review EHR outcome definitions, data extraction algorithms, and subset of patient reported outcome data available from specific immersive pilot Task 3: Guide sampling procedures for future prospective data collection efforts among patient subgroups for specific immersive pilot initiatives to supplement EHR outcomes with supplementally collected patient reported outcomes. Task 4: Code review of VA Puget Sound study analysts and replication of analytic findings to ensure reproducibility and quality control. Task 5: Feedback and review of VA Puget Sound analytic team technical reports detailing internal evaluation findings. Some internal reports may be of interest to broader healthcare community and, if appropriate, will be prepared for peer-review and dissemination through journal publications. Dr. Shaw will serve as a methodologic expert and co-author in manuscript preparation. Deliverable Description Delivery Date 1 Development of study protocols and statistical analysis plans in accordance with STRATOS/STROBE guidelines. As needed 2 Review of EHR outcome definitions, data extraction algorithms, and available patient-reported outcome data from immersive pilot initiatives. As needed 3 Provide sampling recommendations for prospective patient subgroup data collection efforts to supplement EHR outcomes with patient-reported outcomes. As needed 4 Code review and independent replication of analytic findings with written documentation of review outcomes. As needed 5 Review of and feedback on internal technical reports; co-authorship of manuscripts prepared for peer-review submission, as appropriate. As needed 6 Virtual and in-person meeting attendance. As needed Performance Monitoring Dr. Zeliadt will provide oversight of the work, which will be measured by meeting project deadlines in a timely fashion and regularly assessed quality of the…