Take a look at the Recent articles

Abstract

Background and objective: The ABRA Smart BT Blood Glucose Monitoring System (BGMS) was assessed in a clinical user performance study.

Method: In the clinical trial (N=103), untrained subjects with diabetes tested capillary fingertip blood samples and completed an ease-of-use questionnaire. BGMS and YSI glucose analyzer results were compared based on ISO 15197:2015 section 8 accuracy criteria.

Result: In the study, 99% of capillary fingertip results from subjects met ISO 15197:2015 section 8 accuracy criteria. 99% of subject capillary fingertip results were within Zone A in the consensus error grid analysis. Questionnaire results indicate a high usability and consistent user-friendliness.

Conclusion: The BGMS exceeds ISO 15197:2015 section 8 accuracy criteria in the hands of untrained users with diabetes.

List of Abbreviations: BGMS: Blood Glucose Monitoring System; SMBG: Self-Monitoring of Blood Glucose, MARD: Mean Absolute Relative Deviation; ISO: International Organization for Standardization

Keywords

diabetes mellitus, system accuracy, performance evaluation by the user, bias, ad-hoc confidence metric, DIN EN ISO 15197

Introduction

Maintaining stable blood glucose levels throughout the day is one of the main goals of modern diabetes therapy in order to prevent associated long-term complications. As part of the disease’s self-management, monitoring of blood glucose (SMBG) using Blood Glucose Monitoring Systems (BGMS) is a crucial component of attaining glycemic targets and identifying hypo-and hyperglycemic episodes. SMBG empowers patients to monitor and actively manage their condition, enabling them and their healthcare providers to make informed treatment decisions. As therapeutic decisions should always be founded on the actual glycemic status, accurate and reliable measurements are especially important for insulin therapy. A continuous and standardized post-market surveillance of BGMS guarantees the correct functioning, safeguards patient safety, and ensures regulatory compliance. User performance studies play a pivotal in assessing the suitability of BGMS for everyday use and improving patient care by assessing patient-related factors and inaccuracies which along analytical factors contribute to the overall accuracy of BGMS. This study evaluated the BGMS ABRA Smart BT (Diagnosis S.A.) in a clinical trial setting in accordance with the specifications outlined in DIN EN ISO 15197:2015 [1], section 8. The main objective was an assessment of the analytical accuracy and ease of use for individuals with diabetes without prior experience with the devices.

Materials and methods

Clinical trials

All tests were conducted at the institute for diabetes Karlsburg facilities between August and September 2024 after Institutional Review Board approvals were obtained for trial protocols, informed consent and other study forms. The clinical trial was conducted in accordance with DIN EN ISO 15197:2015 section 8-Performance evaluation by the user and enrolled a total of 103 people with type 1 and type 2 diabetes aged ≥ 18 years, amounting to 100 assessable data sets after exclusions. Study participants were limited to individuals without prior experience with the device to focus the impact of the device's design and accompanying materials on usability, eliminating the confounding factor of learned behaviors or biases from previous BGM use. Additional exclusion criteria included pregnancy and/or lactation, and conditions that might impose an additional risk towards the patient or study personnel. Participants were first informed about the objective, procedure, potential risks, and expected duration of the study. After declaration of willingness to participate in the study, a written and signed informed consent from the volunteer was requested and medications and/or dietary supplements were recorded. Participants were excluded if they reported using medications or supplements listed in Appendix A of DIN EN ISO 15197:2015 and their measurements were found to be out of specifications. All participants were provided with the test system, user- and/or quick reference guide and were given sufficient time to familiarize with the device and if necessary perform up to three trial runs. Test subjects obtained capillary blood for glucose testing by fingertip puncture using a disposable lancet. Immediately after self-measurement, a sample of capillary blood was taken by trial staff for the determination of hematocrit and blood glucose using the reference method.

Performance evaluation by the user

Following testing phase, the subjects were asked to complete an ease-of-use questionnaire to evaluate the user experience including factors such as the device’s usability, ease of operation, maintenance, and clarity of instructions. Responses were based on a 5-point Likert scale. No weights were applied.

Assessment of analytical accuracy

Subject’s BGMS results were compared with laboratory reference results from a YSI 2300 stat plus glucose and lactate analyzer (Glucose Oxidase, YSI Xylem Brand) using a trial staff-obtained blood sample (Table 1). All measurements were performed using plasma and presented as plasma equivalents. Accuracy was assessed based on DIN EN ISO 15197:2015 guidelines (i.e., ≥ 95% within ± 15 mg/dL / % of mean reference results) and included regression analysis, construction of modified Bland-Altman plots, Consensus-Error Grid for Diabetes Type I analysis [2], as well as assessment of mean absolute relative deviation MARD, BIAS and an ad-hoc confidence metric, i.e., the narrowest error margin comprising at least 95% of meter inaccuracies [3].

Glucose	Within specified error limits			Performance metrics
	± 5 mg/dL	± 10 mg/dL	± 15 mg/dL	± 15 mg/dL / 15 %	BIAS	MARD	ad-hoc
< 100 mg/dL (n=13)	7 (53.8%)	10 (76.9%)	13 (100%)
68.5-357 mg/dL	± 5 %	± 10 %	± 15 %	99 /100 (99%)	-2.7	6	14.3 mg/dL / %
≥ 100 mg/dL (n=87)	38 (43.7%)	74 (85.1%)	86 (98.9%)

Table 1. Summary of blood glucose monitoring system results from assessment of analytical accuracy according to DIN EN ISO 15197:2015 and additional performance metrics

Results

Analytic accuracy

After exclusion of 3 participants due to intake of substance(s) and/or underlying medical conditions that may interfere with blood glucose measurements as outlined in DIN EN ISO 15197:2015, Appendix A, a total of 100 evaluable measurements and questionnaire user-feedbacks were obtained. Assessment of analytical accuracy in self-application showed that the system fully complied with ISO 15197:2015 acceptance criteria. In the range of 68.5-357 mg/dL, 99% of measurement results were within ± 15 mg/dL or 15% of the reference (Table 1, Figure 1A). Consensus error grid analysis showed that 99% of test results were in zone A (no effect on clinical action) with a single measurement each in zone B (little or no effect on clinical outcome Figure 1B). The ad-hoc confidence metric showed that 95% of meter inaccuracies fell within 14.3 mg/dL or 14.3 %.

Figure 1. Bland-Altman-plot showing deviation from blood glucose monitor glucose to reference measurements (A) and Consensus-Error-Grid for Diabetes Type I (B) including regression analysis (cyan). Nonconforming measurements are depicted with red circles

Performance evaluation by user

Participant demographics are summarized in Figure 2. Subjects were 23-79 years of age (mean: 59 years), 55% female, from a broad educational level ranging from no formal educational attainment (2%) to doctorate/habilitation (1%) with the majority having completed vocational training (52%). All users were able to operate the new BGMS without prior training or use of additional trial runs. The generally high acceptance ratings for design, functionality and operability indicate high usability and consistent user-friendliness. However, some shortcomings and areas of potential improvement identified in the study led to a substantial decrease in user satisfaction (Figure 3). One aspect that was consistently criticized is changing batteries and/or recharging as well as menu navigation of the device. Also, the general layout and size, font type, and font size of the accompanying quick- and user guide materials frequently resulted in point deductions.

Figure 2. Overview of demographic characteristics for the study showing gender distribution (top left), age distribution (top right) and educational levels (bottom)

Figure 3. Results of the ease-of-use questionnaire. Percentages represent the proportion of subjects responding with ‘‘very positive’’ and ‘‘positive’’

Discussion

The continuous post market follow-up set forth in the European In Vitro Diagnostics Regulation [4], ensures continuity in safety, clinical performance and user experience. The system described in this study was found to be generally user-friendly, with high scores in ease of learning and operation, highlighting the successful interplay of technological improvements and normative compliance.                

The BGMS demonstrate excellent accuracy, complying with and exceeding the minimum performance specifications set forth in DIN EN ISO 15197:2015 as demonstrated by the consensus error grid analysis with all but one measurement lying in zone A, making them suitable for routine self-monitoring, also for people without medical expertise [5,6]. Reliability in terms of accuracy and precision as well as usability of blood glucose monitoring systems are paramount for effective diabetes self-management. In particularly among lay patients who may have varying levels of technical proficiency, incomplete or misleading instructions that make a new and unfamiliar BGMS difficult to use increase the risk of patient errors during blood glucose testing. Inaccurate results can seriously compromise patient safety by leading to incorrect or even potentially harmful therapeutic treatment decisions, such as failing to address hypo- or hyperglycemia, or in the worst-case scenario of administrating a wrong insulin dose [7,8]. This underscores the importance of incorporating human factors and usability engineering principles into the development and evaluation of BGMS. Studies have consistently demonstrated a strong link between device usability and adherence to diabetes management protocols [9]. By prioritizing user-centered design and actively soliciting feedback from patients, manufacturers can create BGMS that are intuitive, easy to use, and contribute to improved glycemic control. Furthermore, research emphasizes the need for clear instructions and training materials to support effective BGMS use, especially for individuals with limited health literacy [10,11]. Considering the study population and the limited extent of the demographic data, we found no correlation between patients' health literacy and the device's system accuracy.

While the devices’ design and haptics may reflect personal preferences, other technical aspects that were consistently criticized (i.e., changing batteries and/or recharging as well as menu navigation of the device) should be addressed by the manufacture as a devices’ usability and patient adherence are inextricably linked. Given the potential negative impact on user adherence, ranging from discouragement, making errors, to abandoning its use entirely, prioritizing usability in medical device design is not only about convenience. Also, the correlation between point deductions in user performance evaluations, specific participant complaints, and the patient collective's age distribution highlights the challenge of creating a universally accepted device. Rather than seeking a "one-size-fits-all" solution, manufacturers should focus on tailoring their devices and accompanying materials to the specific requirements of their target demographic (e.g., adjusting IFU font size for older users).

Conclusion

In summary, notwithstanding the identified shortcomings the device's accuracy, usability, and the clarity of its instructions appear to have been influential factors in its positive reception and general acceptance.

Conflict of interest

There are no conflicts of interest.

Funding

All authors are employees of the institute of diabetes Karlsburg GmbH, Germany, which carries out studies evaluating blood glucose meter systems on behalf of various clients. The author(s) received no financial support for the research, authorship, and/or publication of this article.

Acknowledgement

 The authors would like to thank the personnel of the institute of diabetes GmbH for providing technical help, intellectual input and feedback in conducting the studies, and preparing the manuscript. Sponsors were permitted to review and comment on the manuscript, final decision on content was retained by the authors.

References

1. International Organization for Standardization. In vitro diagnostic test systems: Requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus. ISO. 2003.
2. Parkes JL, Slatin SL, Pardo S, Ginsberg BH (2000) A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care 23: 1143-1148. [Crossref]
3. Kenning M, Puchert A, Salzsieder E (2025) Comparative system accuracy of blood glucose monitoring systems-advocacy for a new accuracy assessment metric. J Diabetes Sci Technol 19: 270-271. [Crossref]
4. Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in vitro diagnostic medical devices and repealing Directive 98/79/EC and Commission Decision 2010/227/EU. 
5. Bode BW (2007) The accuracy and interferences in self-monitoring of blood glucose. US Endocr Dis Touch Brief 2: 46-48.
6. Hirsch IB, Bode BW, Childs BP, Close KL, Fisher WA, et al. (2008) Self-Monitoring of Blood Glucose (SMBG) in insulin-and non-insulin-using adults with diabetes: Consensus recommendations for improving SMBG accuracy, utilization, and research. Diabetes Technol Ther 10: 419-439. [Crossref]
7. Hellman R (2012) Glucose meter inaccuracy and the impact on the care of patients. Diabetes Metab Res Rev 28: 207-209. [Crossref]
8. Rebel A, Rice MA, Fahy BG (2012) The accuracy of point-of-care glucose measurements. J Diabetes Science Technol 6: 396-411. [Crossref]
9. Toletti G, Boaretto A, Di Loreto C, Fornengo R, Gigante A, et al. (2024) Enhancing diabetes therapy adherence: A comprehensive study on glucometer usability for type 2 diabetes patients. Front Clin Diabetes Healthc 5: 1328181. [Crossref]
10. Berkman ND, Davis TC, McCormack L (2010) Health literacy: What is it?. J Health Commun 15: 9-19. [Crossref]
11. Marciano L, Camerini AL, Schulz PJ (2019) The role of health literacy in diabetes knowledge, self-care, and glycemic control: A meta-analysis. J Gen Intern Med 34: 1007-1017. [Crossref]