Usability Assessment of a Novel Diabetes Risk Stratification Tool

Sergio Eslava, MD1, Cheryl Pegus, MD, MPH1, Deborah Guattery, MJ, BSN, RN1, Tom Schaible1, Robert Bernstein, MD2, Jerome Fischer MD3 1SymCare, Personalized Health Solutions, West Chester PA; 2DGD Clinic, San Antonio TX; 3Reg. Endocrin. Assoc., Santa Fe NM

Objective
Blood Glucose (BG) Variability, a measure of a person's hypo and hyperglycemic excursions over time, was identified by the Diabetes Control and Complications Trial (DCCT) as a good predictor of Diabetes complications, especially cardiovascular events1. Out of the methods used to measure BG Variability, only a few have shown to be sensitive to both hypo and hyperglycemic events. One of them is ADRR (Average D il Ri k R Daily Risk Range), an algorithm d (A ) l ith developed b K l d by Kovatchev et al f t h t l from U. of Virginia that computes an index from a person's Self Monitoring Blood Glucose (SMBG) data along with a Risk Stratification Group (Low/Moderate/High) which is highly correlated with the incidence of Diabetes complications2. Our aim was to assess the usability of a software implementation of ADRR in patients with Diabetes in a clinical setting.

Methods (cont.)
The likert questions were scored as "Usable" if the lower 90% confidence limit was greater than or equal to 70%, or as "Not Usable" otherwise. The results were then aggregated by category. The responses to the free-text questions were qualitatively analyzed to extract meaningful trends.

Results
7 out of the 8 categories were scored as "Usable based on the criteria defined on the Statistical Analysis Plan, that is, a lower 90% confidence limit greater than or equal to 70% (Table 2).
Number of Valid (nonmissing) Responses 9 9 9 9 5 9 9 9

Methods
28 participants (patients with Type 1 or 2 Diabetes) and 3 physicians (Endocrinologists) in 2 sites (Diabetes clinics) were included in the study. We built a software application for the study, the ADRR Glycemic Variability Calculator (ADRR-GVC), which uses ADRR to compute the index and risk group from SMBG and presents them to physicians using bar charts to help them observe BG Variability trends (Figure 1).

Category Visual Appeal Content Ease of Use Performance Support Clinical Relevance Clinical Usefulness Overall Impression

Lower 90% Upper 90% Confidence Confidence Bound Mean Score Bound 0.85 0.63 1 1 0.87 0.9 0.87 0.78 0.93 0.78 1 1 0.96 0.96 0.93 0.87 1 0.92 1 1 1 1 1 0.95

Usability Usable Not Usable Usable Usable Usable Usable Usable Usable

Table 2: 90% Confidence Interval for the Usability Questionnaire by Category

The only category that was scored as "Not Usable" was "Content, which resulted from 2 low-scored questions related to system completeness by 1 physician who in his comments pointed out that he wanted to see additional functions on the software, namely the integration of ADRR into a well known DMS (Diabetes Management System) and an improved color scheme Among the features most appreciated by physicians were the bar charts to compare results over time and the use of ADRR as a quantitative risk estimator The Areas to change, as identified by physicians, were the current color scheme and a suggestion to include the results from the most recent A1c test Additional functionality suggested by physicians included to display the most recent A1c result and the number of SMBG readings used to compute the ADRR index
Figure 1: Screenshot from the ADRR Glycemic Variability Calculator (ADRR-GVC) displaying the results from the ADRR algorithm

Participants attended 3 visits (days 0, 30 and 60), during which physicians used ADRR-GVC as described. At the end of the study physicians completed a Usability Questionnaire with 24 likert questions in 8 categories (Visual Appeal, Content, Ease of Use, Performance, Support, Clinical Relevance, Clinical Usefulness and Overall Impression) plus 5 free-text questions (areas most liked/disliked/would change/ add/ most helpful). (Table 1).

Conclusions
The ADRR algorithm is an important risk stratification tool that offers clinically meaningful information to physicians treating patients with Diabetes who regularly perform SMBG ADRR can be implemented as a graphic rich software application with a high graphic-rich level of usability by physicians, who want to see ADRR included in other DMS tools. Given the small sample size it may be advisable to conduct further studies in a larger population to confirm results.

References
1. The Diabetes Control and Complications Trial Research Group. Diabetes 44:968-983, 1995 2. Kovatchev BP, Otto E, Cox D, Gonder-Frederick L, Clarke W. Evaluation of a new measure of blood glucose variability in diabetes. Diabetes Care. 2006 Nov;29(11):2433-8

Table 1: Study Timeline