Assessing the usability of complex psychosocial interventions: The Intervention Usability Scale

Abstract Background. Usability – the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction – is a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.Method. A survey administered to 136 medical professionals from 11 primary care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.Results. Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.Conclusions. This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Download Full-text

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

10.21203/rs.3.rs-22305/v2 ◽

2020 ◽

Author(s):

Aaron R Lyon ◽

Michael D. Pullmann ◽

Jedediah Jacobson ◽

Katie Osterhage ◽

Morhaf Al Achkar ◽

...

Keyword(s):

Primary Care ◽

Health Service ◽

Service Delivery ◽

Behavioral Health ◽

Psychosocial Intervention ◽

Primary Care Providers ◽

Health Service Delivery ◽

Behavioral Health Service ◽

Health Interventions ◽

Care Providers

Abstract Background. Usability – the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction – is a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.Method. A survey administered to 136 medical professionals from 11 primary care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.Results. Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.Conclusions. This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Download Full-text

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

10.21203/rs.3.rs-20758/v2 ◽

2020 ◽

Author(s):

Aaron R Lyon ◽

Michael D. Pullmann ◽

Jedediah Jacobson ◽

Katie Osterhage ◽

Morhaf Al Achkar ◽

...

Keyword(s):

Primary Care ◽

Health Services ◽

Behavioral Health ◽

Implementation Research ◽

Psychosocial Interventions ◽

Health Interventions ◽

Evidence Based ◽

Implementation Success ◽

System Usability Scale ◽

System Usability

Abstract BackgroundUsability – the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction – is a key determinant of implementation success. However, usability is rarely assessed in implementation research and no instruments have been developed to measure the design quality of complex health interventions, such as the evidence-based psychosocial interventions that characterize the majority of effective practices in mental and behavioral health services. This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale for digital technologies, when measuring the usability of complex health interventions. Prior studies of the original System Usability Scale have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.MethodsA survey was administered to 205 healthcare professionals working at 11 primary care sites. Surveys collected demographic information, including each participant’s professional role (i.e., medical provider, mental/behavioral health provider, pharmacist), and IUS ratings for one of six common evidence-based psychosocial interventions (e.g., cognitive behavioral therapy, motivational interviewing) that they reported using most regularly. Factor analyses replicated the procedures used in prior research on the System Usability Scale, and a sensitivity analysis using analyses of variance compared IUS scores across different groups of respondents and interventions assessed.ResultsAnalyses indicated that a two-factor solution (with “usable” and “learnable” subscales) in which one item was removed best fit the data. This solution accounted for 52.6% of the variance observed. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .82, and α = .63, respectively. Resulting scores indicated that usability ranged from below acceptable standards to good, depending on the intervention. On average, behavioral health providers found the interventions to be more usable that other types of healthcare providers.ConclusionsThe current study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability and sensitivity to role and intervention. Future directions for implementation research evaluating the usability of complex health interventions are discussed.Contributions to the Literature• The ease with which interventions can be readily adopted by service providers is a key predictor of implementation success, but very little implementation research has attended to intervention usability.• No instruments exist to evaluate the usability of complex health interventions, such as the evidence-based practices that are commonly used to integrate mental and behavioral health services into primary care.• The current study evaluated the first instrument for assessing the usability of complex health interventions and found that its factor structure replicated some research with the original version of the instrument, a scale developed to assess the usability of digital systems.

Download Full-text

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

10.21203/rs.3.rs-22305/v1 ◽

2020 ◽

Author(s):

Aaron R Lyon ◽

Michael D. Pullmann ◽

Jedediah Jacobson ◽

Katie Osterhage ◽

Morhaf Al Achkar ◽

...

Keyword(s):

Primary Care ◽

Health Services ◽

Behavioral Health ◽

Implementation Research ◽

Psychosocial Interventions ◽

Health Interventions ◽

Evidence Based ◽

Implementation Success ◽

System Usability Scale ◽

System Usability

Abstract BackgroundUsability – the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction – is a key determinant of implementation success. However, usability is rarely assessed in implementation research and no instruments have been developed to measure the design quality of complex health interventions, such as the evidence-based psychosocial interventions that characterize the majority of effective practices in mental and behavioral health services. This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale for digital technologies, when measuring the usability of complex health interventions. Prior studies of the original System Usability Scale have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.MethodsA survey was administered to 205 healthcare professionals working at 11 primary care sites. Surveys collected demographic information, including each participant’s professional role (i.e., medical provider, mental/behavioral health provider, pharmacist), and IUS ratings for one of six common evidence-based psychosocial interventions (e.g., cognitive behavioral therapy, motivational interviewing) that they reported using most regularly. Factor analyses replicated the procedures used in prior research on the System Usability Scale, and a sensitivity analysis using analyses of variance compared IUS scores across different groups of respondents and interventions assessed.ResultsAnalyses indicated that a two-factor solution (with “usable” and “learnable” subscales) in which one item was removed best fit the data. This solution accounted for 52.6% of the variance observed. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .82, and α = .63, respectively. Resulting scores indicated that usability ranged from below acceptable standards to good, depending on the intervention. On average, behavioral health providers found the interventions to be more usable that other types of healthcare providers.ConclusionsThe current study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability and sensitivity to role and intervention. Future directions for implementation research evaluating the usability of complex health interventions are discussed.Contributions to the Literature• The ease with which interventions can be readily adopted by service providers is a key predictor of implementation success, but very little implementation research has attended to intervention usability.• No instruments exist to evaluate the usability of complex health interventions, such as the evidence-based practices that are commonly used to integrate mental and behavioral health services into primary care.• The current study evaluated the first instrument for assessing the usability of complex health interventions and found that its factor structure replicated some research with the original version of the instrument, a scale developed to assess the usability of digital systems.

Download Full-text