| Questions About Measuring Percent Useful Outcomes |  | |
General
Why did L&I choose to measure percent useful outcomes?
This measure could be implemented quicker and was sufficiently valid to remove Complexity-Adjusted Cost Outcome (CACO). Percent useful outcomes was one of the proposals by the Methods for Assessing Quality (MAQ) teams (119 KB PDF).
This measure is intended to be a general indicator of quality and effectiveness and used in conjunction with other aspects of vocational quality. It is also aligned with L&I's internal measures for program effectiveness.
Get Help Downloading Files (files open in a new window).- When will claim managers be trained on percent useful outcomes?
Claim managers will be trained prior to October 1, 2008. Emphasis will be placed on using the measure as a general indicator of quality and effectiveness in conjunction with the Performance Criteria & System Values (draft) (59 KB PDF).
In early 2009, a status report will provide additional information about quality and effectiveness of vocational services to support referral decisions. This was another proposal from the MAQ Teams (119 KB PDF).
Get Help Downloading Files (files open in a new window). - How can I improve my percent useful outcomes?
Many tools, forms, and guidelines are available from L&I to help you. In addition, the Performance Criteria (59 KB PDF) describe quality and effectiveness such as:
- Resolve vocational issues.
- Manage cases proactively.
- Communicate effectively and courteously.
- Build constructive relationships.
- Understand and apply the relevant laws and rules.
You can also ensure a manageable caseload size by temporarily stopping referrals. Contact Private Sector Rehabilitation Services (PSRS) provider accounts to temporarily stop referrals.
Get Help Downloading Files (files open in a new window). - How is this measure fair if the referral is complex?
A useful outcome can usually be obtained regardless of an injured worker's level of education or skills, the severity of the injury, the local labor market, or training options. Generally an injured worker can return to work, be found able to work, be eligible for further services, or not be eligible for further services. Handling complex cases in a fair manner was identified as 1 of the advantages of this measure by the MAQ team.
- When will VRCs be trained on percent useful outcomes?
This measure was presented at the Vocational Technical Stakeholders Group (VTSG), L&I's Spring Vocational Conference, sent to a Vocational listserv, and posted on the Internet. Information will also be presented when education for Phase 2 is provided.
- How long has L&I had a focus on useful outcomes?
L&I has consistently communicated the importance of obtaining useful outcomes on vocational referrals for many years. In November 2007, the MAQ team proposal for using the percent useful outcomes to replace CACO was shared at the Fall Vocational Conference. In early 2008, L&I reemphasized its focus on useful outcomes during implementation of the new Vocational Improvement Project (VIP) legislation. In May 2008, the decision to replace CACO with a measure of percent useful outcome was announced via the Internet and listserv. On June 6, 2008, the measure with the updated designations was presented at the Spring Vocational Conference.
Referral outcomes codes
- Which outcome codes are considered "useful" for the new measure?
Useful outcomes include return to work, able to work, eligible for further services, or not eligible for further services. The Designations of Codes (29 KB PDF) identify closing codes as useful, not useful, or neutral.
These designations were updated June 2008 to better reflect Vocational Rehabilitation Counselor (VRC) performance and are based on comments made during Methods for Assessing Quality (MAQ) team meetings and feedback from other stakeholders.
Get Help Downloading Files (files open in a new window). - Which closing code is used in the measure?
The outcome code entered by an L&I claim manager or vocational services specialist (VSS) will be used in the calculation.
- Why is the L&I closing code used instead of the code entered by the VRC?
A VRC provides a recommendation to L&I who is responsible for a legal, adjudicative decision about the referral. In most instances (about 90% of referrals), these outcomes are the same. When they aren't the same, it may be that additional information was needed or that other considerations were factored into an adjudicative decision.
Closures included in a quarterly report
- Is there a minimum number of closures needed before a score is calculated?
Yes. A Vocational Rehabilitation Counselor (VRC) or branch must have 10 or more closures within the sample period before a score will be calculated. Only closures designated as useful or not useful count towards the minimum of 10 closures.
- Which referral outcomes will be considered in the reports?
Referrals closed by L&I within the sample period and designated as "useful" or "not useful" will be used to calculate the percent useful outcome.
Since the first sample period begins June 15, 2008, this will provide just enough data to allow the removal of Complexity-Adjusted Cost Outcome (CACO) on October 1, 2008.
Since new data will be added each quarter, the impact of this first sample period will decline in future quarterly reports.
VOCC display of vocational providers
- What information is on the vocational referral screen (VOCC)?
The VOCC screen shows:
- Percent useful outcome score.
- Number of open referrals.
- Name of a Vocational Rehabilitation Counselor (VRC) or firm.
- Provider ID number.
- Branch number.
- City.
- How will the information be sorted?
Vocational providers will show in descending order of percent useful outcome. If 2 or more providers have the same percentage, they will be listed in random order.
- How do vocational providers show if they have less than 10 closures?
Vocational providers with less than 10 closures won't have a percentage. They will be listed with a blank space and in random order.
Important dates
- When will the first report be available?
October 1, 2008.
- How often will reports be updated?
Reports will be updated quarterly on the first day of January, April, July, and October.
- When does a referral appear in the report?
Referrals closed by L&I within the sample period will be considered in the report. Typically, L&I closes referrals in less than 15 days.
Calculating the percent useful outcome
- How is the percent useful outcome calculated?
- Count the number of "useful" outcomes.
- Total the number of "useful" and "not useful" outcomes.
- Divide the number of "useful" by the total number of "useful" and "not useful" outcomes.
Example: A Vocational Rehabilitation Counselor (VRC) has 15 referrals closed as "useful" and 3 referrals closed as "not useful."
- The number of "useful" outcomes is 15.
- The total number of "useful" and "not useful" outcomes is 18 (15 + 3).
- The percent useful outcome is 15 ÷ 18 = 83%.
- How do "neutral" outcomes affect scoring?
Outcomes designated as "neutral" are excluded from the calculation to ensure they are truly neutral in the scoring. Otherwise, it creates an incorrect and lower score.
- How can I find out about closed referrals?
The RVRS screen in VocLink has a list of referrals that have been closed by L&I under your active provider number. Open a referral and the L&I closing code is in the lower left after "VOCATIONAL OUTCOME CODE."
Aggregation of referrals
- What is aggregation?
Aggregation is when all closed referrals within the sample period are considered.
- What referrals are aggregated for a Vocational Rehabilitation Counselor (VRC) score?
All referrals assigned to the VRC and closed within the sample period are aggregated.
- What referrals are aggregated for a branch score?
All referrals assigned to the VRC are aggregated to the branch the VRC is attached to at the end of the sample period.
- Why are branch totals sometimes greater than the cumulative totals of VRCs listed for that branch?
If a VRC has only 1 closure they won't listed individually but the closure will be included in the branch total.
About specific outcome codes
- Medically Unstable (ADM1) - How can Vocational Rehabilitation Counselors (VRCs) influence this outcome?
Review new referrals to ensure they're ready for vocational work. If the referral isn't ready for vocational work, the VRC should decline the referral with an ADMA closure.
After a VRC accepts a referral, timely communication and working to resolve relevant vocational issues may influence the outcome.
Although neither VRCs nor L&I have complete control over the medical condition of an injured worker, this code is also a "not useful" outcome for L&I's internal measures for program effectiveness.
- Fee cap (ADM7) - Is the fee cap for Assessment referrals set at the right level?
L&I is evaluating evidence regarding the fee cap for Assessment referrals.
- Plan deemed approved (PLN4) - How is this different than Plan approved (PLN2)?
Plans are approved (PLN2) after an L&I vocational service specialist (VSS) has reviewed a proposed plan and decides the necessary elements are in place. The new Vocational Improvement Project (VIP) law requires L&I to either approve or deny plans within 15 days. If L&I doesn't act timely, the plan is deemed approved (PLN4). This code was applied to less than 0.1% of plans during the first five months of the VIP implementation.
- Plan failed (PLN8) - How can VRCs influence this outcome?
VRCs can significantly influence successful completion of a plan through proactive case management, timely communication, and building constructive relationships. The new VIP legislation offers closer evaluation of plans prior to approval, higher training benefits, an opportunity for injured workers to select Option 2, and closer monitoring during Plan Implementation.
Although neither VRCs nor L&I have complete control, this code is also a "not useful" outcome for L&I's internal measures for program effectiveness.
