Annual Portfolio Assessment Expert Review Pilot Employment Portfolio Panel Summary Report Prepared by: New Editions Consulting, Inc. 6858 Old Dominion Dr., Suite 230 McLean, VA 22101 January 4, 2006 Annual Portfolio Assessment Expert Review Pilot Employment Portfolio Panel Summary Report Table of Contents Executive Summary 3 Findings and Recommendations 3 Section 1: Overview of the APAER Process 5 1.1 APAER Purpose 5 1.2 Pilot Goals and Design Challenges 6 1.3 Procedures 6 1.3.1 Portfolio Composition 6 1.3.2 Inputs to the Process 7 1.3.3 Panel Composition 8 1.3.4 Assessment Process 8 1.3.5 Meeting Agenda 9 Section 2: Portfolio Performance 10 2.1 Cluster Level Results and Critique 10 2.1.1 Results 10 2.1.2 Inventory of Accomplishment Nuggets 11 2.1.3 Cluster Level Reports to Panel 12 2.2 Portfolio Level Results 12 Section 3: NIDRR Management of Portfolio 18 3.1 Portfolio Level Results 18 3.2 Closing Discussion 20 Section 4: Feedback on APAER Process 22 4.1 Portfolio Level Results 22 4.2 Cluster Level Comments 22 Section 5: Summary of Discussion 24 5.1. Implications for the Employment Portfolio 24 5.2 Implications for the APAER Process 24 5.3 Grantee Comments 25 APPENDIX A: NIDRR PERFORMANCE MEASURES 26 APPENDIX B: PANEL MEMBERS 35 APPENDIX C: AGENDA 40 APPENDIX D: ACCOMPLISHMENT NUGGETS BY EMPLOYMENT CLUSTERS 44 APPENDIX E: PARTICIPANT EVALUATION SUMMARY 52 Executive Summary The National Institute on Disability and Rehabilitation Research (NIDRR) conducted its Annual Portfolio Assessment Expert Review (APAER) Process Pilot for the Employment Portfolio on September 29 and 30, 2005, in Washington, D.C. The APAER process was developed by NIDRR to assess the agency's progress in meeting Federal performance requirements under the Government Performance and Results Act (GPRA) and the Program Assessment Reporting Tool (PART). The PART is a systematic method of assessing and improving program performance across the Federal government, instituted by the Office of Management and Budget (OMB). The APAER process was designed as an external expert review of NIDRR grantee accomplishments using a three-year cycle, in which one-third of the overall research agenda is reviewed each year. NIDRR's Employment Portfolio was the first review completed under the 2005 pilot phase of this new process. A panel of nine researchers, consumers, clinicians, policy experts and administrators, reviewed NIDRR's employment portfolio, based on reports from 16 grantees with active awards in 2004, across three program mechanisms: Rehabilitation Research and Training Centers (RRTCs), Disability and Rehabilitation Research Projects (DRRPs), and Field Initiated Projects (FIPs). The 16 awards that were reviewed were grouped into two topical clusters based on the focus of the award: addressing systems level issues and addressing individual level issues. The review covered: * Objective 1: The quality and relevance of NIDRR-funded research and the extent to which outputs and outcomes are contributing to the agency's long-term performance measures and strategic goals. * Objective 2: The strengths and weaknesses of the employment portfolio as a whole, including recommendations to improve the portfolio. And, * Objective 3: The quality and relevance of the agency's management of research directions and award decisions. Since this was a pilot process, data were also collected to assess the feasibility of the process design and implementation. Findings and Recommendations The panel provided frank evaluations and recommendations to NIDRR in three areas: (1) the Employment portfolio performance, (2) NIDRR management, and (3) the APAER process. Overall, the panel expressed concern about performance of the portfolio and the soundness of the evaluation and review process. Given the pilot nature of the 2005 APAER process, care must be taken in interpreting and applying the findings contained in this report on NIDRR's Employment portfolio. Portfolio Performance. Panelists identified a number of outputs and outcomes that appeared to be important to improving employment services, including projects that focused on employment for culturally diverse populations and aging populations, and projects focused on telework and other important accommodations. Overall, the panel found the research ideas in the portfolio to be of very high quality. However, they expressed concern about: (1) the apparent lack of scientific rigor behind the identified outputs, (2) the lack of sufficient information on the methodologies used by grantees; (3) the lack of evidence supporting many of the claims made by grantees in their Supplemental Information Reports, and (4) the lack of peer reviewed publications. In the area of knowledge translation, the panel was uncertain about the adequacy of consumer-oriented outputs. Some panelists believed the number of outputs was good, but could not rate the quality of these products based on the information provided. Many reviewers felt that grant activities had the potential to impact individuals with disabilities, but that these activities did not seem connected to the research. The panel concluded, that in terms of progress toward NIDRR's capacity-building performance measures, the Employment portfolio was limited. Many awards were judged lacking in the use of multidisciplinary teams and no controlled studies were identified. The panel could not judge the adequacy of awards to diverse individuals and institutions, nor the integration of research and training. Management of the Portfolio. Because persons with disabilities continue to be unemployed and underemployed in disproportionately high numbers, the panel recommended that: 1. NIDRR should consider whether this portfolio area is adequately funded. 2. NIDRR should reconceptualize the role of the FIPs and how results from FIPs could translate into other, larger, more comprehensive funding mechanisms within NIDRR. They felt that even though the science was exploratory, the push to disseminate often led to findings being presented as too definitive. 3. NIDRR should consider more cross-award and cross-agency work. 4. NIDRR should strengthen its unique role in distributing information to consumers. 5. NIDRR should provide more focus on employers and labor/demand side issues, as well as on empowerment and school-to-work transition. 6. NIDRR should consider using an advisory group to help determine priorities. The panel also suggested that NIDRR examine the unintended bias to continue funding the same centers vs. its role in funding new researchers. Critique of the APAER Process. In general, the panel commended NIDRR on its efforts to evaluate the Employment portfolio. However, panelists expressed many concerns about the process and their ability to accurately evaluate the portfolio, including: (1) the quality of the grantee reports, suggesting that perhaps grantees did not understand how to complete the online form, were confused by the terminology, or simply did not put much effort into the process; (2) receipt of under-processed data for review; and (3) the general lack of clarity in the definition of terms (multidisciplinary, publication, short-term outcome and intermediate outcome). Panelists suggested that project officers might provide more guidance to grantees in preparing reports. Annual Portfolio Assessment Expert Review Pilot Employment Portfolio Panel Summary Report The National Institute on Disability and Rehabilitation Research (NIDRR) conducted its Annual Portfolio Assessment Expert Review (APAER) Process Pilot for the Employment Portfolio on September 29 and 30, 2005, in Washington, D.C. This summary report presents a brief description of the APAER process and the findings and recommendations of the expert panel. Section 1: Overview of the APAER Process The APAER process was developed by NIDRR to assess its progress in meeting Federal performance requirements under the Government Performance and Results Act (GPRA) and the Program Assessment Reporting Tool (PART)1, and to capture progress of its grantees using a three-year cycle, with one-third of NIDRR's portfolio reviewed every three years. A team of experts reviewed NIDRR's Employment portfolio as a pilot of this new process. This first year is a baseline year for the Employment portfolio. Through this pilot, NIDRR intends to gather data to compare its baseline performance with results at the end of a specified time period in 2013. The challenges to NIDRR of APAER were numerous and included establishing an integrated and methodologically sound portfolio assessment and independent expert review process that would: * Be manageable with existing resources and consistent with emerging standards of practice for performance assessment in Federal R&D agencies, * Result in consistent and reliable data on NIDRR's long-term performance measures, and * Generate evidence-based recommendations for program improvement, consistent with the accountability and program management requirements of both GPRA and PART. 1.1 APAER Purpose The APAER was intended to provide NIDRR with a programmatic level, independent assessment of three objectives: * Objective 1: The quality and relevance of NIDRR-funded research and the extent to which outputs and outcomes are contributing to the agency's long-term performance measures and strategic goals, * Objective 2: The strengths and weaknesses of the Employment portfolio as a whole, including recommendations to improve the portfolio, and * Objective 3: The quality and relevance of the agency's management of research directions and award decisions within the portfolio. 1.2 Pilot Goals and Design Challenges The specific goals of the 2005 APAER pilots were to test: * Strategies to collect reliable and evidence-based information on significant outputs and outcomes, and * The feasibility of the design and implementation of this portfolio level assessment of grantee performance. In designing and conducting this pilot, NIDRR experienced a number of challenges. The key challenges follow. * Federal agencies have not yet agreed upon guidelines or methods for reporting outcomes and conducting portfolio level assessments, and NIDRR is one of only a handful of agencies attempting this approach. * The timing was premature for NIDRR in that the outcomes-oriented performance reporting system for its grantees is currently under development. * Grantees completed a Supplemental Information Form designed to collect outcomes-oriented data at the same time that their regular Continuation Reports were due. * Data collected from grantees in the Supplemental Information Form were sparse. * Grantees received little technical assistance to support outcomes planning and reporting, and therefore lacked knowledge (and practice) on reporting accomplishment nuggets, outputs and outcomes. 1.3 Procedures NIDRR's annual and long-term performance measures served as the basis of the process. These OMB-approved measures are listed in Appendix A. Key design features and steps in the process are summarized below. 1.3.1 Portfolio Composition For APAER, NIDRR grouped eligible awards into portfolios according to the applicable outcome arena of the Logic Model presented in its 2005 Long-Range Plan. For the research and development arena, NIDRR further subdivided these awards into five domains according to its Long-Range Plan. The criteria for inclusion of an award in the Employment portfolio were: employment related focus; active in 2004 and had completed at least one full year of work; and one of the following eligible program funding mechanisms-Rehabilitation Research and Training Center (RRTC), Disability and Rehabilitation Research Project (DRRP), or Field Initiated Project (FIP). For the Employment portfolio pilot, NIDRR identified 18 eligible awards. To facilitate the review process and reduce reviewer burden, NIDRR divided the awards into two clusters with a similar number of projects in each - awards addressing systems level issues and awards addressing individual level issues. 1.3.2 Inputs to the Process NIDRR developed an online supplemental information form for grantees to report retrospectively on outputs and outcomes between 2002-2004 corresponding to selected GPRA performance measures. Grantees had six weeks to complete the form, and submit electronic documentation of outputs and outcomes. They provided information on the multidisciplinary teams of investigators, study descriptions, a roster of fellows/trainees and doctoral students, journal publications, and nominations of up to two "best" accomplishment nuggets for each of three arenas: scientific publications, short-term outcomes with evidence, and intermediate outcomes with evidence. NIDRR requested that participating grantees provide feedback on this pilot process. Individual reports, developed by grantees as they completed their online data entry, served as the basis for the individual level review. The response rate was 100% for the 9 system level projects; 75% (7) of the 9 individual level projects responded. Panelists received electronic copies of these reports seven business days prior to the APAER meeting. To help prepare for the review, panelists received a background report describing the APAER process, providing information about NIDRR, the employment domain, and the aggregate data from the individual grantees. The aggregate data included information on funding mechanisms (19% of the portfolio were RRTCs, 31% were DRRPs, and 50% were FIPs); first time investigators (four FIPs and one DRRP); years of operation (the average was 2.2 years, and none of the projects that reported were in their 4th or 5th year); the number of awards with at least one previous cycle (three RRTCs, three DRPPs, and three FIPS); study examples; trainees by discipline; number of articles published (a total of 22); data instruments; and the number of nuggets reported as publications (14), short-term outcome nuggets (19), and intermediate outcome nuggets (9). This information was designed to give panelists the broad context at NIDRR and a sense of the accomplishments of the Employment portfolio in general. Additionally, NIDRR provided panelists with a report to help assess NIDRR management. This report provided NIDRR published funding priorities for a sample year, 2003; a compilation of 81 comments from consumers related to employment needs; an overview of the NIDRR peer review process; selection criteria for each type of funding mechanism; data on peer review scoring for FY 2003 competitions; information on the timeliness of the process; and panel composition. Reviewers accessed programmatic and logistical information as it became available through a Web site designed for this purpose. (http://www.neweditions.net/APAER2005/) 1.3.3 Panel Composition NIDRR developed guidelines for panel composition to ensure appropriate representation and expertise. The main criteria were: * A mix of "senior-level" scientists, clinicians/practitioners, educators/administrators, policy experts/Federal partners, industry representatives and consumer advocates, * A balance of disciplines, types of institution, geography and individual diversity, * A majority of non-NIDRR grantees, * A minimum of two researchers per cluster, and * No conflict-of-interest defined as direct financial or in-kind relationships. Nine reviewers - three researchers, one university administrator/researcher, two economists/policy experts, two clinician researchers, and one representative of a consumer organization - participated on the Employment panel. The expertise of the researchers included medical and vocational rehabilitation, social security and employment statistics, health and disability policy, hearing impairments, self-determination, and community-based mental health programs. (See Appendix B.) 1.3.4 Assessment Process Panelists participated in two types of assessments: individual grantee and portfolio level performance. Review of the individual grantee reports was intended to assist panelists in judging the overall quality, relevance, and performance results of NIDRR's entire Employment portfolio under the APAER process, rather than the performance of individual grantees. Individual grantee assessment. The panel received instructions for scoring the individual reports using the Scoring Form for Individual Awards and Accomplishment Nuggets, with an example of a completed form. Using this form, three panelists independently reviewed each report. They were instructed to identify the best published nuggets in the report, based on scoring, and select one nugget at each of three levels: (1) short-term outcomes, (2) intermediate outcomes, and (3) scientific accomplishments. Each panelist scored between five and six grant reports prior to the two-day meeting. At the on-site cluster meeting, NIDRR instructed panelists to: 1. Discuss individually identified nuggets to generate an inventory of agreed upon accomplishment nuggets. 2. Give the nugget a brief title to describe it. 3. Establish whether the accomplishment nugget was an output or outcome. 4. Rate the nugget and provide a rationale or justification for the rating. NIDRR asked panelists to rate nuggets on a scale of 1-3, with "1" being a minor contribution and "3" being an outstanding contribution. They used a rating of "0" for "unable to determine." Panelists were instructed to: * Consider outputs as the direct results of an activity. * Identify an accomplishment as a short-term outcome only when an evidence-based claim clarified how it contributed to the advancement of knowledge. Examples included citations, requests for reprints, or how publications were having an effect on advancing knowledge. * Identify accomplishments as intermediate outcomes if grantees demonstrated that they were using knowledge to create change. Portfolio level review. NIDRR developed a set of questions to guide the APAER portfolio level review and elicit the panel's general assessment of: (1) the performance of the portfolio based on NIDRR's GPRA measures and PART requirements; (2) NIDRR management activities; and (3) the APAER process itself. These broad questions were discussed at both the cluster level and the portfolio level during the meeting. 1.3.5 Meeting Agenda The panel met for two full days, with the panelists divided into cluster review groups on the afternoon of Day 1. The agenda is included in Appendix C. Dr. Susan Daniels, an independent consultant and former Deputy Commissioner for Disability and Income Security Programs at the Social Security Administration, facilitated the panel meeting. Contract staff provided technical assistance to panelists and logistical support. NIDRR staff participated as presenters, cluster liaisons, and observers. As cluster liaisons, NIDRR staff were available in each room to facilitate the process, primarily by reminding panelists of their charge and the recommended steps to follow. No NIDRR staff were present during the closing session of the meeting. The meeting opened with introductions and presentations by NIDRR staff. Steven Tingus, Director of NIDRR, provided opening remarks. Dr. Richard Melia, Director of the Research Sciences Division, provided an overview of NIDRR's mission, projected long-range outcomes, three primary goals and research objectives, and highlights from NIDRR's new Long-Range Plan. Ms. Shelly Reeves and Dr. Edna Johnson, NIDRR project officers, presented background information on NIDRR's Employment portfolio and research directions. Some of the recent employment accomplishments include: 1) research with the Social Security Administration on ways to use research findings to increase benefits and the numbers of unemployed individuals with disabilities returning to work; 2) strategies for job coaching; and 3) best practices for supported employment. Ms. Reeves and Dr. Johnson indicated that input from this APAER panel on gaps in projects was essential for shaping the future of approaches to employment for persons with disabilities. Dr. Margaret Campbell, Coordinator of Evaluation, reviewed the concept of the 2005 pilot, its relationship to PART, and how APAER was designed. She explained the procedures for the meeting, presented definitions for accomplishments, outputs, and outcomes, and clarified NIDRR's expectations of the panel. Prior to breaking into clusters, a brief discussion was held to clarify the process. Section 2: Portfolio Performance 2.1 Cluster Level Results and Critique After meeting separately, each cluster reported its findings to the entire panel, including the number of outputs, examples of the best accomplishment nuggets, and impressions of the quality of these nuggets. The aggregated data from these reports follow. The two clusters used two different approaches in their cluster level review to address questions. Panelists in the individual level employment issues cluster agreed to settle issues about terminology and adequacy of data as they arose during the review process. When that group was uncertain about the evidence provided by a grantee, they discussed their interpretations and decided if they could rate the particular accomplishment. In contrast, the cluster focused on systems level issues was guided by the meeting facilitator and NIDRR Coordinator of Evaluation to help the group clarify its rating of accomplishments as outputs or outcomes. Consequently, that cluster ranked more of the accomplishment nuggets as either outputs or outcomes. Panelists were originally instructed to only consider published accomplishments as outcomes nuggets. However, during the meeting, panelists also considered Web site products, the impact of grantees' technical assistance, and other reported accomplishments, such as influence on employment and rehabilitation policies and procedures. 2.1.1 Results The APAER Employment Panel examined reports from 88% of the awardees in the Employment portfolio. Fifty percent of the total employment awards reviewed were FIPs, 31% were DRRPs, and 19% were RRTCs. Selected common elements across the individual level and systems level clusters are shown in Tables 1 and 2. Cluster composition is presented in Table 1, and types of accomplishment nuggets are listed in Table 2. In general, panelists reported that they did not have enough information to give scores other than "0" to the identified nuggets. Table 1 Composition of Clusters Cluster 1 Cluster 2 Individual Level Systems Level By Number of Grantees # of Grant Reports Reviewed 7 9 Total # of Grantees 9 9 Percentage Grantees Reporting and Reviewed 78% 100% By Program Mechanism # of FIPs 4 4 # of RRTCs 0 3 # of DRRPs 3 2 Table 2 Type of Accomplishment Nuggets Cluster 1 Cluster 2 Individual Level Systems Level Academic Publications 5 8 Non-academic/ consumer-oriented publications or products 5 10 Discoveries, advancing knowledge and understanding 8 11 Tools 2 6 Interventions 8 6 Total 28 41 Note: Numbers reported reflect a compilation of potential nuggets identified individually by reviewers prior to attending the panel meeting and may not reflect cluster or panel discussion. 2.1.2 Inventory of Accomplishment Nuggets A compilation of accomplishment nuggets identified individually by panelists from grantee reports is presented in Appendix D. Some accomplishments were not classified due to insufficient information. Notes from individual and group review were compiled into a comment section. At the cluster level meetings, the panelists decided which of these nuggets merited joint review. In some cases, the grantee reporting was premature in that a product or accomplishment had not yet been completed, but rather was a work in progress. These were included in the inventory to provide examples of expectations for future accomplishments. 2.1.3 Cluster Level Reports to Panel During the APAER panel assessment, highlights of findings and impressions from each cluster meeting were reported. Key areas of agreement from each cluster are summarized below. Systems Level Cluster. The panel found it could not make many solid evaluations about scientific rigor and productivity based on the available data. Given NIDRR's expectations, the panel felt that the level of funding for these awards was low. The panel felt the cluster was narrow and conservative in scope and needed more emphasis on school-to-work transition issues. Panelists suggested NIDRR encourage interdisciplinary participation from non-traditional professions such as business, information technology and medicine to enhance the research. Individual Level Cluster. This cluster included four FIPs and two DRRPs. The panel determined it could not assess the scientific rigor and productivity of the awards in this cluster due to insufficient information. They noted a disconnect between the goal of the award activities and the reported accomplishments for many awards. Several panelists had the sense that awardees were doing more than was reported. They felt that the funding level for FIPs was low and suggested NIDRR clarify the purpose of FIPs. This panel also suggested that transition research is a gap in the portfolio. They commented that there is little attention given to individuals who have severe disabilities because it is more difficult and costly to conduct research with this population. The panel felt the awards needed more consumer involvement - including employers. APAER Process. After the presentations, panelists presented some overarching concerns about the assessment process. * Panelists would have expected a larger number of awards in this portfolio, given the importance and intractable nature of the problem. * One could assume that it would be difficult for a FIP in its first year to demonstrate very much. Because of this, there was almost a bias in the APAER model toward underrepresenting accomplishments. * If grantees had known when they received their award that they would be rated by APAER with its criteria, the results might have been better. * It seemed that NIDRR's definitions of outcomes were very restrictive, making it hard on grantees and NIDRR, compared to definitions used by other agencies. * A number of reviewers gave the benefit of the doubt to the awardees, assuming that the results reported were probably a conservative estimate of the project's impact, compared to a mature reporting system. It was an active process to identify accomplishment nuggets. 2.2 Portfolio Level Results After the cluster reports, the moderator asked panelists to consider all of the information presented as they evaluated the portfolio as a whole in relationship to specific NIDRR long-term performance measures and the PART criteria for R&D investment for quality, relevance and performance. Using a nominal process to stimulate discussion, the facilitators guided a thoughtful review of the Employment portfolio. To start the discussion, panelists jointly identified the best accomplishments from the portfolio: Outcomes: 1. A service delivery model for individuals who are deaf or hard of hearing that is now under Congressional review. 2. A policy brief on perspectives on disabilities (by race and ethnicity) for the Ticket to Work program. Outputs: 3. Publication of a book on disability law and policies. 4. Publication of a report and compendium on barriers to employment according to race and ethnicity for individuals who are blind, with documentation of a high number of sales. 5. Tutorials that were developed and used to train individuals who are blind to use Excel and Microsoft products. 6. Survey finding that interagency initiatives were the only employment policies that benefited individuals who are deaf or hard of hearing. 7. A literature review of self-employment among people with disabilities. 8. An investigation of business, employment, and economic development of persons with disabilities in rural areas, including development of a program for vocational rehabilitation counselors, with research conducted by multiple sites. 9. An accessible e-learning authoring system. Activities: 10. A qualitative study of persons with mental illness who got jobs on their own. 11. Broad dissemination of research on service delivery to persons with mental illness for performance improvements. 12. Development of a sample for a cohort study regarding individuals who are deaf and aging. When asked to make an overall assessment of the Employment portfolio in four areas, the panelists determined that: * The level of scientific excellence was generally inadequate, lacking scientific rigor and hypothesis testing. * The scope and breadth of the projects were individually narrow, but collectively the portfolio covered a broad area. * The overall productivity of the portfolio was disappointing and limited. * Consumers would likely find the portfolio to be disappointing, without as much useful information as might be anticipated. NIDRR developed a series of questions to elicit feedback on the progress of the portfolio on selected GPRA and internal long-term performance measures that relate to the NIDRR strategic goals and objectives in capacity building, research and development and knowledge translation. Panelists used response cards for "agree," "disagree," and "uncertain" to indicate their judgment for each question. These responses were not meant to serve as quantitative data, although in some instances the responses were counted as an indication of majority opinion. Key agreed upon points or themes are presented below. Comments that reflect one panelist's opinion are indicated as such. The first NIDRR strategic goal is to increase capacity to conduct and use high-quality and relevant disability and rehabilitation research. The following four questions address specific objectives for this area. Agreed. A few panelists who agreed, suggested that there might be a difference in interpreting the question. Since the projects reviewed were very narrow, they considered whether staffing across the current set of projects was adequate to meet the needs of these underfunded projects. They suggested that, within the current market, NIDRR is getting the teams it expects, and that the teams seem to be adequate to solve problems in creative ways. Also, they noted that it would have been helpful to have information on outside consultants. Disagreed. The majority disagreed with the adequacy of the multidisciplinary teams, indicating that there was not a true representation across disciplines and that sometimes a key discipline was not represented on a team. There was not enough cross-fertilization. With the FIPs, it seemed to be an issue of resources and the difficulty in employing additional staff on a very limited budget. The panel recommended considering incentives for cross-disciplinary participation in competitions. The panelists noted that there also was a considerable lack of consumer involvement in all aspects of design, implementation, and evaluation. Uncertain. None. Agreed. None Disagreed. The panel was split between those who disagreed and those who were unable to determine the adequacy of integrating research, education, and training. Those who disagreed indicated that they were not clear how the knowledge from the projects would integrate with education and training at both the individual and systems levels. Uncertain. Those who were uncertain if the portfolio had an adequate number of awards integrating research, education and training voiced concern that some of the projects are completing accomplishments that did not show up in the grantee reports. They commented that the emphasis on the "nuggets of accomplishments," as defined, would hinder acknowledgement of these accomplishments. They suggested that the data were sometimes incomplete, and that the time horizon for the snapshot was too narrow to expect research, development, and training. There was also a question regarding the relationship between OMB requirements and NIDRR's emphasis on training. Agreed. None Disagreed. The panel unanimously disagreed with this question. They did not believe these were adequately represented, and noted that they found no evidence of multi-site controlled trials for either individual or system level grantees. They thought, however, that this was a reflection of the funding amount. They noted that even the DRRPs provided no evidence of a controlled study, let alone a multi-site study. Several panelists also expressed a belief that some of these projects are conducting but not reporting such studies. They also were distressed that sometimes projects did not acknowledge NIDRR as a funding source, which they believed should happen even if NIDRR provides only partial funding for specific research. Uncertain. None The panel had difficulty answering due to lack of data, and recommended skipping these questions. The second NIDRR strategic goal is to generate scientifically based knowledge, technologies, and applications to inform policy, change practice and improve outcomes. One question addressed objectives under this goal. Agreed. None. Disagreed. Those who disagreed indicated that the methodology for the research was inadequate, and although there was some movement, the portfolio as a whole did not appear to be advancing knowledge. It seemed that nothing was particularly innovative. One panelist suggested that funding should be invested elsewhere, and that although there were some good ideas, she would want to see more progress next time. Uncertain. Those who couldn't decide indicated that they did not have access to information on the methodology used by the grantees and very few peer reviewed publications were reported, which might provide some reassurance regarding high quality methodologies. The panel also suggested that "they needed the fullness of time" to determine the quality and significance of the research, and that since the term "significant" was ambiguous, it was difficult to evaluate this. They thought that perhaps "significantly advanced" might be too high a bar, given the timeframe for the grantees. The third NIDRR strategic goal is to promote the effective use of scientific-based knowledge, technologies, and applications to inform disability and rehabilitation policy, improve practice, and enhance the lives of individuals with disabilities. The following three questions correspond to specific objectives in this area. Agreed. Those who agreed stated that many of the projects demonstrated their value, even if it wasn't directly related to the research or results. They felt that the grantees' activities demonstrated definite potential to positively impact the individuals they were serving. Some were giving the grantees the benefit of the doubt, even though they were not certain that the grantees were producing high quality results. They also noted that these projects historically have developed and disseminated non-academic products. Disagreed. Those who disagreed acknowledged, however, that some projects demonstrated potential. They also commented that the non-academic publications appeared neither to have been expected as part of a priority, nor planned, but rather appeared almost through serendipity. Uncertain. Those who were uncertain remarked that, given the way the instructions were interpreted, this is likely an under-reported category. While the number of non-academic publications was high overall, they were uncertain of the quality. These panelists were uncertain regarding NIDRR's expectations, and recommended that a better process might involve first developing a stated purpose for the projects and then evaluating progress against the stated purpose. Also, given that all of the projects had something of this nature, they questioned what was adequate. The panel did not know of any registries focusing on the disability employment arena. They suggested checking with Statistics Canada and ARCH (a Legal Resource Center for People with Disabilities, also in Canada), the GAO standards on reliability and validity, the NIDRR's Interagency Subcommittee on Disability Statistics, the Research Triangle Institute (RTI), and the Office of Disability Employment Policy (ODEP). After some discussion, a question was raised about whether there should be any standards and guidelines for disability employment research, and if so, who should develop these standards? Reviewers noted the importance of the definition of disability and its numerous sub-populations, and suggested caution in relying on secondary data sources. They noted that the work-limited population is an important sub-population, and the focus of some attention. They discussed the controversy that has been generated when ADA data has been used in policy analysis, and suggested that NIDRR: * Examine theoretical concepts and different studies and data sets (the ICI, Nagi, medical and social models) and identify the overlaps. * Improve cross-agency dialogue to facilitate consideration of standards. * Address differences in studying secondary and primary data and the need to attach research to funding mechanisms. * Develop priorities for both exploratory research and research designed to investigate generalization across conditions, populations, etc. * Address both issues and procedures in measuring policy impact in the employment area. * Consider limitations of measures and how to improve measurement based on current disability conceptualizations. * Consider the implications of the current emphasis on evidence-based research and the importance of participatory action research. Section 3: NIDRR Management of Portfolio Assessments of NIDRR management were offered on the cluster and portfolio levels. At the cluster level, panelists made judgments about each cluster after reviewing, rating and discussing a set of individual grantee reports. The portfolio level assessment was based on reports from both clusters. Discussion focused on portfolio research directions and NIDRR management activities. 3.1 Portfolio Level Results Facilitators led the group discussion using questions developed by NIDRR. Not all questions were addressed due to time constraints. Group judgments are reported below for research directions and NIDRR management activities. Agreed. None Disagreed. Those who disagreed expressed a common concern that the FIPs are not adequately funded. Overall, the panel believed that the size of the grants was too small for what they were trying to accomplish, making it difficult to have appropriate multidisciplinary teams and facilitate replicability. The panelists thought that there was an engineering rather than a social science bias within NIDRR, suggesting that grantees may not necessarily be drawing on the best expertise. They also noted that lack of funding for research related to employer-based, labor/demand issues, as well as projects building on the value of self-determination in employment. Several panelists reflected on the lack of school-to-work transition projects, even though this was a prominent goal in the 1999 Long-Range Plan. The panel also had several recommendations regarding increasing funding for the FIPs and reexamining the purpose of these projects, including improving the alignment between the FIPs and the larger centers. Uncertain. None Agreed. None. Disagreed. The panel indicated that there was no specific focus on self-determination, which should be highlighted because of its importance both in terms of motivating people to work and in the broader scheme of helping plan for careers. They also commented that: * While the RRTCs are more aligned with the state of the science, the FIPs vary. * There should be some mechanism to focus on the employer side of employment concerns. The panel recommended that NIDRR be more directive, and use more of a contract-based model, with a more specific statement of the problems. However, NIDRR should move away from dictating methodologies, leaving that to the creativity of the researchers and being careful not to constrain the researchers. They also noted that the priorities did not identify the stage of knowledge development, which could be useful. In a similar vein, they emphasized that they would welcome and encourage more project officer involvement in the portfolio in guiding and reporting the accomplishment nuggets. The panel recommended that NIDRR consider more opportunities for dialogue and collaboration between traditional NIDRR researchers and other researchers. They noted that "quality of the ideas" in the grants was high. They also urged NIDRR to find ways to increase communication and sharing among its researchers in the employment arena. Agreed. Those who answered "yes" stated that within the limitations imposed by inadequate funding, NIDRR was accomplishing what it can. They believed that other agencies were "just as dysfunctional," and commended NIDRR for distributing materials to its audience. Disagreed. Those who disagreed compared NIDRR to NIH and foundations. They recognized that NIDRR typically provides funding at a higher level than foundations; however, foundations can sometimes do more and might really be getting "a bigger bang for the buck." They mentioned both NIMH and SAMHSA as agencies that seem to have larger budgets and address a larger breadth of research with more advanced standards for methodology. While they were uncertain of the impact of the larger budgets, they believed that budget differences made comparison difficult. Uncertain. Panelists who were unable to determine the adequacy suggested that most other agencies do not have portfolio systems, and that it may be both difficult and unfair to compare NIDRR to other agencies, especially given the differences in budgets. The panel recommended dropping this question because of the limited information provided and difficulties in making fair comparisons. 3.2 Closing Discussion During this final session, the panel discussed issues emerging from earlier discussions relating to a continuum of NIDRR interests, a process to identify critical gaps, and topics that should be discontinued. They also provided recommendations regarding NIDRR management of activities. NIDRR staff were not present. The panel began by commending NIDRR on its willingness to be innovative and bring in experts to assist with this review. The panel recommended that NIDRR consider an advisory group that would help determine priorities for funding and funding mechanisms, and suggested that: * NIDRR develop evidence-based practices, perhaps with a standing group of scientists, contracted on an ongoing basis to advise NIDRR. * NIDRR may obtain the best results if it is more directive in its priorities, but should not be restrictive in terms of imposing methodologies on researchers. The panel recommended that NIDRR consider using standing panels to provide feedback to assist project directors in revising and enhancing the quality of the proposals submitted to NIDRR. The panel also recommended that: * NIDRR consider expanding its research priorities for investigating and improving employment of individuals with severe and profound disabilities. NIDRR should recognize such projects may be more difficult and more expensive, and fund them accordingly. * NIDRR be involved in more cross-agency research and promote collaboration among agencies. The panel was concerned that the difficulties they experienced in evaluating the quality of some of the accomplishments reported by grantees may be reflective of the type of projects being funded, and recommended that NIDRR focus grantees on products that have proven validity. The panel also suggested that NIDRR: * Consider reducing the number of FIPs, unless there is a strong possibility that a FIP will contribute to employment outcomes. NIDRR should stop funding FIPs as they are now conceptualized with the small amount of money and no systematic structure to build on early findings from FIPs, integrating that research with the broader, more heavily-funded mechanisms. Instead, FIPs could serve as a mechanism for stimulating the research of new researchers, with guidance from the project officers. * Examine what seems to be a bias to continue funding Centers, and hold these Centers more accountable. The panel recommended that NIDDR continue its unique role in funding disability-related portfolios, continue to be an agent of change, and continue its important role in funding new researchers. They also suggested that NIDRR: * Obtain more input from the public. * Use project officers as guides to help the research community write the strongest proposals. * Move to a standing panel peer review process to stabilize the reviews and change the selection criteria to be less prescriptive. The panel also provided initial comments concerning the content and scientific undertakings related to the Employment portfolio, encouraging NIDRR to: * Explore possibilities for researching the impact of self-directing the employment process by individuals with disabilities. One panelist indicated that he has seen greater employability among individuals with disabilities who value employment, and suggested that research might focus on changing values so that employment is valued, perhaps in partnership with NIMH. * Reexamine "transition from school to work" with OSERS, with NIDRR taking the lead to focus on the adult perspective. Section 4: Feedback on APAER Process Panelists at both the cluster and portfolio levels offered assessments of the pilot APAER process. At the portfolio level, after both clusters reported, panelists were asked to assess the entire portfolio process. At the cluster level, panelists made judgments based on their experience reviewing, rating and discussing a set of individual awards. 4.1 Portfolio Level Results The panel observed that APAER provides NIDRR with a means to improve its Employment portfolio. They felt that portfolio review is the first step in a necessary but difficult process, and that despite the complexity, NIDRR's broad audience is a strength. Having completed the portfolio review, the panel perceived a disconnect between the goals of the projects and what was reported. They indicated that the information they received was inadequate to provide a valid evaluation of this portfolio, and that the data reviewed were underprocessed. They suggested that more of the work should be completed by the grantee, and provide a rationale for why the proposed outcomes were important. They reported what they believed to be structural issues with APAER, including the need for more active management of research programs, with project officers facilitating more interaction with grantees. The panel also discussed terminology, and recommended that future reviews include more terms in a glossary, and changing certain terms, such as "adequate/inadequate," which they felt were not accurate in conveying how projects fit together and fill the gaps in the existing research. The panelists suggested that the APAER process could be improved by: * Providing more information on grantee accomplishments prior to the meeting, * Sifting through the information and improving the files before sending them to the review panel, * Allowing reviewers more time for discussion, although they felt that the on-site process seemed to be fairly efficient, * Providing specific content guidelines for grantees in writing their reports used for portfolio-level review, * Sending a summary to all panelists for reaction after the review, then revising the process, and sending it to selected panelists for review, to then introduce a refined process next year. 4.2 Cluster Level Comments In their cluster level reports, panelists identified several concerns with how the data nuggets were identified and reported, indicating that: * It was difficult to determine what the grantees were reporting, and difficult to determine what really merited being termed a nugget of accomplishment. * In the future, grantees should be given a tutorial on completing the online form with feedback. * NIDRR might want to consider a process where instead of grantees nominating the nugget, the staff and reviewers work from grantee-generated reports to identify the accomplishments. * With small grants, pushing for results too soon could lead to "bad science." * NIDRR may want to rethink the APAER sampling; if a project is conducting a trial and pressured to come to conclusions prematurely, this could lead to ethical problems. * Allow for reporting of interim products. * NIDRR might consider basing accomplishments on NIDRR's long-term measures as each grantee identifies its goals and milestones. Section 5: Summary of Discussion 5.1. Implications for the Employment Portfolio According to the APAER panelists, improving employment outcomes for persons with disabilities remains challenging and yet is crucial to both improving their overall quality of life and reducing their dependence on governmental support. These long-term challenges emphasize the potential impact of any discoveries NIDRR and its grantees make in this arena. Therefore, the panel believed that whatever steps NIDRR can take to improve the Employment portfolio are noteworthy. One of the primary implications of the review of this portfolio is that enhanced communication from NIDRR, cross-agency collaboration, and sharing among researchers conducting similar investigations is highly recommended by the panel. Another is that an advisory group and standing panels could be instrumental in both increasing the quality of research initiatives and the quality of proposals that are funded. Limitations of the Findings. As panelists expressed many times throughout the APAER review, better data from the grantees could have improved the validity of the outcomes obtained in this review. Panelists felt restricted by confusion over terminology, grant reports that were incomplete, and inconsistencies in the data review process. Panelists wanted more information from grantees on how they conducted their research as well as the impact of their findings. Because of these difficulties, it may be that the grantees have completed more high-quality accomplishments than were identified through APAER. 5.2 Implications for the APAER Process Panelists made several recommendations for improving NIDRR's APAER process, including enhancing the role of the project officer, simplifying the terminology and the APAER process, and providing panelists better access to publications of grantees. Limitations with the Employment Pilot. The newness of the portfolio level assessment approach for Federal research agencies contributed to some difficulties in both conceptualizing the process and preparing materials for rating both cluster-level and portfolio-level accomplishments at NIDRR. A number of difficulties were identified in advance of the APAER meeting and others were identified on-site, suggesting that caution be used in interpreting the findings. These difficulties included: * A lack of independent field-testing of instruments prior to their use in the pilot, * Grantee lack of knowledge and experience with the outcome measure terminology, as well as insufficient technical assistance to grantees, resulting in inconsistent and limited data collection, * Providing a sample of best accomplishments to the panelists by the grantees, without prior review (and perhaps selection) by NIDRR staff, * Competition data about the peer review process was limited to 2003 data, as it was difficult to collect this information from NIDRR. However, the data were only reviewed very briefly at the APAER meeting due to time constraints. * Panelists had limited time to review materials and some materials were only available on-site. * Terminology was confusing to panelists. * Scoring of accomplishment nuggets was not consistent across the two clusters. 5.3 Grantee Comments Grantees were given the opportunity to provide feedback on the supplemental information form and the online data collection process. Regarding the online data collection process, grantees indicated that the form helped them figure out how to communicate accomplishments in a way that supports GPRA requirements. However, they had several recommendations for simplifying and improving the process, including providing alternative reporting methods (optional text fields) to allow for providing a fuller report and explanations, improving the online navigation, and changing some of the default formats. They recommended that project officers work more closely with them ahead of time so that they have a better understanding of NIDRR's reporting requirements. APPENDIX A: NIDRR PERFORMANCE MEASURES OSERS: National Institute on Disability and Rehabilitation Research: RA Title II FY2006 CFDA Number: 84.133 - National Institute on Disability and Rehabilitation Research Goal 7: Special Education and Rehab. Services Internal Goal. Objective 7.1 of 2: Advance knowledge through capacity building: Increase capacity to conduct and use high-quality and relevant disability and rehabilitation research and related activities designed to guide decision-making, change practice, and improve the lives of individuals with disabilities. Indicator 7.1.1 of 2: By 2013, increase by at least 25% the number of new NIDRR grants awarded to multidisciplinary teams of investigators that represent a relevant balance of sub-fields within medicine, behavioral and social sciences, education, engineering, information sciences, and design. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of new grants awarded to multidisciplinary teams of investigators that meet the stated criteria. (Input-Oriented Capacity Building) Year Actual Performance Performance Targets 2006 999 Additional Source Information: Peer review of applications. Frequency: Annually. Collection Period: 2005 - 2006 Data Available: November 2006 Indicator 7.1.2 of 2: By 2013, increase by at least 25% the number of new grants awarded to minority serving institutions and to first time NIDRR investigators from underrepresented populations. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of grants awarded to minority serving institutions and to first time NIDRR investigators from underrepresented populations. (Input-Oriented Capacity Building) Year Actual Performance Performance Targets 2006 999 Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Additional Source Information: IPEDS for minority serving institutions. Additionally, grant applications for first time investigators. Collection Period: 2005 - 2006 Data Available: November 2006 Objective 7.2 of 2: Advance knowledge through translation and dissemination: Promote the effective use of scientifically-based knowledge, technologies, and applications to inform disability and rehabilitation policy, improve practice, and enhance the lives of individuals with disabilities. Indicator 7.2.1 of 1: By 2013, increase by at least 20% the number of non-academic and consumer-oriented dissemination publications and products, nominated by grantees to be their best outputs based on NIDRR funded research and related activities, that demonstrate ''good to excellent'' utility for intended beneficiaries. Targets and Performance Data Assessment of Progress Sources and Data Quality The percentage of non-academic and consumer-oriented dissemination products and services, nominated by grantees to be their best outputs based on NIDRR funded research and related activities, that meet the stated criteria. (Outcome-Oriented Knowledge Translation) Year Actual Performance Performance Targets 2005 999 2006 999 Explanation: Approximately 1/3 of NIDRR's grants, based on a judgmentally selected sample of grantee nominated ''non-academic, consumer-oriented dissemination publications and products'' are reviewed annually. The first completed three-year cycle of portfolio assessments will include FY 2005, 2006, and 2007. Additional Source Information: Expert Review Panel. Frequency: Annually. Collection Period: 2005 Data Available: September 2005 Program Goal: To conduct high-quality research and related activities that lead to high-quality products. Objective 8.1 of 4: Advance knowledge through capacity building: Increase capacity to conduct and use high-quality and relevant disability and rehabilitation research and related activities designed to guide decision-making, change practice, and improve the lives of individuals with disabilities. Indicator 8.1.1 of 3: By 2013, at least 10% of all newly awarded projects will be multi-site, collaborative controlled studies of interventions and programs. Targets and Performance Data Assessment of Progress Sources and Data Quality The percentage of new projects conducting multisite, collaborative controlled trials. (Output-Oriented Capacity Building) Year Actual Performance Performance Targets 2005 999 2006 999 Explanation: This applies only to RERCs, RRTCs, Model Systems Grants, and DRRPs. Additional Source Information: Staff review of grant applications. Collection Period: 2004 - 2005 Data Available: November 2005 Indicator 8.1.2 of 3: By 2013, at least 100 individuals from diverse disciplines and backgrounds will be actively engaged in conducting high-quality disability and rehabilitation research and demonstration projects. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of former pre- and postdoctoral students and fellows who received research training supported by NIDRR who are actively engaged in conducting high-quality research and demonstration projects. (Outcome-Oriented Capacity Building) Year Actual Performance Performance Targets 2007 999 Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Frequency: Annually. Collection Period: 2006 - 2007 Data Available: November 2007 Indicator 8.1.3 of 3: Percentage of NIDRR-supported fellows, post-doctoral trainees, and doctoral students who publish results of NIDRR-sponsored research in refereed journals. Targets and Performance Data Assessment of Progress Sources and Data Quality The percentage of NIDRR-supported fellows, postdoctoral trainees, and doctoral students who meet the stated criteria. (Output-Oriented Capacity Building) Year Actual Performance Performance Targets Fellows Post-Doc Trainees Doctoral Students Fellows Post-Doc Trainees Doctoral Students 2005 999 999 999 2006 999 999 999 Explanation: The review is conducted using an accepted standard, such as the International Scientific Index (ISI). Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Frequency: Annually. Collection Period: 2004 - 2005 Data Available: November 2005 NIDRR is planning to work with other ED staff to conduct an audit of publications entered into the Web-based reporting system to verify grantees' self-reports of peer-reviewed journal articles. Objective 8.2 of 4: Advance knowledge through research and related activities: Generate scientific-based knowledge, technologies, and applications to inform policy, change practice, and improve outcomes. Indicator 8.2.1 of 6: By 2013, increase by at least 20% the number of discoveries, analyses, and standards developed and/or tested with NIDRR funding that have been judged by expert panels to advance understanding of key concepts, issues, and emerging trends and strengthen the evidence-base for disability and rehabilitation policy, practice, and research. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of discoveries, analyses, and standards developed and/or tested with NIDRR funding that meet the stated criteria. (Outcome-Oriented Research & Development) Year Actual Performance Performance Targets 2007 999 Explanation: Approximately 1/3 of NIDRR's grants, based on a judgmentally selected sample of grantee nominated ''discoveries'' will be reviewed. The first completed three-year cycle of portfolio assessments will include FY 2005, 2006, and 2007. Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Additional Source Information: Expert Review Panel. Frequency: Annually. Collection Period: 2005 - 2007 Data Available: November 2007 Indicator 8.2.2 of 6: By 2013, increase by at least 20% the number of new or improved tools and methods developed and/or tested with NIDRR funding that have been judged by expert panels to improve measurement and data collection procedures and enhance the design and evaluation of disability and rehabilitation interventions, products, and devices. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of new or improved tools and methods developed and/or tested with NIDRR funding that meet the stated criteria. (Outcome-Oriented Research & Development ) Year Actual Performance Performance Targets 2007 999 Explanation: Approximately 1/3 of NIDRR's grants, based on a judgmentally selected sample of grantee nominated ''tools and methods'' will be reviewed. The first completed three-year cycle of portfolio assessments will include FY 2005, 2006, and 2007. Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Additional Source Information: Expert Review Panel. Frequency: Annually. Collection Period: 2005 - 2007 Data Available: November 2007 Indicator 8.2.3 of 6: By 2013, increase by at least 20% the number of new and improved interventions, programs, and devices developed and/or tested with NIDRR funding that have been judged by expert panels to be successful in improving individual outcomes and increasing access. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of new and improved interventions, programs, and devices developed and/or tested with NIDRR funding that meet the stated criteria. (Outcome-Oriented Research & Development) Year Actual Performance Performance Targets 2007 999 Explanation: Approximately 1/3 of NIDRR's grants, based on a judgmentally selected sample of grantee nominated ''interventions'' will be reviewed. The first completed three-year cycle of portfolio assessments will include FY 2005, 2006, and 2007. Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Additional Source Information: Expert Review Panel. Collection Period: 2005 - 2007 Data Available: November 2007 Indicator 8.2.4 of 6: Percentage of grantee research and development that has appropriate study design, meets rigorous standards of scientific and/or engineering methods, and builds on and contributes to knowledge in the field. Targets and Performance Data Assessment of Progress Sources and Data Quality The percentage of grantee research and development that has appropriate study design, meets rigorous standards of scientific and/or engineering methods, and builds on and contributes to knowledge in the field. (Activity-Oriented Research & Development) Year Actual Performance Performance Targets 2002 82 65 2003 96 70 2004 89 70 2005 999 2006 85 Explanation: The methodology for assessment of quality of funded projects changed in 2004. NIDRR no longer uses a second expert peer review of grantee research designs. The current measure is the ''Percentage of funded grant applications that received an average peer review score of 85 or higher.'' The data have been recalculated using the new methodology. Source: Other Other: Peer Review. Date Sponsored: 05/27/2005. Frequency: Annually. Collection Period: 2004 - 2005 Data Available: November 2005 Indicator 8.2.5 of 6: Average number of publications per award based on NIDRR-funded research and development activities in refereed journals. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of publications per award meeting the stated criteria. (Output-Oriented Research & Development) Year Actual Performance Performance Targets 2002 2.74 2003 2.84 8 2004 5 2005 5 2006 2 Explanation: An accepted standard, such as the International Scientific Index (ISI) will be used. Data for publications will be collected over a calendar year, instead of fiscal year. Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Frequency: Annually. Collection Period: 2004 Data Available: September 2005 Improvements: Publication data from four additional program funding mechanisms (DBTACs, DRRPs, FIPs, and KDU (Dissemination & Utilization) projects) will be included. Indicator 8.2.6 of 6: Percentage of new grants that include studies funded by NIDRR that assess the effectiveness of interventions, programs, and devices using rigorous and appropriate methods. (Output-Oriented Research & Development) Targets and Performance Data Assessment of Progress Sources and Data Quality The percentage of new grants that meet the stated criteria. Year Actual Performance Performance Targets 2002 65 2003 59 2004 59 2006 999 Additional Source Information: Review of grant applications. Frequency: Annually. Collection Period: 2005 - 2006 Data Available: November 2006 Objective 8.3 of 4: Advance knowledge through translation and dissemination: Promote the effective use of scientific-based knowledge, technologies, and applications to inform policy, improve practice, and enhance the lives of individuals with disabilities. Indicator 8.3.1 of 2: By 2013, increase by at least 20% the number of tools, methods, interventions, programs, and devices, developed and/or validated with NIDRR funding, that meet the standards for review by independent scientific collaborations and registries. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of NIDRR-funded tools, methods, interventions, programs, and devices that meet the stated criteria. (Output-Oriented Knowledge Translation) Year Actual Performance Performance Targets 2007 999 Explanation: Approximately 1/3 of NIDRR's grants, based on a judgmentally selected sample of published discoveries, tools, methods, interventions, programs, and devices. The first completed three-year cycle of portfolio assessments will include FY 2005, 2006, and 2007. Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Additional Source Information: Expert Panel Review Frequency: Annually. Collection Period: 2005 - 2007 Data Available: November 2007 Indicator 8.3.2 of 2: Number of new or improved assistive and universally-designed technologies, products, and devices developed and/or validated by grantees that are transferred to industry for potential commercialization. Targets and Performance Data Assessment of Progress Sources and Data Quality The number of new or improved assistive and universally designed technologies, products, and devices. (Outcome-Oriented Knowledge Translation) Year Actual Performance Performance Targets 2005 999 2006 999 Source: Performance Report Grantee Performance Report: 1820-0642 Annual Performance Reporting Forms for NIDRR Grantees (RERCs, RRTCs, DBTACs, DRRPs, Model Systems, Dissemination & Utilization Projects). Frequency: Annually. Collection Period: 2004 - 2005 Data Available: April 2006 Objective 8.4 of 4: Enhance efficiency of NIDRR grant award process. Indicator 8.4.1 of 1: Notification: Notification of applicants. Targets and Performance Data Assessment of Progress Sources and Data Quality The percentage of competitions announced by Oct. 1. Year Actual Performance Performance Targets 2003 21 2004 23 2005 8 2006 999 The percentage of grant awards issued within 6 months of the competition closing date. Year Actual Performance Performance Targets 2003 70 2004 83 2006 999 Additional Source Information: GAPS and Federal Register Notice. Frequency: Annually. Collection Period: 2005 - 2006 Data Available: October 2006 Source: 2006PM 08/01/2005 01:26 PM APPENDIX B: PANEL MEMBERS Facilitator Susan Daniels, Ph.D. Daniels & Associates 3001 Veazey Terrace, NW #633 Washington, DC 20008-5413 Phone 202-363-8970 Fax 202-363-0145 E-mail SMDaniels@earthlink.net Panelists Tom Hale, Ph.D. Economist Social Security Administration 6401 Security Boulevard. 3535 Annex Baltimore, MD 21235-6401 Phone 410-965-6636 Fax 410-597-0825 E-mail tom.hale@ssa.gov Tanya M. Gallagher, Ph.D. Dean University of Illinois College of Applied Life Studies 1206 South 4th Street Champaign, IL 61820 Phone 217-333-2131 Fax 217-333-0404 E-mail tmgallag@uiuc.edu Debra Harley, Ph.D. Professor University of Kentucky 229 Taylor Education Building Lexington, KY 40506-0001 Phone 859-257-7199 Fax 859-257-1325 E-mail dharloo@uky.edu Glenn Hopkins Executive Director The Main Place Inc. 33 West Main Street Newark, OH 43055 Phone 740-345-6874 x 22 Fax 740-345-5157 E-mail ghopkins@themainplace.org Debra Lerner, M.D., Ph.D. Director of Program on Health, Work and Productivity Associate Professor, Department of Medicine Tufts University School of Medicine 750 Washington Street T-NEMC # 345 Boston, MA 02111 Phone 617-636-8640 Fax 617-636-8351 dlerner@tufts-nemc.org Donald Moores, Ph.D. Professor Gallaudet University 800 Florida Avenue NE Washington, DC 20002 Phone 202-651-5530 Fax 202-651-5860 E-mail donal.moores@gallaudet.edu Jo-Anne Sowers, Ph.D. Associate Research Professor Regional Research Institute Portland State University P.O. Box 751 Portland, OR 97207 Phone 503-725-9610 Fax 503-725-4040 E-mail soweri@pdx.edu John S. Trach, Ph.D. Associate Professor, Special Education Director, RSA Region V CRP RCEP University of Illinois at Urbana Champaign, IL 61820 Phone 217-333-2325 Fax 217-244-0851 E-mail jtrach@uiuc.edu David Wittenburg, Ph.D. Senior Researcher Mathematica Policy Research 600 Maryland Avenue South West Suite 550 Washington, DC 20024 Phone 202-484-4527 Fax 202-863-1763 E-mail dwittenburg@mathematica-mpr.com NIDRR and Other Department of Education Staff Steven J. Tingus, M.S., C.Phil. Director, NIDRR U.S. Department of Education Room 6056 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7549 Fax 202-245-7643 or 202-245-7323 E-mail Steven.tingus@ed.gov Kelly E. King, M.D. Deputy Director, NIDRR U.S. Department of Education Room 6059 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7639 Fax 202-245-7643 or 202-245-7323 E-mail Kelly.king@ed.gov Judith Anderson Budget Service U.S. Department of Education 400 Maryland Avenue, SW Washington, DC 20202 Phone 202-401-3944 Fax 202-401-0220 E-mail Judith.anderson@ed.gov Ellen J. Blasiotti NIDRR U.S. Department of Education Room 6038 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7275 Fax 202-245-7643 or 202-245-7323 E-mail Ellen.blasiotti@ed.gov Margaret Campbell, Ph.D. NIDRR U.S. Department of Education Room 6022 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7290 Fax 202-245-7643 or 202-245-7323 E-mail Margaret.campbell@.ed.gov Grace Cooney NIDRR U.S. Department of Education Room 6034 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-6721 Fax 202-245-7643 or 202-245-7323 E-mail Grace.cooney@ed.gov Edna Johnson, Ph.D. NIDRR Room 6040 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7594 Fax 202-245-7643 or 202-245-7323 E-mail Edna.johnson@ed.gov Richard K. Johnson, Ed.D. NIDRR U.S. Department of Education Room 6033 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7631 Fax 202-245-7643 or 202-245-7323 E-mail Richard.johnson@ed.gov Richard P. Melia, Ph.D. NIDRR U.S. Department of Education Room 6053 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7446 Fax 202-245-7643 or 202-245-7323 E-mail Richard.melia@ed.gov Shelley Reeves, M.S. NIDRR U.S. Department of Education Room 6031 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7486 Fax 202-245-7643 or 202-245-7323 E-mail Shelley.reeves@ed.gov William Schutz, Ph.D. NIDRR U.S. Department of Education Room 6063 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7519 Fax 202-245-7643 or 202-245-7323 E-mail William.schutz@ed.gov Delores L. Watkins NIDRR U.S. Department of Education Room 6074 400 Maryland Avenue, SW Washington, DC 20202-2700 Phone 202-245-7568 Fax 202-245-7643 or 202-245-7323 E-mail Delores.watkins@ed.gov New Editions Staff Betsy Tewey, M.S. Vice President New Editions 6858 Old Dominion Drive Suite 230 McLean, VA 22101 Phone 703-356-8035 Fax 703-356-8314 Email btewey@neweditions.net Christine Mason, Ph.D. Senior Research Scientist New Editions 6858 Old Dominion Drive Suite 230 McLean, VA 22101 Phone 703-356-8035 Fax 703-356-8314 Email cmason@neweditions.net Eldri Ferguson Conference Manager New Editions 6858 Old Dominion Drive Suite 230 McLean, VA 22101 Phone 703-356-8035 Fax 703-356-8314 Email eferguson@neweditions.net APPENDIX C: AGENDA Agenda Annual Portfolio Assessment Expert Review Employment Portfolio Pilot National Institute on Disability and Rehabilitation Research September 29 and 30, 2005 Holiday Inn Capitol 550 C Street, SW Washington, DC 20024 Thursday, September 29, 2005 8:15 a.m. Continental Breakfast 9:00 a.m. NIDRR WELCOME Ste Steven James Tingus, M.S., C.Phil., Director of NIDRR 9:15 a.m. ORIENTATION Overview of NIDRR Mission, Program Funding Mechanisms, & Logic Model Richard Melia, Ph.D., Director Research and Sciences Division Overview of Employment Portfolio and Research Directions: LRPs 1999-2003 vs. 2005-2010 Edna Johnson, Ph.D. Shelley Reeves, M.S. Research Sciences Division The What & Why of APAER: Relationship to PART & NIDRR Logic Model Margaret Campbell, Ph.D. Coordinator of Evaluation 10:15 a.m. APAER ROLES & PROCEDURES Susan Daniels, Ph.D., Facilitator and New Editions, Project Staff 10:45 a.m. Break 11:00 a.m. WORKING GROUP - CLUSTERS * Start process of reviewing awards & accomplishment nuggets and complete individual level ratings Cluster Facilitators & NIDRR Staff Liaisons 12:30 p.m.* Lunch (on your own) 1:30 p.m. WORKING GROUP - Clusters, Cont. * Finish reviewing awards & accomplishment nuggets and complete individual level ratings * Submit individual score sheets to contractor for aggregation by 3:30 p.m. Cluster Facilitators & NIDRR Staff Liaisons 3:30 p.m. WORKING GROUP - Clusters, Cont. * Summative Discussion of Cluster Performance (i.e., relevance, quality, productivity, & significance of topics and accomplishments) Cluster Facilitators & NIDRR Staff Liaisons 5:00 p.m. ADJOURN Annual Portfolio Assessment Expert Review Employment Portfolio Pilot Friday, September 30, 2005 8:00 a.m. Continental Breakfast 8:30 a.m. WELCOME Kelly E. King, M.D. Deputy Director 8:45 a.m. OVERVIEW OF DAY 2 Susan Daniels, Ph.D. Facilitator 9:00 a.m. PORTFOLIO CLUSTER PRESENTATIONS Cluster Facilitators 9:45 a.m. PANEL DISCUSSION CORE QUESTIONS - Report Preparation Susan Daniels, Ph.D. Facilitator 12:15 p.m. Working Lunch & Q&A with NIDRR Senior Management Steven James Tingus, M.S., C.Phil., Director of NIDRR, and Arthur M. Sherwood, Ph.D. Science and Technology Advisor 1:30 p.m. REVIEW OF PORTFOLIO ASSESSMENT from Morning Session Susan Daniels, Ph.D. Facilitator 2:00 p.m.* Identification of Key Weakness in Portfolio and Recommendations for Improvement (NIDRR management & Grantee performance) Susan Daniels, Ph.D. Facilitator 3:30 p.m. WRAP-UP AND EVALUATION Susan Daniels, Ph.D. Facilitator, and New Editions, Project Staff 4:00 p.m. ADJOURN **NIDRR Staff will not be in attendance at the afternoon sessions APPENDIX D: ACCOMPLISHMENT NUGGETS BY EMPLOYMENT CLUSTERS This is a compilation of accomplishment nuggets identified individually by panelists from grantee reports. (Note: in compiling these data, the type of accomplishment was classified either individually, during panel discussion, or by staff based on criteria established by NIDRR. Some accomplishments were not classified due to insufficient information, and some were reclassified by staff after noting discrepancies in applying definitions. Therefore, the notations in this table may not correspond to the totals in Table 1.) The Comment section is a compilation of notes from individual and group review. It is important to note that the review of the individual grantee reports was intended to assist panelists in judging the overall quality, relevance, and performance results of NIDRR's entire Employment portfolio under the APAER process, rather than the performance of individual grantees. Program Mechanism & Grant Title Title of Accomplishment Nugget Type of Accomplishment Nugget Comments Outputs Outcomes Systems Level Cluster 1. Grantee A: RRTC 1.1 Published Web site findings; feasibility of using VR counselors for job placement and support 10 year retrospective study 1.2 Adoption of policy and practices by Dept of Vet Affairs; collaboration with Vermont 1 Policy/practice Invited to conference. Evidence of potential work with VR in rural areas in VT. Model for economic development 1.3 Multidisciplinary Team Mixed review; some questioned adequacy 2. Grantee B: RRTC 2.1 Survey of administrators and program specialists of 43 state VR agencies Academic Publication Impact of federal legislation & policy on VR services for consumers who are deaf and hard of hearing Published by request -American Annals of the Deaf 2.2 Developed service delivery model Policy Presentations and Web site distributed information; TX VR adopted materials. Legislation in Congress to review and improve service delivery model plans for implementing new national certification system. 2.3 Multidisciplinary Team Counseling, Experimental Psych, Rehab, Sociology 2.4 Promoting improved practices - nonacademic services - including proceedings Academic Publication Practice Impact of VR services on deaf and hard of hearing; needs of youth with disabilities in post-secondary education; importance of workplace accommodations 2.5 Resource curriculum & audiovisual electronic media resource Tool Requested for statewide use by Rehabilitation Services in TX. Facilitated obtaining funds for national organizations. 3. Grantee C: RRTC 3.1 First to profile access to VR for this population by race, ethnicity. Academic Publication Advancing knowledge with potential to impact practice Technical research report - promotes heightened understanding of barriers; promising survey methodology; compendium can contribute to the change process 3.2 Barriers to employment of the blind/training for VR Invited presentations; 100 complimentary copies of publications to VR administrators; 125 copies purchased by stakeholders 3.3 Multidisciplinary Team Rehabilitation Counseling, Business, Counselor Ed, Psychology (rated inadequate) need more/other social scientists, e.g., sociologists and anthropologists 3.4 Implemented information to better serve people who are blind/VI Non-academic/ consumer publication Information improved service delivery Recipient of William H. Graves Awards 4. Grantee D: DRRP 4.1 Advancing knowledge about welfare and rehabilitation for women with disabilities in poverty Publication Promoting improved practice. 4.2 Multidisciplinary Team Included Social Welfare, Sociology, Economics, Nursing, and Human Development - very balanced. 4.3 Evidence of Technical Assistance Service Email exchange helped Georgia; provided assistance to Missouri to write NIH/NIMH grant to improve employment services; evidence of international requests for information. 5. Grantee E: DRRP 5.1 Extensive ADA analysis in legal framework Findings Advancing knowledge Presentations, articles, IT Awards. Need to be sure we continue to renew the information on Disability Civil Rights law. Limited analysis of IT outside book. Book used in other courses at law school; strong technical approach. 5 articles, 1 book, 2 trainee publications. 5.2 Book Non-academic publication Policy Widely distributed 5.3 Multidisciplinary team MDs, JDs, Psychology; - very nice 6. Grantee F: FIP 6.1 Developing modules In process- unable to determine. Interviews by major media/newspapers Professional journals. Not clear that this contributed to specific goals. 6.2 Multidisciplinary Team Psychology, Social Work, Rehabilitation Counseling, Sociology, Biostatistics. Perhaps include an ethnographer and psychiatrist. 6.3 Publication 2002 Academic Publication Role of vocational success and recovery to SSA is overstated. 7. Grantee G: FIP 7.1 Advancing Knowledge about Ticket to Work Self-published; publication not in designated performance period; need more information to support claims. Only local presentations; user friendly format. 7.2 Multidisciplinary Team Mixed opinion. Adequate representation and balance; unable to determine - the team has one bona fide social scientist. 7.3 Policy Brief Non-academic/ consumer publication Helpful, if disseminated. 8. Grantee H: FIP 8.1 Accessibility to technology to improve employment journal article Non-academic/ consumer publication (Buzzard, 2004) guild publication; nothing new in this publication - recommends building on existing interfaces 8.2 Survey developed (not published) Enabling discovery, learning and innovation. 8.3 Multidisciplinary Team Mixed opinion. Two disciplines: Psychology and Business Education - needs more representation; some thought the balance was adequate. 8.4 Disseminating information Missing data, non-academic service 9. Grantee I: FIP 9.1 Increase understanding of the disablement process and influence of local environment Findings Missing data; data set of dubious value. 9.2 Multidisciplinary Team Economics, Rehabilitation, Psychology, Sociology 9.3 First time information on housing to be included in 2005 Price Out report Not yet published; non-academic product; data on public housing. Individual Level Cluster 1. Grantee J: DRRP 1.1 Offer online courses and tutorials on Microsoft Office Suite at beginner intermediate, and advanced levels Nonacademic/ consumer product Distributed 1,567 tutorials, guides, and course packets; offered 24 online courses (2002-2004) & published training tutorial packets listed' another 64 items were distributed to almost 300 individuals. Productivity is excellent. No evaluation of courses and products by consumers; lacked evidence regarding competitors. Results not published in refereed journal. Ready for dissemination to service providers. 1.2 Multidisciplinary Team Rated inadequate. No information on PI; other three investigators minimally qualified. Staff with disability and teaching backgrounds. 2. Grantee K: FIP 2.1 Technical research report - Identification of Job Retention Factors1 Academic Publication Should also be extended to MH systems - need more information. Wrote a number of papers- none published on issues of homeless; reported significant impact on local agencies and their staff serving this population. 2.2 Publication-Challenging Expectations: How Individuals with Psychiatric Disabilities Find and Keep Work Academic Publication May also need to include legal advocates training for staff - need more information. 2.3 Multidisciplinary Team Mixed opinion about adequacy. 2.4 Staff training Service Has developed some staff training based on results of the study. These are not published. Training seminar for staff training of local employment assistance agencies is reported to have had impact on their knowledge, skills, and policies. 3. Grantee L: FIP 3.1 Theory of stigma Unable to determine- missing data. No publications, no description of methods. May serve as basis for future measurement tool. RTC training curriculum informed by this study that will be delivered by peers and teach how to deal with stigma. 3.2 Emerging findings about first qualitative stage of the study/developed new grant proposal/plans to develop an internet device/anticipate a internet photovoice exhibit on overcoming stigma. Findings & new grant proposal Unable to rate; missing data. Could be a powerful tool- very early in the product's development. 3.4 Multidisciplinary Team Inadequate 4. Grantee M: FIP 4.1 Results of in-depth interviews with 32 individuals with psychiatric disabilities Findings Not reported as a nuggets - unclear if this was the result of the current work or another paper. 5. Grantee N: DRRP 5.1 Recruited 400 participants for national study; resulting in development of a critical data source This short-term accomplishment can have a major impact on knowledge to the degree that an active and interested research has been created. 6. Grantee O: FIP 6.1 Article on telecommuting in Journal of Vocational Rehabilitation Academic Publication List of companies who want to use telecommuting to accommodate workers with disabilities. Led to one job placement and a project funded by DOE/ODEP for developing telework for veterans with disabilities and worker compensation clients. Accomplishments don't seem to match objectives of the abstract - modeling predictors of success in telework. Level of commitment of employers showing interest is not described. A 2001 publication in Journal of VR was cited, but grant did not begin until 2002. Missing data. 6.2 Identified several hundred businesses interested in teleworking Describes a new approach that can be used an option for assisting individuals with disabilities to become employed. APPENDIX E: PARTICIPANT EVALUATION SUMMARY Annual Portfolio Assessment Expert Review (APAER) 2005 Employment Portfolio Pilot September 29-30, 2005 Evaluation Results Expert Reviewer Response Design 1 2 3 4 5 6 7 8 9 Ave Instructed to consider the potential of the APAER process to...using a scale of 1-3, 3 being very well designed Assess the quality and relevance of NIDRR-funded research 1 1 3 1 1 2 2 1 2 1.56 Identify the extent to which outputs and outcomes are contributing to the agency's long-term performance measures and strategic goals 1 1 3 2 2 2 1 1 2 1.67 Provide NIDRR with an assessment of the agency's management of research directions and award decisions 1 1 3 2 2 3 1 1 2 1.78 Total 1.67 General Expert Reviewer Response 1 2 3 4 5 6 7 8 9 Ave Rated on a scale of 1-5, 5 being strongly agree The concept of NIDRR's portfolio assessment is sound 2 3 5 5 5 4 5 2 4 3.89 The reviewers were well qualified and presented a good balance of expertise 4 5 5 4 4 4 5 5 5 4.56 The background materials were useful 4 4 5 4 4 3 4 2 4 3.78 The on-site registration was convenient for you 4 4 5 5 4 5 4 2 5 4.22 The web site was useful 3 2 5 2 n/a 5 n/a 2 3 3.14 Day 1, overview by NIDRR staff was helpful 4 5 5 5 4 5 5 4 4 4.56 General Expert Reviewer Response 1 2 3 4 5 6 7 8 9 Ave Rated on a scale of 1-5, 5 being strongly agree Day 1, discussion in Cluster Groups was effective 5 4 5 4 4 5 5 5 5 4.67 Day 2, whole group APAER Discussion & Scoring was effective 5 4 5 4 4 4 5 4 4 4.33 The discussion of recommendations to NIDRR was important and useful 5 4 5 4 4 3 5 4 5 4.33 The meeting facilitators were effective 4 5 5 5 4 4 5 4 5 4.56 The support provided by staff on site was helpful 5 5 5 5 4 5 5 4 5 4.78 The hotel and meeting space, including accessibility, met your expectations 5 5 5 5 4 5 5 5 5 4.89 The food and beverage services were excellent. 5 5 5 1 3 4 5 4 4 4.00 Total 4.28 Comments What are the strengths of the APAER process: Moving toward quality research The use of multidisciplinary reviews Openness & fairness of review process; range of expert reviews (including consumer expertise) in process; basic structure is good but needs improvement; hard working & dedicated staff; portfolio level review is a good idea Staff Provides a big picture view of current research, attempt to assess impact of funding opportunities in various domains Focus on change The openness, willingness to explain the portfolio by all involved This provides a general overview of the NIDRR effort in employment, the review process is very healthy What are the weaknesses of the APAER process: Poor data - clearly did not get most of data Not enough information in time It's a work in process, more consumer participation Needs more connection between project goals and outcomes Needs to take NIDRR logic model through more compression planning The materials provided & when we received them needs to be improved Be sure all the necessary info is available and in sufficient quantity to evaluate Other Comments: Try to align evaluation process with grant application process so that in the grant writing stage, the application is required to "des.." expected outputs & outcome within a situated time frame; need consideration of key concepts-ST nugget, LT nugget, nominated publications, evidence, multidisciplinary. Improve scan system with more levels to reflect important variations between projects The two day process was excellent. Amount of work= amount of time Given that this was a pilot, a lot was learned for the process that was very important Thanks, it was a good experience 1 PART is a systematic method of assessing and improving program performance across the Federal government, instituted by the Office of Management and Budget (OMB). The APAER was also designed to satisfy specific recommendations from NIDRR's 2003 PART review. ?? ?? ?? ?? NIDRR Annual Portfolio Assessment Expert Review Pilot: Employment Portfolio Panel Summary Report 53 NIDRR Annual Portfolio Assessment Expert Review Pilot: Employment Portfolio Panel Summary Report 56