This article provides a comprehensive guide on conducting Item analysis using YouTestMe GetCertified. Item analysis allows you to assess the reliability and effectiveness of test questions.
There are three ways to conduct item analysis:
- Predefined Test Report – You can utilize the item reliability report provided by YouTestMe GetCertified. This report offers insights into the performance of each test question, allowing you to assess the reliability and effectiveness of the items.
- Report Builder – The platform also offers a report builder feature that enables you to download all the answers to the questions. With this downloaded data, you can perform item analysis independently, gaining more flexibility and control over the analysis process.
- Success ratio – Within YouTestMe GetCertified, you have access to the Question tab, where you can easily view the success ratio for each test question. This information helps you evaluate the difficulty level and effectiveness of individual items
Using Predefined Reports
To access the Item Reliability report, follow the next steps:
- Select the Reporting/Predefined reports tab.
- Select the Test reports.
- Choose the Item reliability report.
- Select the Details button to preview each question’s success ratio and item reliability.
On the displayed screen, there are the following metrics shown:
- The success ratio provides the percentage of candidates that answered the question correctly.
- Item reliability is based on point-biserial correlation and will let you know how items correlate with the test as a whole, allowing you to identify the questions that do not belong in a test. When determining the correlation between the item and the test as a whole, the system correlates the responses on that item (dichotomous variable – correctly or incorrectly answered) with the test’s outcome (continuous variable – the score on a test). The values can range between -1 and 1. Negative values usually mean that:
- Candidates that got lower scores on the exam answered this question correctly.
- Candidates that got higher scores on the exam answered this question incorrectly.
In this case, the question does not correlate with the exam, and you may want to exclude it from the next exam versions or substitute it. Positive values usually mean that:
-
- Candidates that got higher scores on the exam answered this question correctly.
- Candidates that got lower scores on the exam answered this question incorrectly.
In this case, the question correlates with the exam and should be included in the next exam versions.
- Standard deviation measures the variability of success ratios for individual questions, helping identify inconsistent performance and correlations between question difficulty and overall test scores
Note: The item reliability report shows manually created tests.
Using Question-Level Statistic Report
To access the Questions report, select the Tests/Manage tests, click on the name of the desired test, and navigate to the Questions tab.
In this tab, you can:
- Access the Test Structure Information, which lets you review the number of questions used in the test for each difficulty level, along with the associated points, penalties, and durations.
- Decide whether to randomize the questions and answers for each candidate.
To compare the performance of candidates for each question used in the test, follow these steps:
- Select the desired test version.
- Click on Compare Questions By and choose the parameter you want to use for comparison (Success Ratio, Time to Answer, or Question Occurrence).
The Questions table will present the selected data alongside the question details, including difficulty, duration, type, and pool name. Moreover, you can preview the statistics for each answer to analyze the quality of the answers.
To examine the statistics for answers related to a specific question, click the Details button.
The Answers Report provides the following information:
- The percentage and number of candidates who selected each answer.
- Global statistics for the question, including the success ratio and the standard deviation.
