User guide for Principal Examiners and

Team Leaders

Version 10.0
For the September 2016 RM Assessor release and later
User instructions for RM Assessor, the worlds
most widely used high stakes onscreen marking software
International Baccalaureate
Peterson House, Malthouse Avenue, Cardiff Gate
Cardiff, Wales GB CF23 8GL
United Kingdom
Phone: +44 29 2054 7777
Fax: +44 29 2054 7778
Website: http://www.ibo.org

International Baccalaureate Organization 2016

The International Baccalaureate (IB) offers three high quality and challenging
educational programmes for a worldwide community of schools, aiming to create a
better, more peaceful world.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval
system, or transmitted, in any form or by any means, without the prior written
permission of the IB, or as expressly permitted by law or by the IBs own rules and
policy. See http://www.ibo.org/copyright.

The RM Assessor trademark is owned by RM. International Baccalaureate hereby

acknowledge all trademarks and copyrights of RM products.

2
Table of Contents
PE and TL navigation aid ............................................................ 5

Scripts available to mark for standardization, email

notification ................................................................................... 5

E-standardisation set up............................................................. 6

Producing provisional responses .............................................. 8

Finding and viewing responses............................................................... 8
Creating provisional responses ............................................................ 10
Marking provisional responses ............................................................. 11
Discarding selected provisional responses ......................................... 12
Adding RIG-level comments .................................................................. 14
Submitting provisional responses ........................................................ 17
Monitoring provisional marking ............................................................ 18
Reviewing submitted provisionally marked responses ...................... 19
Setting the definitive marks ................................................................... 21

Classifying responses .............................................................. 22

Reclassifying responses ........................................................................ 24
Declassifying responses ........................................................................ 24

Completing standardisation setup ........................................... 25

What happens next? ............................................................................... 25

Standardisation reports ............................................................ 26

Standardisation Set-up ..................................................................................26
Standardisation Progress ..............................................................................27
Standardisation Feedback .............................................................................27
Promote to seeding ................................................................................ 27

The approval process for examiners ....................................... 30

Auto-approval ......................................................................................... 30
Feedback Required status ..................................................................... 31
Viewing your team .................................................................................. 31

3
 Approval management ................................. Error! Bookmark not defined.

Supporting your team ............................................................... 35

Messaging within RM Assessor ............................................................ 35
Finding a response ................................................................................. 36
Sharing a response................................................................................. 37
Exceptions ............................................................................................... 38
Resolving exceptions.....................................................................................38
Escalating exceptions ....................................................................................40
Viewing actioned exceptions .........................................................................41
Marking reports available....................................................................... 41
General Seeding Inaccuracy .........................................................................41
Marker Seeding Tolerance ............................................................................41
Marking - Daily Seeding Feedback ................................................................42
Examiner Feedback ................................................................................ 42
Requesting a re-mark ............................................................................. 42

Getting Support ......................................................................... 44

4
PE and TL navigation aid
If you are involved in supervision or standardisation will notice the following icons on
the title bar once you open a worklist, team management or standardisation screen.

In the example shown the worklist is being viewed, indicated by the box around the
icon. The icons allow you to switch between worklist, team management and
standardisation screens without the need to return to the Overview screen as in
previous versions of the software.

Scripts available to mark for standardization,

email notification
If you are a Principal Examiner for a component, you can choose to receive an
automatic email to let you know when scripts are available to review and provisionally
mark for the purpose of standardization .Once you receive the email alert you can pass
the information on to the other members of the standardization team.

Note: You only get a notification if you enter your email address, as described below,
before the first script has arrived.

To receive an email notification:

1. Click the Controls icon at the top right of the page and select Change User
Options.

The User Options dialog box is displayed:

5
 2. Scroll down the options, and then type your email address in the Email
Notifications field.
3. Click OK, your email address is then saved and you will receive notification once
scripts are available in RM Assessor.

E-standardisation set up
As part of the standardisation set up process, you will be required to set up practice
scripts, 1st qualification scripts, 2nd qualification scripts and seeding scripts on RM
Assessor.
All members of the standardisation team are able to view the E-standardisation tab on
RM Assessor and the scripts which have been scanned and uploaded to RM Assessor.
Each member of the team can select scripts and provisionally mark in RM Assessor, if
a team member moves the script to their worklist to provisionally mark, the rest of the
team are only allowed to view a copy of the script, they are unable to mark the same
response. Once provisional marking occurs in e-standardisation the mark scheme
structure (i.e. mark values) can NOT be altered.
The Principal Examiner will have access to all the provisionally marked scripts, and
would select the scripts that are agreed during standardisation as practice, qualification
and seeding scripts. The Principal Examiner will set the marks as definitive. Once
standardisation set up is complete on RM Assessor, the marks of submitted, 100%
marked, provisional responses or definitively marked responses that have not been
classified are credited to the provisional examiners. These responses then appear in
the examiner's closed responses Worklist as normal marking. Un-submitted provisional
responses are returned to the marking pool to be marked during live marking.
To access Standardisation Setup tasks in RM Assessor, log in as usual and on the
Home page click the E-standardisation tab.

The Action Required indicator will appear on this tab if you have actions
outstanding.

This page shows a list of open QIGs (Question Item Groups) for which you are an STM
(Standardisation Team Member). The information shown summarises how the process
is proceeding and helps you to establish where you need to take action.
If you are a Principal Examiner (with overall responsibility for the standardisation setup
process), you will see your own progress on provisional marking as well as the
progress of the standardisation team.
If you are a team leader you will only see your own progress on provisional marking.
By default, the screen shows all relevant QIGs, but you can use the filter options to
filter the list:
Click Responses available to show only QIGs with responses that are available for
use in standardisation setup.
Click In Progress to show only QIGs where standardisation setup is in progress.
Click Open Provisionals to show only QIGs that have open provisional responses.

6
 As a Principal Examiner only, click Further Provisionals Required to show only
QIGs where the total number of practice and qualification provisional responses
has not yet reached the required target.
Click New Messages to show only QIGs where you have new unread messages.
Click the Messages icon ( ) to view your messages.
You can also use the drop-down list at the top right of the screen to group the displayed
QIGs.
The Provisionals section in the table shows progress in marking provisional responses.
The indicator in the Responses Available column tells you whether there are scanned
scripts available for provisional marking. The number of bars in the indicator tells you
from how many schools you have responses available to download and provisionally
mark:
1 bar Responses available from 1 school

2 bars Responses available from 2 to 9 schools

3 bars Responses available from 10+ schools

Greyed out with no No responses available at present.

bars

For example, the following indicator with three bars indicates that you have responses
available from 10+ schools:

The My Provisionals column shows how many provisional responses you have
marked and submitted.
The Team Provisionals column is only visible to Principal Examiners; it shows how
many provisional responses the standardisation team has marked and submitted.
The Classification section in the table shows how many practice, first and second
qualification, and seeding responses have been classified in relation to the relevant
targets.
The State column indicates overall progress of standardisation setup for each QIG. It
will be one of:
Awaiting scripts
In progress
Complete
To open a QIG and carry out standardisation setup tasks, click Open.

7
Producing provisional responses
The Available Responses worklist is available to Principal Examiners and
standardisation team members in the E-standardisation tab in RM Assessor.

Finding and viewing responses

To find and view responses:
1. Log into RM Assessor.
2. On the Home page click the E-standardisation tab at the top of the page then
open a QIG by clicking the relevant Open button. The E-standardisation page
opens with the Available Responses worklist screen.

The screen displays a list of all the schools for the relevant QIG. The QIG name is
displayed at the top of the screen, and a summary panel indicating progress
against standardisation setup targets is shown at the bottom left of the screen.
Initially, the first examination centre in the list is selected and available responses
for this centre are displayed.
Select a different centre to see the available responses for this centre.
For each school, the Scanned Responses figure shows the total number of
responses currently available for use in the standardisation setup process and the
Available for provisional marking figure shows the number available for
provisional marking. The difference between these figures is the number of
responses already selected for provisional marking by you or other standardization
team members.
3. Click on a school to expand it and see the available scanned responses for the
school.

Note: Each scanned paper is given a unique Response ID number. When you
move or mark a response, it is assigned to you and given a Display ID. In your
worklist you will see columns for both ID numbers.

4. Responses that have already been selected for use in standardisation either by you
or other standardisation team members, are flagged as shown on the example
screen above. You can't select an already selected response for provisional
marking, but you can view it and you will receive a message indicating whether you
or another team member has selected it for provisional marking.

8
 Use one of these methods to view a response:
Double-click the response.
Select the response, then click the View button on the toolbar.
The marking page opens.

With a response open in view mode, you can:

Close the response and return to the list of centres and responses.

Use the Previous Response and Next Response buttons to view other responses
from the currently selected school.

It is possible for your team members to provide you with the Display Ids of
simulation responses whilst standardisation setup is in progress. To find a
simulation response by Display ID provided by an examiner:
Enter the supplied display ID in the search box at the top of the screen,
Information similar to that shown below is displayed, allowing you to find the
simulation response.

9
Creating provisional responses
You can create provisional responses and mark them immediately or create them for
later marking.
When you create provisional responses, they are added to your My Provisional
Marking worklist within Provisional Marking on the E-standardisation page.

To create provisional responses:

1. You can select the provisional responses you want to create from the Available
Responses list on the E-standardisation Setup page.

Responses that have already been selected for use as provisionals will show as
Provisional in the Provisional column.
To select further responses for use as provisionals, highlight one or more
responses that have not already been selected for use as provisional responses,
then click Provisional Response.

To select multiple provisional responses, use Ctrl+click (to select multiple non-
adjacent responses) or Shift+click (to select a range of responses).

Alternatively, if you want to view a response before selecting it as a provisional, first

find the response in the Available Responses list then either double-click on it or
select it then click View. This will open the response in the Marking page.

10
 If you are happy that you want to use the response as a provisional, click
Provisional Response.
2. Whichever method you used to select the provisional responses that you want to
create, the following dialog box is displayed.

3. Tick the Mark response now box if you want to start marking the selected
response(s) immediately.
4. Click Continue.
5. If you chose to mark the selected responses immediately, they will open in the
marking page and you can mark and submit them.
6. If you were viewing a response and chose not to mark it immediately, close the
response to return to the E-standardisation page or use the Next and Previous
buttons to view other responses in the currently selected school.
On the Available Responses list on the E-standardisation Setup page,
Provisional is now displayed against the responses you have made provisional.
These responses cannot now be selected by anyone else.
7. Repeat the process until you have as many provisional responses as you need.
You can now select, mark and submit provisional responses from your My Provisional
Marking worklist.
You can discard provisional responses from your worklist should you no longer wish to
use them in the standardisation set up.

Marking provisional responses

You can if you wish create provisional responses and mark them immediately.
Alternatively, you can create your provisional responses and then mark them later. In
either case, the provisional responses are placed in your My Provisional Marking
worklist within Provisional Marking on the E-standardisation page.

To mark a provisional response:

1. Go to the E-standardisation page for the relevant QIG and in the left hand
navigation bar select the Provisional Marking section and then My Provisional
Marking.

11
 2. A list of provisional responses that you have created displays. To open a response
for marking, either double-click it or select the response then click on the Mark
button.
3. Start marking the response. If, during marking, you decide the response is not
suitable, you can discard it by clicking on the Discard Response button.
4. If you want to add a comment to your response before you submit it click on the
Add Comments button. This is useful for communicating your thoughts to your
Principal Examiner and the other standardisation team members.
5. When you are satisfied that you have finished marking a response and it is 100%
marked, you can close it and submit it for review by the Principal Examiner and
other standardisation team members.

Discarding selected provisional responses

At any time before you submit a provisional response, you can choose to discard it if
you decide that it is not suitable. Any marks you have added to the response will be
lost.
Discarded responses become available for selection by other standardisation team
members.

To discard a response from your My Provisional Marking worklist:

1. Go to the E-standardisation page for the relevant QIG and, in the left hand
navigation bar, click Provisional Marking and then My Provisional Marking.

2. Select the response(s) you want to discard and click Discard Response.

Note: You can use Ctrl and click to select multiple non-adjacent responses, or
Shift and click to select a range of adjacent responses.

12
3. Click Yes. The selected provisional responses are discarded and removed from
your My provisional responses worklist.

13
 To discard a response in the marking page:
1. With the provisional response open for marking, click Discard Response.

2. A confirmation message displays.

3. Click Yes. You are returned to your My Provisional Responses worklist, and you
can see that the discarded provisional response is no longer listed.

Adding RIG-level comments

During marking you may wish to communicate something about a particular response
item group (RIG) to others in the standardisation team.
There are two ways to add comments:
From the relevant worklist
When viewing or marking a response
Comments added in either of these ways are merged into the same list of comments.

To add a RIG-level comment from a worklist:

1. In the My Provisional Responses or Response Classification worklist as
appropriate, select the response to which you want to add a comment and click
Add Comment. . Alternatively, click F8 as a shortcut to open the Add Comment
box.

Note: You cannot add comments to multiple responses at a time. RIG-level

comments can only apply to individual responses.

14
 The RIG Level Comments box displays, showing the display ID of the response and
any previous comments already added to this response.

2. If you don't want your name and the date to appear against the comment, you can
type your comment directly into the top box.
If you do want your name and the date to appear, type your comment in the bottom
box then click Add Comment.

3. Your new comment displays in the top box. If you used Add Comment then your
name and the date will appear alongside the comment.
4. Add further comments if you wish, then close the comments window by clicking on
the X in the top corner of the comments box.
5. When you return to the relevant worklist, notice that a comment icon has been
added to your response list, and that if you move your mouse pointer over it you
can read the comment, as shown below.

15
To add a RIG-level comment when marking a response:
1. Open the response to which you want to add a comment and click the Add
Comments button at the top of the screen.

A comments box is displayed.

2. Enter your comment then optionally click the Add Comments icon again to turn it
off and close the comments box.
3. As with adding comments from a worklist, when you return to the relevant worklist,
a comment icon will have been added to your response list, and you can move your
mouse pointer over it to read the comment.

16
Submitting provisional responses
At any time you can submit provisional responses that you have fully marked for review
by the Principal Examiner and other standardization team members.

To submit provisional responses from the My Provisional Marking page:

1. Go to the E-standardisation page for the relevant QIG and in the left hand
navigation bar select Provisional Marking and then My Provisional Marking.

2. You can see from the Progress column in the list of responses which ones you
have 100% marked and hence are ready to submit. You can either submit all
responses that are 100% marked or submit selected responses.
To submit one or more marked responses, select the response(s) you want to
submit, then click the Submit Response drop-down menu and select Submit
Selected.

Note: You can use Ctrl and click to select multiple non-adjacent responses, or
Shift and click to select a range of adjacent responses.

3. Click Yes to confirm the submission.

Alternatively, to submit all responses that are 100% marked, click the Submit
Response drop-down menu and select Submit Completed.

4. Click Yes to confirm the submission.

To submit the current response from the Marking page:

As soon as an open response is 100% marked, you can click the Submit button in
the toolbar to submit it.

17
 The marks you have awarded will be saved before the submission occurs.

Note: If you want to submit responses from the Marking page as you finish marking
them, you need to turn off Auto Advance. If Auto Advance is on, then as soon as
you finish marking each response you will be moved on to the next response before
you have had a chance to submit the current response.

Submitted responses, including those submitted by the Principal Examiner, are moved
from your My Provisional Marking worklist to the Response Classification worklist
on the Principal Examiner's E-standardisation page.

Monitoring provisional marking

As provisional marking proceeds, Principal Examiners can monitor overall progress as
well as the progress of individual standardisation team members.

To monitor STM team marking:

1. Go to the E-standardisation page for the relevant QIG and in the left hand
navigation bar select the Provisional Marking section and then Team Provisional
Marking.

2. A summary of overall progress against the target number of provisionals required

displays, along with a standardisation team hierarchy showing a summary of open
and submitted provisional responses against each person. To view a person's
provisional marking in more detail, either double-click on their name or select them
and click View.

A list of the provisional responses that the team member has created displays, and
you can see the progress they have made.
To see provisional responses that the team member has submitted, click the Closed
Responses radio button.
To open a response to look at it more closely, either double-click it or select it then click
View.
Use the Send Message button to send the team member a message if necessary to do
so.
When you have finished viewing the team member's progress, click Back to Team
Marking worklist.

18
Reviewing submitted provisionally marked responses
All the submitted provisionally marked responses can be found in the Response
Classification worklist on the E-standardisation page. The Principal Examiner always
has access to this worklist.
In the Classification worklist it is possible to:
Sort and order by marks allocated
See which examiner originally marked the response
See if any comments are associated with a response
See the original Response ID (the ID displayed in Available Responses) and the
Display ID (the unique ID generated by RM Assessor)

To review submitted provisional responses:

1. From the Home page, click E-Standardisation tab.
2. In the left hand navigation bar on the E-standardisation Setup page, click
Response Classification. The screen displays a list of provisional responses to
classify along with a summary of progress against targets.

The Classification Summary at the bottom left of this screen indicates the number
of classified responses and the target number of responses to be classified in each
group.

The targets for Practice and Qualification are mandatory and standardisation setup
cannot be completed until they are met. The seeding target is purely a guide. It is
advisable for as many seeding responses as possible to be created during the
standardisation setup process and then additional can be created after this has
been completed. The seeding target is typically to highlight the number needed to
provide enough coverage during the marking session without seeding responses
having to be used more than once.
3. To open and review a response, use one of these methods:
Highlight a response, and then click View.
Double-click a response.

19
 The selected response is opened in view mode.
. You can turn off the visibility of the current annotations or previous annotations
depending which ones you would like to view by clicking the button below on the top
toolbar and selecting to hide.

You may want to do this when an STM has added many annotations and it is
difficult to see the candidates response underneath.

20
Setting the definitive marks
There are two ways you can set the definitive marks in RM Assessor:
Copy the provisional marks as definitive in a single action, and edit if required
Start marking and create a completely new set of definitive marks
You can always edit the definitive marks, whichever method you choose.

To copy the provisional marks as definitive:

1. From the Home page, click the E-Standardisation tab.
2. In the left hand navigation bar on the E-standardisation Setup page, click
Response Classification. The screen displays a list of provisional responses to
classify along with a summary of progress against targets.
3. Select and open the provisional response of interest, either by double-clicking on it
or by selecting it then clicking View.
4. On the marking page, click the Copy Marks As Definitive button on the toolbar.

A new column appears in the mark input panel, so this now shows both the
definitive marks you are working on and the previously entered marks.

Note: Copy marks as definitive copies marks, annotations and comments. Once
this action has been taken it is possible to amend the marks annotations and
comments. However, after edits have been made and if the copy marks as
definitive button is clicked again assessor will re-apply the provisional marks,
annotations and comments.

5. Click Start Marking and if required amend any of the definitive marks, annotations
or comments.
6. When you are satisfied with the definitive marks, the annotations and comments,
close the response or click the Next Response button on the toolbar to definitively
categorise or mark the next provisional response.

To mark a response and use your own marks as definitive:

1. Open a provisional response of interest.
2. Click Start Marking. As above, a new column appears in the mark input panel, so
this now shows both the definitive marks you are working and the previously
entered marks. However, this time there are no marks yet recorded in the Marks
column (unless you provisionally marked the response yourself - see Note below).

Note: If as the Principal Examiner you provisionally marked a response yourself,

then the marks column will always remain the same even after editing

3. Mark the response.

21
 4. When you are satisfied with the definitive marks shown, save and close the
response or click the Next Response button on the toolbar to definitively
categorise or mark the next provisional response.

Note: If required you can come back and amend definitive marks at any point up
until you choose to complete standardisation setup.

When you have finished setting definitive marks, click the E-standardisation tab to
return to your worklist of provisional responses to classify.
Notice that for this response, Definitive Marks has now replaced the examiner's name
in the Marker column, and the Classify button is now active when the response is
selected.

You can if you wish tag responses that have definitive marks or change an existing tag
by clicking More, Tags, and then clicking a tag.

Note: Tagging a response makes it easy to identify how you want to classify it later
on. It is a good idea to decide on colours to represent the response types.

You can click on a column heading in the table to sort on that column. Click again to
toggle the sort order between ascending and descending. For example, you might want
to sort on all responses that have definitive marks, or sort by total marks.
You can classify responses at this point if you wish.

Classifying responses
Definitively marked responses need to be classified as either practice, qualification or
seeding within RM Assessor.

Note: Please be aware of the classification targets and do not exceed the target
number for practice or seeding responses. RM Assessor will not allow you to classify
more than the target number of qualification responses but there is not this systematic
restriction with practice or seeding responses. However, an exclamation mark will be
displayed next to the Practice target as an indication if the user has classified
responses which exceeds the target.

22
Note: Make sure you have reviewed the definitive marking against the final mark
scheme before classifying a response.

To classify a response:
1. Log into RM Assessor (Principal Examiner only).
2. On the Home page, click E-standardisation tab.
3. In the left hand navigation bar on the E-standardisation tab, click Response
Classification. The screen displays a list of provisional responses to classify along
with a summary of progress against targets.

Note: If you tagged your responses, as suggested earlier, you can quickly identify
and highlight responses of a particular type.

4. You can either:

Select the responses you want to classify, which must all have definitive marks
set. From the Classify Response drop-down menu on the toolbar select the
required classification: Practice, 1st Qualification, 2nd Qualification or
Seeding.
Open a response either by double-clicking on it or by selecting it then clicking
View. On the Marking page, from the Classify Response drop-down menu on
the toolbar, select the required classification: Practice, 1st Qualification, 2nd
Qualification or Seeding.
5. The classified responses will now be removed from the Unclassified list of
responses, but if you click Classified you will see them there, and you can use the
Classify button to modify the classification of selected responses if you wish.

Note: All classified responses except seeds are visible to STMs. Seeds are only
visible to the Principal Examiner.

6. Repeat the process to classify more responses.

To view the classified responses, at any time, click Classified in the Response
Classification worklist.
Notice that a classification icon in now displayed next to each response; blue for
practice ( ), red for qualification ( ), and yellow for seeding ( ).
To see a table of the classified responses showing the marks for each question, click
Classified Matrix radio button.

23
 Note: The types of response you can view in the Classified Matrix list depends on
your position in the standardisation team hierarchy.

This view helps STMs ensure that a suitable range of definitive marks is provided
amongst all of the selected classified responses.

Reclassifying responses
It is possible to re-classify responses which have been classified up until the
standardisation set up is complete.
To re- classify a response:
Select the responses you want to re-classify, from the Classify Response drop-
down menu on the toolbar select the required classification
Or
Open a response either by double-clicking on it or by selecting it then clicking
View. On the Marking page, from the Classify Response drop-down menu on
the toolbar, select the required classification

Declassifying responses
There may be occasions when you want to return some previously classified responses
to the 100% definitively marked state. When this functionality is enabled you can
declassify responses from either the Response Classification page or the Marking
page.

24
Completing standardisation setup
This process applies to Principal Examiners only.

Note: Ensure that you are happy with your selection of classified responses and
definitive marks before proceeding as there is no way to correct any mistakes once
standardisation setup is complete.

Once you complete standardisation setup, the definitive marks remain fixed for
practice, qualification and seeding responses.

To complete standardisation setup:

1. On the E-standardisation tab click the Complete Setup button.

2. A confirmation message displays.

3. Click Yes only if you are satisfied with your qualification samples and definitive
marks.
Notice that the QIG status shown on the Home page changes from In Progress to
Complete.

What happens next?

At the same time that the Complete Setup button is clicked, RM Assessor
automatically performs these actions:
Makes practice and qualification responses available to all examiners.
Definitive marks are submitted to the IB.

25
 The marks of submitted, 100% marked, provisional responses or definitively
marked responses that have not been classified are credited to the provisional
examiners. (These responses then appear in the examiner's closed responses
Worklist as normal marking.)
Un-submitted provisional responses are returned to the marking pool to be marked
during live marking.

Standardisation reports
Note: The header of each report shows the date and time the report was run, and the
date and time when the data was valid.

There are reports available in order for you to monitor standardisation setup
progress and to monitor your teams progress.

To run a report:
1. At the top right of the page, click the reports icon.

The List Reports screen is displayed.

2. Click a report on the list. A form is displayed.
3. Select the options that provide the data you want to see.
4. Click Execute Report. The report is displayed.

You can click any underlined item on a report to drill down into the data and see more
detail. If required, use the controls on the report toolbar to:
Export the report to various file formats, for example, Excel.
Print the report.
When you have finished with the report, click Back to return to the list of reports, or
click Close to shut the List Reports screen.

Standardisation Set-up
This report shows the standardisation setup targets and the progress made to meet
these targets.

26
 The report is useful to the standardisation team in the period between the start of
scanning and the standardisation meeting to ensure that sufficient provisional marking
occurs, and during the standardisation meeting to monitor progress towards the
targets.

Standardisation Progress
This report monitors the progress of standardisation as examiners work towards
approval by marking and submitting their qualification samples.
The report can be run for a single component or for all components an examiner is
associated with. At the top-level, it displays a row per QIG and a count of the number of
examiners that have reached each milestone in the standardisation process. It is
possible to drill down for an individual QIG and view a line per examiner to review what
stage each individual examiner is at.
The report is most useful in the early stages of marking to ensure that examiners are
making progress towards being approved. This report shows the examiner side of the
approval process while the Supervision Progress report shows the team view.

Standardisation Feedback
The Standardisation Feedback report is available when examiners have marked and
submitted their qualification responses. The report displays:
A summary of how many of a subordinate's qualification responses were accurate,
within tolerance and outside of tolerance.
A detailed analysis of each submitted qualification response showing:
For each of the questions within the QIG and for the total mark, a display of the
definitive mark, the examiner's mark, and the deviation between these (where the mark
deviation also shows if the examiner's mark was under or over the definitive mark).
The total deviation of the examiner's response.
Your subordinates can see their own standardisation feedback report in RM Assessor.

Promote to seeding
As Principal examiner it is possible to promote responses to seeding during the
marking period. This can either be from your own submitted responses or from the
submitted responses of one of your team members.

Note: You can only promote closed live responses to become seeds, not closed
practice or qualification responses.

The marks for each promoted response automatically become the definitive marks, so
you don't need to set these as a separate task.

The Promote to seed option is not available if:

There is no seeding target set for the QIG (as displayed in the standardisation
setup area),
and
No markers in the QIG have any seeding targets.

To promote a live response to seeding from a subordinate's submitted live

responses:
1. Log in as a Principal Examiner and on the Home page, click Team Management.

27
2. To open the required QIG, either double click it or click the relevant Open button.
The Team Overview page opens.

3. If not already selected, in the left hand navigation bar, select Marking and then
Team Overview.
4. Double click on an appropriate team member to look at their work.

5. Select Closed responses and review any submitted responses for suitability.

6. Select one or more responses that you want to promote to seed and click Promote
to Seed.

28
 Note: If you promote a response with a status of Pending (still within the grace
period), upon promotion the response will move directly to a Closed state.

7. A dialog box displays asking you to confirm that you want to proceed.

8. Click OK to proceed, otherwise click Cancel.

A confirmation message displays.

9. Click OK.
10. Repeat to add more seeding responses as required.

To top up the number of seed responses from your own submitted live
responses:
1. Log in as a Principal Examiner and on the Home page, click Go to Worklist.
If not already selected, in the left hand navigation bar, select Marking and then
Live.

2. Select Closed responses and review any submitted responses for suitability.
3. Select one or more responses that you want to promote to seed and click Promote
to Seed.

Note: If you promote a response with a status Pending (still within the grace
period), upon promotion the response will move directly to a Closed state.

4. A dialog box displays asking you to confirm that you want to proceed.

29
 5. Click OK to proceed, otherwise click Cancel.
A confirmation message displays.

6. Click OK.
7. Repeat to add more seeding responses as required.

The approval process for examiners

Auto-approval
The Principal examiner and standardisation team members will be automatically
approved to mark once e-standardisation set up is complete.
For examiners who are not part of the standardisation team, they will be required to
mark and submit the practice responses before they can commence with the
qualification responses. The purpose of qualification is to get all of your team to the
appropriate marking standard and to make sure they are using the markscheme
correctly. To help you achieve this, the specially prepared qualification responses are
allocated to each examiner. These responses must be marked to the required standard
and submitted by each examiner before they can be approved to mark any live
candidate responses.
RM Assessor compares the marks of the qualification responses with the definitive
marks awarded during the standardisation meeting and if the qualification responses
are marked to the required standard the examiners become Approved and are able to
continue seamlessly onto live marking. However, if the qualification responses are not
marked according to the markscheme (that is, that the marks awarded are not within
the set tolerance), the examiner will remain with a status of Not Approved and the
examiner appears in the Senior Examiner pool as requiring action by a Team Leader.
The examiner will need to revisit the practice scripts to correct their understanding of
the markscheme, be provided with feedback from a team leader within the Senior
Examiner pool and then be given further qualification responses to mark. This is known
as second qualification.

30
 If the examiner is unable to mark the second qualification responses to the required
standard despite the further advice given, the IB Assessment Centre will review their
marking and make a decision as to whether they continue marking for the session.

Feedback Required status

Feedback required status is based on seeding response marking accuracy, and the
threshold level which is set. It is the responsibility of the Senior Examiner pool to deal
with any members of the examiner team the first time they are stopped from marking
and have a status of Feedback Required.
If an examiner becomes stopped with a status of Feedback Required for a second time
they will require action by a colleague at IB Cardiff and will be contacted by the
assessment officer responsible for their subject. The Principal Examiner or Team
Leader cannot action an examiner who has been stopped for the second time.

Viewing your team

When you log in as a team leader or Principal Examiner your screen will display
Marking and Team Management tabs, and will normally open on the Marking page;
an E-standardisation tab will also be displayed if you are a standardisation team
member.
If you have any locked examiners awaiting action a warning message is initially
displayed followed by the Team Management page as shown below.
Any tab with the star icon ( ) indicates you have actions to perform.

Approval management
Once the Team Management tab is selected the default view shown is the Approval
Management screen. When this function is in use a group of senior examiners, defined
by IB, forms a pool allowing anyone in the pool to deal with any examiners who require
assistance. Use of a senior examiner pool ensures that these examiners are dealt with
in a timely manner since they are not reliant on their immediate team leader who may
be working in a different time zone.
Examiners will require assistance for one of these reasons:
Not marking the first sample of qualification responses within the required
tolerance.
Submission of a specified number of seeds outside of the required tolerance and
being stopped from marking for the first time.

Note: Examiners who do not meet the auto approval criteria after completing their
second qualification sample or who are automatically stopped from marking for a
second time will be actioned by the IB Assessment Centre.

You can see a list of examiners on your Approval Management page for which you
have permission to action.

31
The filter buttons at the top of the list show the total number of examiners requiring
action, the number you have locked, and those that are unlocked. Click a filter button to
display those examiners meeting the selected criteria only.
On the Approval Management page you can:
Filter examiners using the filter buttons or by selected criteria in any column
header.
See the status of examiners awaiting action.
Select and lock an examiner for action. As long as examiners remain locked to you
they cannot be dealt with by anyone else.
Once you have locked an examiner and the examiner is selected you can use the
buttons that appear to view the examiner's submitted qualification and/or seeding
responses, send a message and provide an action (either provide second
qualification or re-approve, depending on the situation).

When you select a locked examiner, and then click View, the examiner's seeding or
qualification responses are listed. This list shows the accuracy of marking and the
mark difference.

You can open any of these responses to see how the examiner's marks differ from
the definitive marks.

32
To lock an examiner to you for action:
In the list, select an examiner, and then click Lock. Yes is displayed in the Locked
column and your name appears in the Locked By column.

To action a locked examiner:

From the approval management screen, select the examiner, and then do the
following:
Send a message to provide the examiner with some additional guidance on
how to improve their marking quality. When a Team Leader sends a message
to another examiner and they are not his/her hierarchical Team Leader from
within the Approval Management pool, the message shall automatically be sent
to examiners Team Leader. Using the Actions button, select an action.

33
Note: Any examiners you fail to action in a timely manner may be unlocked by IB to
be returned to the pool. IB will set a limit on the number of examiners you can lock at
one time, this is usually 3. If you exceed this number you will receive a message
letting you know that you cannot lock any more examiners until you have dealt with
the examiners you already have locked.

The Team Management page lists all of the QIGs for which you have subordinates. On
this page you can:
Click any of the filter buttons to display those QIGs only with:
Open exceptions requiring your escalation or resolution
New messages in your inbox
Locked examiners (those locked for action by you)
Examiners you can assist, who are in the Approval Management screens
awaiting action
Click Show Team Overview to display extra columns
Click Hide Approval Management to hide those columns
Click Open to view the team for a particular QIG
Click to open messages or click to open exceptions
Use Group list by to group the list by session

To view your team and tasks for a particular QIG:

On the Team Management page, click Open on the QIG of interest. A page similar
to the one shown below is displayed.

34
 On the right you will see a list of your team members and their status and performance.
If you are further up the marking hierarchy you will see your team and you can also drill
down to see their subordinates. You can use the My Team (Assistance Required)
and Indirect Team (Assistance Required) buttons to filter the team list to only show
examiners who are suspended or have completed their qualification response marking
and are not approved.

Note: Any examiner withdrawn from marking by system administrators is removed

from the list.

At the bottom left of the Team Overview page a summary of the line currently
highlighted in the examiner list is shown.
On the left of the page you will see the familiar Marking, Messaging and Exceptions
options. However, the sub-options for Marking and Exceptions are configured for
supervisory tasks instead of marking tasks.

To return to the Team Management home page:

Do one of the following:
On the title bar, click Back to Home Page to the left of the QIG name.
On the Team Overview tab, click X.

Supporting your team

Messaging within RM Assessor
Messaging in RM Assessor is very like emailing on your computer. You can:
Send a message to an examiner, automatically referencing a response.
Send messages to individuals or your whole team.
Forward messages.

35
 To send a message, automatically referencing a response:
1. On an examiner's Live Responses page, select a response and then click Send
Message.
Notice that the email form is displayed with the recipient and the related item fields
already populated.

2. Complete and send the message.

To send a message, not referenced to a response, to one or more people:

1. On the Team Overview or Approval Management page, select an examiner and
then click Send Message.
The message form is displayed.
2. If required, click the To button to add more names.

3. Select the additional recipient(s) for your message, or select Entire Team. When
two or more recipients have the same name their unique IDs are added to their
names.
4. Click OK.
5. Complete and send the message.

Finding a response
There may be a number of reasons why you need to find a response. The most
common reason is because an examiner has requested your help and wants you to
look at one or more or his or her responses. There are two methods of finding
responses:
Look on one of an examiner's marking lists.
Use the Find function if you know the display ID of the response.
If an examiner has sent you a message about the response, follow the display ID link to
view the response.

36
 To find a response using its display ID:
1. At the top right of the screen, type into the box the display ID of the response you
want to find and view.

The following message may be displayed.

2. Click Yes. The response you are looking for is displayed.

Sharing a response
This feature allows you to share one of your responses with your immediate team or all
subordinates. You might want to do this to illustrate a particular marking issue or to
show an example of how a response has been marked.

Note: To find and see the shared response your team will need its display ID. You
can conveniently provide this information, and a note letting your team know why you
are sharing the response, by sending a message referencing the response (see
Messaging within RM Assessor).

To share marks:
1. In your Worklist, select the response whose marks you want to share.
2. Click More, and then click Share Marks.

A User Options dialog box is displayed.

3. Select either Share marks with immediate subordinates or Share marks with all
subordinates.

37
 4. Do one of the following:
If you wish to share marks and send a message, click Send Message.
If you wish to share marks without sending a message, Click OK.
Notice that in your worklist column indicators show whether you have shared a
response and with which group, and whether you have sent a message, as shown
in the example below. You can pause your pointer over the share symbol and a tool
tip shows which group you have shared the marks with.

To stop sharing marks:

Repeat the above process, but select Do not share marks at step 3.

Exceptions
Examiners raise exceptions in RM Assessor when they encounter something out of the
ordinary. Depending on the type of exceptions, they are routed to you, the RM
Assessor service provider or IBs assessment officer.
The most likely type of exception to be routed to you for resolution is answer outside of
guidance.
If there are any exceptions to resolve you are notified on the Team Management page
when you log in. The number of QIGs with one or more exceptions is shown on the
Exceptions button. The number of exceptions on a single QIG is displayed on its
exceptions icon. In the example shown below there is only one exception on one QIG.
In practice you may see several exceptions on a single QIG.

Resolving exceptions
When an exception is routed to you, you must either resolve it or escalate it.

To resolve an exception (either original or escalated):

1. Do one of the following:
To see all exceptions for a particular QIG: On the Team Management page,
click the exceptions icon ( ) next to a question paper. The Raised
Exceptions page is displayed.

38
 To see the exceptions for a particular QIG and examiner: On the Team
Management page, open a QIG and then click View for the examiner of
interest. The examiner's Raised exceptions list is displayed.

2. Select an exception in the list to see the details and apply an action.
3. Click the Resolve button. The Resolve Exception form is displayed.

39
 4. Type your resolution in the box, and then click OK. The Success box is displayed.

5. Click OK.

Escalating exceptions
Team leaders can escalate exceptions to Principal Examiners, and they in turn can
escalate exceptions to IB.

Note: Escalated exceptions that are resolved require no further action by the person
that made the escalation. They go straight to the originator, who will normally close
out the exception.

To escalate an exception:
1. Follow step 1 to 2 of the procedure above.
2. Click the Escalate button. The Escalate Exception form is displayed.

3. Type your message in the box, and then click OK. The Success box is displayed.

4. Click OK.

40
 Viewing actioned exceptions
To view actioned exceptions:
On the Raised Exceptions page, select Closed exceptions.

Marking reports available

There are a number of reports available in RM Assessor to help monitor your team.

General Seeding Inaccuracy

For a selected QIG, this report lists all the seeding responses in the system and the
number of times they have been marked accurately, within tolerance or inaccurately.

You can drill down into a particular RIG to see how each examiner marked each
response item. For example:

Marker Seeding Tolerance

For a selected QIG, this report lists any examiners that have been allocated seeding
responses. You can see how many seeds an examiner marked accurately, within
tolerance or inaccurately.

41
 Use this report to monitor the marking of individual examiners against the approved
standard and for post session analysis. You can drill down to a comparison between
the seeding marks and the definitive marks for a particular examiner and response. For
example:

Marking - Daily Seeding Feedback

For a particular QIG you can see how many questions in your, or a subordinate's,
seeding responses are accurately marked and how many are either under or over the
definitive marks. Where there is a difference in the marking an average deviation mark
is provided.

Your subordinates can see their own daily seeding feedback in RM Assessor.

Examiner Feedback
It is not necessary to identify scripts as suitable feedback scripts for your team
members or to complete an examiner feedback report on IBIS for e-marked
components.

Requesting a re-mark
There may be situations in which you might request a re-mark, for example if one of
your team members has highlighted a problem script to you and you are required to re-
mark it. This can be a subordinate examiner's closed response or one of your own
closed responses. The system places the requested response(s) in your Re-mark work
list.

42
Note: Please contact IB Cardiff should you need to request more than 10 remarks
from the assistant examiners in your team.

To re-mark a closed response:

1. On either an examiner's or your Live Responses page, select Closed responses.

2. Select the response(s) you want to re-mark, and then click Re-mark Response.

3. Click OK or Cancel as appropriate.

Note: It is not possible to raise a supervisor remark on internal assessment

responses.

43
Enquiry upon results re-mark
Seeding responses may be used during enquiry upon results re-marking. As with live
marking, if you mark and submit a seed outside of tolerance you will have quality
feedback to review on the seed but you will not be stopped from marking. When
undertaking an Enquiry upon results re-mark, if you change any marks in comparison
to the original marks, a reason for the change must be given, as shown below, before
the response can be submitted. In the enquiry upon results re-mark worklist a column
with ticks and crosses is used to indicate whether a reason for change has been
entered against any re-marked responses (or whether no reason is required given the
re-marks are the same as the original marks).

Note: If you try to save or submit a re-marked response in the worklist, the reason for
change pop up box shown above is automatically displayed and must be completed
before you can proceed.

Getting Support
For support with the use of RM Assessor, please contact the IB Assessment Centre, using
either the dedicated e-marking mail box emarking@ibo.org or by calling the e-marking
helpline +44 29 2051 7114.

44