CoBaTrICE has identified and described a number of different work-place based assessment methods currently in use in specialist medical training programmes and means by which evidence of competence may be recorded. Such strategies may be adopted at a national level in order to standardise the assessment of ICM trainees. These include:

• Workplace-based observation

• Simulation
• OSCE
• Formal examination
• Formal certification
• Structured case summaries
• Case based discussion
• Log books

Work-place based observation

What is it?

Trainees will be observed in the workplace by different members of the health care team and on a number of occasions, over a period of time. Such observations can form the basis of both informal and formal assessments. The assessment tools used may vary according to national guidelines, duration of training and whether specific knowledge, skills or behavioural aspects of the trainees practice are being assessed.

What is it used for?

Used to assess knowledge, skills, behaviours and attitudes in the workplace under normal working conditions.

How is it used?

Interactions between the trainer and trainee during routine clinical work offer frequent opportunities for informal observation. Over a period of time, these encounters provide the trainer with a general overview of the trainees performance and may highlight strengths / weaknesses which can be the basis for feedback to the trainee, identify future learning needs, or may indicate the need further formal assessment based on direct observation (using specific tools).

Observations may be direct (real-time - trainer is with the trainee throughout the encounter) or indirect (retrospective - based upon the outcome of the encounter, e.g. plan of management, patient records etc). Concurrent or subsequent brief questions may be asked to assess underlying knowledge.

Observations may be formally structured and documented by using a variety of assessment tools such as:
Multi-source feedback / Peer assessment tool
Direct observation of procedural skills (DOPS)
Clinical Evaluation Exercise (mini-CEX)

Direct Observation of Procedural Skills (DOPS)

What is it?

Direct observation of a trainee performing diagnostic and interventional procedures during normal ('routine') clinical practice

What is it used for?

Used to assess the trainees? procedural skills (associated attitudes / behaviours) and formally document this during normal clinical practice. Used to observe the trainee-patient interaction and the process as a whole, not just the procedure itself. Offers an opportunity for the trainer to provide feedback to the trainee.

How is it used?

Trainee-led: the trainee identifies the assessor and the procedure under the guidance of their trainer (via their learning agreement). It is the trainee's responsibility to ensure the required number and types of assessments are completed (according to national requirements).
The trainee should be observed performing a practical procedure in the workplace - they should be allowed to proceed as far as possible without interference, but patient safety should be maintained at all times. Concurrent or subsequent brief questions may be asked to assess underlying knowledge.
The assessment is recorded using a structured check-list which enables the assessor to provide feedback to the trainee.

Examples:

UK: Direct Observation of Procedural Skills (DOPS)

Links & references:

UK: DOPS - Description: http://www.mmc.nhs.uk/pages/assessment/dops

Assessment proforma, guides for trainers and trainees and video clip demonstrating the use of DOPS: http://www.hcat.nhs.uk/foundation/DOPS.htm

UK: Intercollegiate Surgical Curriculum Project - CBD Guidance for trainers and trainees http://www.iscp.ac.uk/Documents/CBD_Guidance.pdf

Clinical Evaluation Exercise (CEX)

What is it?

Direct observation of a normal ('routine') clinical encounter.

What is it used for?

Used to assess the trainee's skills, attitudes and behaviours (and formally document this) during normal clinical practice & offer an opportunity for the trainer to provide feedback to the trainee.

How is it used?

Observations may be for a limited period of time (e.g 5 minutes), or last the complete period of the patient encounter (e.g 15 - 20 minutes) according to the assessment tool in use.
Trainee-led: the trainee identifies the assessor and the clinical encounter under the guidance of their trainer (via their learning plan).
The trainee should be observed delivering patient care (i.e doing what they would normally do in that situation) - they should be allowed to proceed as far as possible without interference, but patient safety should be maintained at all times.
The assessor notes strengths & weaknesses using an assessment proforma; this may include concurrent or subsequent discussion with the trainee (to assess underlying knowledge).
Immediate feedback is provided to the trainee after the encounter.
Feedback is most important - it provides information that guides and develops the trainee's practice by identifying and agreeing strengths, areas for development and a plan of action.

Examples:

Mini-CEX (UK) = 15 minute snapshot of trainee/patient interaction

Links & references:

Mini-CEX Description: http://www.mmc.nhs.uk/pages/assessment/minicex

Assessment proforma, guides for trainers and trainees and video-clip demonstrating use of mini-CEX: http://www.hcat.nhs.uk/foundation/mini-CEX.htm

Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini CEX: a method for assessing clinical skills. Ann Int Med 2003; 138: 476-481

Multi-source Feedback (MSF)

What is it?

Multi-source feedback (MSF) is assessment of professional behaviour and attitudes.
It is also known as 360° assessment or team assessment of behaviour. It consists of a rating form completed by multiple known people (subordinates, peers, superiors) in the workplace.

What is it used for?

To seek views of other workers on the doctor?s professional attitudes and behaviour in day-to-day work. It is important to assess professional attitudes and behaviour for a number of reasons:
Medicine is a fusion of interpersonal and cognitive skills
Problems with attitudes are common to borderline trainees
Poor practice, uncorrected, is repeated, with greater confidence
Assessment drives learning and improves performance
Working environment & patterns present specific challenges e.g. Shift working - handover, teamwork
Public expectations - professional self regulation

How is it used?

Forms are distributed to a predetermined number of raters (maybe specification of staff categories) by the trainee. Completed forms are returned to a central point and are summarized. Feedback is provided during a meeting between the trainee and the trainer, supervisor or college tutor. Remedial action is planned as appropriate.
Number of raters and the format of the rating scale varies according to the tool in use.
Outcome: Not a pass / fail event in itself. Used for the purpose of providing meaningful feedback to the trainee about their attitudes/behaviours from different members of the team.

Examples:

UK: Team Assessment of Behaviours (TAB)
UK: Peer assessment Tool (mini-PAT)

Links & references:

TAB assessment proforma http://www.mmc.nhs.uk/download_files/360-Team-Assessment-Behaviour-TAB-Form.doc

Mini-PAT assessment proforma http://www.hcat.nhs.uk/foundation/mini-PAT.htm

Evans, R., G. Elwyn, and A. Edwards, Review of instruments for peer assessment of physicians. BMJ 2004. 328(7450): p. 1240.

Ramsey PG, Wenrich MD, Carline JD, Larson EB. Use of peer ratings to evaluate physician performance. JAMA 1993; 269: 1655 ? 1660

Ramsey PG, Carline JD, Blank LL, Wenrich MJ Feasibility of hospital-based use of peer ratings to evaluate the performances of practising physicians. Academic Medicine 1996; 71: 364-370

Simulation

What is it?

Simulation is artificial representation of real situations. A simulation may use mannequins, actors or computer-based programmes. Simulation centres create a mock ward environment. Simulations allow trainees to practice skills in a safe, controlled environment and they can also be used for assessment purposes.

What is it used for?

Simulations can be used in the assessment of decision-making, communication and technical skills.

Mannequins of the whole body, body parts (eg an arm) or systems are used to simulate a wide array of situations, for example, heart attack or allergic reaction. Computerized mannequins can be programmed to mimic the patient in different scenarios (eg trauma, airway management) allowing skills to be assessed (eg intubation).

A 'standardised patient' is an actor playing the role of a patient with specified symptoms. These may be used in the assessment of, for example, history taking, diagnosis, communication skills.

Computerized simulations are usually screen-based graphical representations. They are suited to physiological and pharmacological modelling and can be used in the assessment of knowledge and decision-making skills.

How is it used?

Assessment using simulations may be much like work-place based observation. They may be part of an OSCE (objective structured clinical examination) - standardised clinical problems presented at a number of stations in a simulation centre or skills laboratory. Assessments of simulated activity may employ rating scales. However, simulations are resource, cost and time - intensive therefore may be of limited use as a work-place based means of assessment.

Simulations can not take the place of direct experience with real patients and assessments are often used formatively to provide feedback on developing knowledge, skills and attitudes.

Examples:

Links & references:

Bryne AJ, Greaves JD. Assessment instruments used during anaesthetic simulation: review of published studies. Br J Anaesth 2001;86:445-50

Devitt JH, Kurrek MM, Cohen MM, Cleave-Hogg D. The validity of performance assessments using simulation. Anesthesiology 2001; 95: 36-42

HolxmanRSCooper JB, Gaba DM, Philip JH, Small SD, Feinstein D. Anesthesia crisis resource management: real life simulation training in operating room crises. J Clin Anesth 1995; 7:675-87

Moorthy K, Munz Y, Adams S,Pandey Y, Darzi A. A human factors analysis of technical and team skills during procedural simulations. Br J Surg 2004;90(suppl 1): 88-9

Objective Structured Clinical Examination (OSCE)

What is it?

Direct observation and objective testing of trainee performance during planned clinical encounters using simulated patients and a marking schedule prepared in advance.

What is it used for?

Used for the assessment of knowledge, skills, behaviours and attitudes. OSCE is an effective tool for assessing the ability to obtain information from a patient, establish rapport and communicate effectively, interpret data and solve problems.

How is it used?

The OSCE involves several 'stations' in which trainees are expected to perform specific clinical tasks within a specified time period. To complete the examination trainees rotate through a series of stations (from 2 - 20). Clinical tasks or scenarios that would be commonly encountered in the hospital / ICU setting are included (e.g. performing a physical examination or procedure). A trained observer scores the trainees performance.
Stations can involve different methods of testing including interaction with a standardised patient, multiple choice questions or short-answer questions.
However, OSCEs are resource and time intensive therefore may be of more use at a regional / national level rather than as a means of local assessment.

Examples:

Links & references:

Formal Examination

What is it?

A standardised written and/or oral examination set by an authorising body (e.g. national training organisation)

What is it used for?

Standardised assessment of knowledge

How is it used?

• Non work-place based.
• Standardised examination conditions for all candidates - time limited; no consultation / reference materials permitted etc.
• A curriculum is provided upon which all questions should be based, but the candidate does not know the questions in advance. Each examination uses different questions.
• Written examinations may comprise multiple-choice questions (MCQs), short answer questions and/or essay questions. Marking may be positive (marks awarded for correct answers) or negative (marks awarded for correct answers & deducted for incorrect answers).
• Oral examinations may include case based discussions, OSCEs, question and answer.
• Outcome: pass / fail based on numerical score.

Examples:

European Diploma of Intensive Care (EDIC) part 1
Diploma of the Irish Board of Intensive Care medicine (DIBICM)

Links & references:

European Diploma of Intensive Care (EDIC) http://www.esicm.org/PAGE_europeandiploma?bq6t

Diploma of the Irish Board of Intensive Care medicine (DIBICM) http://www.icmed.com/i_b_i_c_m.htm#Diploma%20in%20Intensive%20Care%20Medicine

Formal Course Certificate

What is it?

A formal certificate of satisfactory completion of a nationally recognised course, which provides standardised training and assessment, and is issued by a professional / academic organisation.

What is it used for?

A nationally recognised course which tests the candidates' knowledge and relevant skills / behaviours to a common standard can be regarded as an acceptable means of external assessment. This clearly defined set of knowledge / skills should not need to be re-tested at local level (unless a trainer subsequently identifies this as an area of concern). The certificate may be regarded as 'transferable evidence'

How is it used?

Minimum requirements:

The certificate and supporting evidence should include:

• Title of course
• Issuing authority (professional or academic organization)
• The candidates name
• Date of issue and expiration (as applicable)
• Outline of the topics covered & type of assessment

Types of courses:

• Courses based on well-established nationally accepted protocols or guidelines (e.g. ALS)
• Courses which provide training about rare topics / conditions that may otherwise not be experienced / assessed in the workplace (e.g. disaster management course)

Outcome: pass / fail (+/- feedback)

Examples:

ALS / PALS / ATLS certificate
Fundamentals of Disaster Management (FDM)

Links & references:

Structured Case Summaries

What is it?

A written report that summarises a case the trainee has encountered during their ICM training and, with reference to relevant literature, reflects upon the management of this case.

What is it used for?

Used as both a learning and assessment opportunity:
Learning opportunity:

• reflection on practice
• review of literature
• information retrieval skills (IT / literature search)
• principles of evidence based medicine
• presentation skills (e.g. preparation of written report)

Assessment opportunity:

• management of the patient (retrospective) & future management strategy
• critical analysis of evidence (identification of suitable sources & critique)
• application of theory to clinical practice
• self-reflection & identification of learning needs ('self governance')

How is it used?

A minimum number of case summaries (usually 10) are completed during ICM training and submitted to the educational supervisor / trainer who confirms that they have been completed to an acceptable standard.
Topic:
The topic is selected by the trainee in discussion with the trainer. Summaries should cover a broad range of topics relevant to intensive care practice - to complement an area of special interest, provide evidence of competence related to prior experience or help to develop an area of weakness for the trainee.
Content:
Report of approx. 1000 words. Subheadings (used as a guide):

1. 1. Clinical problem

1. 2. Relevant management (with reference to literature)
2. 3. Additional information
3. 4. How would you change your future management?
4. 5. References / Bibliography

Presentation options:
Written report (review and discussion with trainer or basis for viva)
Presentation at a departmental meeting (e.g. 15 minute presentation)
Basis for submission of a peer reviewed publication
Outcome: satisfactory / non satisfactory judgement made by trainer.

Examples:

UK: IBTICM requires completion of 10 expanded case summaries E.g. Example of an expanded case summary prepared by a UK Trainee

41.8 KB

Links & references:

IBTICM The CCST in Intensive Care Medicine - Competency Based Training and Assessment: Part II (version 6). http://www.rcoa.ac.uk/ibticm/docs/CBTPart2.pdf

Case Based Discussion

What is it?

Structured discussion between the trainer and trainee of a challenging clinical case which has been managed by the trainee. It uses a patient record as the basis for discussion to explore clinical reasoning and management.

What is it used for?

Used to assess clinical judgement, decision making and the application of knowledge in relation to patient care. It allows the trainee to explain the reasoning behind choices they have made. It provides an opportunity for the discussion of the ethical and legal framework of practice and facilitates feedback in order to guide learning. It is not focussed on the ability to make a diagnosis. As an actual patient record is used, it also allows the trainer to evaluate the quality of record keeping and presentation of the case.

How is it used?

• A minimum number of assessments may be set by the national training authority
• Trainee-led: trainee must ensure that the trainer is aware of issues within their patient cases that offer learning opportunities for discussion and that the required number of assessments are completed during the training placement. Cases may also be initiated by the trainer.
• Trainer discusses the case in depth with the trainee for approx 15 - 20 minutes (what occurred, considerations and reasons for actions). An assessment form is completed by the trainer who then provides immediate feedback to the trainee (5 mins).
• Feedback is most important - it provides information that guides and develops the trainee?s practice by identifying and agreeing strengths, areas for development and a plan of action.
• Not a viva-style assessment

Examples:

UK: Case based discussion (CbD)
USA / Canada: Chart simulated recall
Europe: EDIC Pt II.

Links & references:

UK: Case based discussion (CbD): http://www.mmc.nhs.uk/pages/assessment/cbd

UK CbD Assessment proforma:
http://www.hcat.nhs.uk/foundation/forms/CbD for F1 2005-6.pdf
http://www.hcat.nhs.uk/foundation/forms/CbD for F2 2005-6.pdf

UK: CbD training video-clip: http://www.hcat.nhs.uk/foundation/videos/CbD.wmv

UK: Intercollegiate Surgical Curriculum Project - CBD Guidance for trainers and trainees http://www.iscp.ac.uk/pdf/assessment/CBD_Guidance.pdf

Log book

What is it?

A log book is record book or file. It is not an assessment tool as such but is evidence of experience.

What is it used for?

A logbook is used to record experiences and patients treated, identifying practical skills and procedures. In addition, a log book can be used to record training placements, teaching programmes undertaken, courses or conferences attended, audits, publications etc. It informs educational agreements and allows the trainee to reflect on their progress and learning needs.

How is it used?

It is used to provide evidence of the volume of experience. Log books should record the date of the experience and the role of the trainee. Some log books provide checklists of, for example, practical procedures and emergency situations, with a column to indicate whether the experience was observed. Space is often provided for the trainee's reflections on the experiences.

Some training programmes set a minimum number of specific experiences which must be achieved during training - the log book provides a means of providing evidence to meet this requirement

Log books are used at meetings with the trainers to review experience. They are especially useful when goals for a new placement (or a learning contract) are being established.

Outcome: minimum numbers of different experiences may be required nationally, in which case the logbook may be used to pass / fail a candidate, but generally they provide a source to assess the quality and breadth of the trainees' experience, rather than a pass/fail event.

Examples:

Examples of logbooks from Spain, Poland & Latvia (in national languages) will be available to view here soon

Links & references: