phd assessment criteria

• Graduate Research School
• Assessment criteria

Further information

• Classification of theses

Examiners are asked to comment on the following when examining thesis.

• The thesis as a whole is a substantial and original contribution to knowledge of the subject with which it deals.
• The student shows familiarity with, and understanding of, the relevant literature.
• The thesis provides a sufficiently comprehensive study of the topic.
• The techniques adopted are appropriate to the subject matter and are properly applied.
• The results are suitably set out, and accompanied by adequate exposition.
• The quality of the English and general presentation are of a standard for publication.

Master's

• The thesis is a substantial work generally based on independent research which shows a sound knowledge of the subject of the research, evidence of the exercise of some independence of thought and the ability of expression in clear and concise language.
• Any other examinable component submitted in partial fulfilment for the award of master which you are examining demonstrates technical accomplishment and imaginative resource and/or advanced technical and interpretative accomplishment, as appropriate (if applicable),
• The thesis provides a sufficiently comprehensive study of the topic. Any other examinable work shows sufficient master of technique and/or style (if applicable).
• The quality of English and general presentation are of a standard for publication.

• Current Students
• Examination process
• News and events
• Research proposal
• Your candidature
• Confidentiality and IP
• Editorial Assistance
• Style and format
• Thesis as a series of papers
• Graduate research coordinators
• Resources and training
• Funding, grants & prizes
• Policies and rules
• Postgraduate forms
• Researcher Training Framework

Aarhus University logo

Graduate School of Health

Guidelines for assessment of phd dissertation and phd defence.

The assessment committee assesses the academic quality of the PhD dissertation in question. Prior to the submission of a dissertation, the main supervisor and the management of the Graduate School of Health have ensured that the PhD process has been satisfactory and that all formal requirements have been met. It is the responsibility of the chair of the committee to keep the other members informed about procedures, deadlines and practical matters, including travel arrangements and accommodation in connection with the publicdefence.  

General requirements regarding the content of the PhD dissertation

As stated in the Executive Order of 25 March 2013 regarding PhD programmes at universities, the PhD degree programme trains PhD students to undertake research, development and teaching assignments at an international level. The PhD degree programme concludes with a submission of a PhD dissertation which must document the PhD student’s/author’s ability to apply relevant research methods and to conduct research work meeting the international standards for PhD degrees within the field in question.

The Graduate School of Health recommends that the PhD dissertation consists of a review and a number of papers, based on original data from the PhD project, equal to 3 years of scientific work. The papers should either be published in international journals, accepted for publication in international journals, submitted for publication in international journals or available in manuscript form ready for submission to international journals or in preparation. This corresponds to the international level required in the field of health sciences. The PhD student must usually be first author of the papers.

Co-author declarations must be included for each contribution where the submitted PhD dissertation includes work by several authors. First- and last author, main supervisor and the PhD student must sign the declaration, in which is should be indicated whether the paper in question has previously been included in a thesis or a PhD dissertation.

The review (30-50 pages) should normally contain the following:

• Table of contents and introduction
• Problem definition and hypotheses, including a summary of relevant literature supporting the issue and the hypotheses
• Clarification of considerations in regard to choice of method(s), including a presentation of the methodology used, which should reflect a good understanding of the applied methodology and a critical evaluation of the choice of method(s)
• Brief presentation of the most important results
• In-depth discussion of the results in the light of a critical evaluation of the basic theories and the methodologies used
• Section discussing future aspects, if applicable
• Summary in Danish and English.

The review can also be a review article in which the student’s own findings are incorporated on a par with present literature in the field. The review article should be supplemented by a chapter on the methodologies used (approx. 10 pages). The review article must be published or submitted for publication in an international journal, and the PhD student must be the sole author.

If the PhD student would like additional material to be assessed, such material can be incorporated as a separate chapter in the review or attached as an appendix.

The assessment committee must assess whether the dissertation complies with the abovementioned international standards in the field of health sciences, and the number of papers required will therefore depend on the quality of the papers, the author’s share in the individual works and the scope of the review.  

Very occasionally the research carried out has not resulted in manuscripts or published papers. In such cases the PhD dissertation should be submitted in the form of a monograph. The monograph should be approx. 100 pages with the following format:

• Introduction
• A presentation of the research carried out and the results
• A comprehensive discussion of the results in the light of a critical evaluation of basic theories and the methodologies used
• Discussion of future aspects, if relevant
• Summary in Danish and English
• A statement - prepared in collaboration with the main supervisor - of the PhD student’s share of the work, mentioning any contributions by others. This statement must be signed by both the PhD student and the main supervisor.

In the PhD dissertation it must be indicated how the monograph will be published as a scientific work in the usual international tradition within the specific discipline.

The dissertation must be written in English. In exceptional cases, the dissertation can be in Danish if the student has applied for and been granted exemption.

Final version

The dissertation is screened for duplicate text. After submission it is therefore not possible to make any changes to the wording, correction of spelling mistakes or replace papers in manuscript form with published articles etc. The submitted dissertation is the final version.  

Assessment of the PhD dissertation

The standard scope of an assessment is about 6-8 pages and a specific form is used ( the form can be found on this page ). The assessment should reach the Graduate School of Health no later than six weeks before the public defence in order for the public defence to take place at the agreed upon date.

It is the responsibility of the chair of the committee to divide the work among the members of the assessment committee and to integrate the individual contributions so that the assessment forms a whole without inconsistencies in style or opinions (e.g. use 'we' instead of 'I'). For this reason appendices to the assessment will not be accepted.

The assessment must be approved by the Head of the Graduate School of Health before the public defence can take place. The chair of the committee is contacted in case the assessment is found to need elaboration or clarification. The form used for the assessment follows the following format:

The assessment begins with a description of certain formal requirements:

• Name of the PhD student
• Title of the dissertation
• Date of the appointment of the assessment committee
• Composition of the assessment committee

The content of the dissertation – number of pages, tables, papers etc. – is described by completing the pre-printed paragraph.

The scope of the dissertation should be briefly outlined; what is the purpose, why, who is it aimed at, what are the boundaries of the study, what is included and what is not?

It should not be a summary, but a short description of the range or span of the dissertation.

Evaluation of co-author statements

The co-author declarations must be described, and the assessment must evaluate whether the work involved in the dissertation was carried out by the PhD student. If co-author declarations are missing or inadequate, the Graduate School of Health must immediately be contacted, as these declarations must be available before the dissertation can be assessed.

In assessing the review the following questions should be addressed:

• Is the discussion of the literature satisfactory?
• Does the dissertation include a clearly formulated hypothesis, which was relevant at the time the PhD project commenced in the light of the available knowledge in the research field in question?
• Have the considerations in regard to choice of method(s) been clarified, including a presentation of the methodology used, which should reflect a good understanding of the applied methodology, and a critical evaluation of the choice of method?
• Have the results been critically interpreted and has relevant knowledge in the field been included to a sufficient extent in the interpretation?
• Does the student master relevant terminology and is the wording unambiguous?
• Is the summary comprehensible?
• Is the review satisfactory overall?

Consideration must be taken as to whether the review forms a logical part of the dissertation as a whole. Finally, language and presentation skills can be commented on, if necessary.

The individual papers

The assessment of each paper should begin with a full title, list of authors and status of the paper ("published” (journal, year, volume and page numbers), "accepted", "submitted" or "in preparation"). The following questions should then be addressed:

• Is the aim clear?
• Have the methodologies been described in detail?
• Have the results been clearly described?
• Is the discussion exhaustive?
• Is the paper satisfactory overall?

A mere summary of the paper is not sufficient, as the aim is to achieve a critical assessment on a par with an assessment of a manuscript by international referees.

On the basis of the above-mentioned criteria, the assessment should reach a conclusion that states whether or not the dissertation complies with the international standard for PhD dissertations in the field of health sciences. The assessment committee can recommend that the dissertation should:

• Be accepted for public defence
• Be returned for revision. The PhD student is entitled to a resubmission deadline of at least 3 months (with the possibility of an earlier resubmission to maintain the original date for public defence). The deadline must be stated in the conclusion.
• Be rejected

It is important that the conclusion is in line with the rest of the assessment regarding both criticism and praise. In case of disagreement, the recommendation is based on the majority opinion.

The assessment must be signed by all members of the committee.

The outcome of the assessment

There are three possible outcomes of the assessment of a dissertation:

• The dissertation is recommended for defence
• The dissertation is returned for revision
• The dissertation is rejected

If the recommendation is positive, the PhD student will receive an e-mail with the assessment attached and the defence of the dissertation can take place.

Returned for revision

If the assessment committee recommends revision of the dissertation, the PhD student needs to submit a new, revised dissertation together with a point-by point reply where he/she comments on the assessment committee's remarks. 

The author will get at least 3 months to submit a revised dissertation (with the possibility of an earlier resubmission to maintain the original date for public defence) The deadline must be stated in the conclusion of the assessment.

The graduate school needs to receive any comments to the assessment committee’s decision no later than two weeks after the date where the PhD student received the assessment.

Unless special conditions apply, the same assessment committee will reassess the dissertation. The new assessment consists of a free text assessment of the revised dissertation stating the background, a listing of the changes made as well as an assessment of these changes and a recommendation as to whether the dissertation can now be accepted for public defence. 

The new assessment must be signed by all members of the committee and should reach the Graduate School of Health no later than six weeks after the resubmission.

If the committee finds that the academic level in the revised dissertation complies with the international standard for PhD dissertations in the field of health sciences, the defence can take place, and the PhD student and the assessment committee need to agree on a new date for the defence.     The public defence is based on the revised dissertation, and the printed version must also be based on the revised dissertation.

Recommended revision of revised dissertation (second revision)

If the assessment committee finds that the academic level in the revised dissertation does not comply with the international standard for PhD dissertations in the field of health sciences, but can be accepted with minor adjustments, the committee can recommend the dissertation for a second revision.  

If the assessment committee finds that the academic level in the original dissertation does not comply with the international standard for PhD dissertations in the field of health sciences, and that the quality of the dissertation is below an adequate level for acceptance, the assessment committee should recommend the dissertation for rejection.

If the assessment committee recommends rejecting the dissertation, the graduate school needs to receive any comments on the assessment committee’s decision no later than two weeks after the date where the PhD student received the assessment.

The graduate school management will review the assessment committee’s recommendation, including any comments from the main supervisor and PhD student.    The management then makes the final decision on whether the PhD student is asked to resubmit the PhD dissertation to the assessment committee, resubmit to a new assessment committee or whether the dissertation is rejected completely. 

If the dissertation is in fact rejected, the PhD student will be disenrolled from the PhD programme without obtaining a degree. 

The public defence

The public defence should take place within three months of submission of the dissertation . It consists of a lecture by the PhD student followed by an examination by the assessment committee. The lecture normally takes 45 minutes and should cover subjects dealt with in the dissertation. If the assessment committee recommends a title for the lecture other than the title of the dissertation, the PhD student and the Graduate School of Health are informed. The subsequent examination must comprise an in-depth discussion and critical analysis of selected parts of the dissertation and the lecture. The examination is expected to take at least 1 hour, but is often longer. The audience subsequently has an opportunity to ask the PhD student questions.

Following the examination the assessment committee convenes in a separate room in order to discuss and evaluate the defence and sign the recommendation that the PhD degree should be awarded.Within a week of the defence, the chair of the committee sends a signed assessment of the public defence to the Graduate School of Health based on the discussion with the other members of the committee and using a specific form . 

If the recommendation is that the PhD degree should not be awarded, a letter stating the reasons and signed by all members of the assessment committee should be sent to the Graduate School of Health as soon as possible.  

Faculty of Social Sciences | Lund University

The browser you are using is not supported by this website. All versions of Internet Explorer are no longer supported, either by us or Microsoft (read more here: https://www.microsoft.com/en-us/microsoft-365/windows/end-of-ie-support ).

Please use a modern browser to fully experience our website, such as the newest versions of Edge, Chrome, Firefox or Safari etc.

Assessment criteria

Assessment criteria when accepting new PhD students

Applications will be assessed by at least two senior faculty members, who have no conflict of interest.

Applications are assessed with regard to:

• The quality of earlier scientific papers and reports (e.g. Master's theses, published scientific papers).
• The feasibility, theoretical foundation, and innovative value of the research plan.
• Earlier experience of particular relevance to the research plan (research, clinical work, or similar experience).

These assessments are used to rank applicants. If necessary, top-ranking applications are submitted to a second evaluation round.

Åse Innes-Ker Director of Doctoral Studies Ase [dot] Innes-Ker [at] psy [dot] lu [dot] se (Ase[dot]Innes-Ker[at]psy[dot]lu[dot]se)

Anita Lennerstedt Administrator +46 46 222 91 21 anita [dot] lennerstedt [at] psy [dot] lu [dot] se (anita[dot]lennerstedt[at]psy[dot]lu[dot]se)

Looking for a project?

Project suggestions for aspiring PhD-students.

Lund university

PhD studies at Lund university

To read this content please select one of the options below:

Please note you do not have access to teaching notes, assessing the phd: a constructive view of criteria.

Quality Assurance in Education

ISSN : 0968-4883

Article publication date: 1 June 2003

Research and informed debate reveals that institutional practices in relation to research degree examining vary considerably across the sector. Within a context of accountability and quality assurance/total quality management, the range and specificity of criteria that are used to judge doctoral work is of particular relevance. First, a review of the literature indicates that, although interest in and concern about the process is burgeoning, there is little empirical research published from which practitioners can draw guidance. The second part of the paper reviews that available research, drawing conclusions about issues that seem to pertain at a general level across disciplines and institutions. Lest the variation is an artefact of discipline difference, the third part of the paper focuses on a within discipline study. Criteria expected/predicted by supervisors are compared and contrasted with those anticipated and experienced by candidates and with those implemented and considered important by examiners. The results are disturbing.

• Qualifications
• Quality culture
• Academic staff

Denicolo, P. (2003), "Assessing the PhD: a constructive view of criteria", Quality Assurance in Education , Vol. 11 No. 2, pp. 84-91. https://doi.org/10.1108/09684880310471506

Copyright © 2003, MCB UP Limited

Related articles

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

Assessment criteria and grading scale

The assessment criteria for a doctoral dissertation (thesis) at Åbo Akademi University are as follows:

Choice of subject and research problem

The dissertation must present well-founded, new knowledge, demonstrate innovative approaches and it can also open up an entirely new branch of research. The dissertation must be appropriately defined. Aims, problem-setting, and research questions must be formulated so that they can be answered in a meaningful way.

Conceptual clarity and theoretical mastery of the subject

The work must be conceptually clear and the author must master the underlying theories and be able to conceptualise the research problem. Definitions and conceptualisations must be clearly expressed.

Research methods

The methods must be properly motivated and appropriate, with respect to the research problem. A consistent and insightful application of the research methods is considered a merit.

Material and source criticism

The material must be of a qualitatively high standard and quantitatively adequate with regard to the research problem and the research method. The source material is critically analysed and reliably accounted for by means of correct citations and references.

Presentation of the results

The research results must be presented clearly and logically. The analysis must be consistent and well-founded. The presentation of the results must correspond with the aim and problem-setting.

Conclusions

The conclusions must be systematic, well-founded and drawn from the aim, research problem, material and method.

The work as a whole and linguistic form

The dissertation must be logically organised, the language must be clearly articulated and the graphic layout must be clean and easy to follow. The text must be consistent and logical, focusing on the essential. The argument must be made logically. The author must show proof of critical thinking, originality and independence.

The defence of the doctoral dissertation is also taken into consideration in its assessment.

The grading scale for a doctoral dissertation at Åbo Akademi University is two-graded:

Updated 10.2.2022

• Research Excellence Framework (REF) 2021

Written by Mark Bennett

The Research Excellence Framework, or ‘REF’, is the most detailed and extensive assessment of university research in the UK. First carried out in 2014 (and replacing the previous Research Assessment Exercise), it evaluates research performance across 34 different subject areas at each UK university. The results of the REF are used to determine the proportion of public funding allocated to individual universities for research.

For prospective PhD students, the REF 2021 can be a very useful resource… provided you know what to look for.

That’s why we’ve created a convenient and accessible breakdown of the REF 2021 results .

On this page

What is the research excellence framework.

The REF is used to determine how much public money is allocated to each UK university in order to fund their research. This funding accounts for the largest proportion of research support received by UK universities.

In addition to determining funding allocations, the REF process ensures universities are accountable for the public investment that supports their research. It also provides a means of benchmarking university research performance for other users… like prospective research students!

Is the REF like a university rankings table?

Not as such. The REF does ‘rank’ university research according to its ability to meet given quality levels, but these are standards specified by the REF, not a direct comparison between different universities and their departments. This means the REF itself isn't a PhD ranking .

How can postgraduates use the REF?

In some ways, the REF result is more useful than a simple university league table. It assesses the specific departments that conduct research and allows you to ‘zoom in’ on the criteria that’s most important to you as a prospective research student: research .

What’s more, the REF provides a detailed breakdown of different aspects of university research, including the quality of academic publications, the positive effect of research in wider society and – most importantly for PhD students – the standard of the departmental units and structures in which research is actually produced, including systems for enabling and supporting successful PhD projects!

What research areas and academic subjects does the REF cover?

All of them! Whatever research topic you’re interested in studying for your PhD and whichever university department you’re looking to do your research in, you’ll be able to access an appropriate REF result. This is because the REF organises university research into 34 different broad subject areas, referred to as ‘units of assessment’.

What is a unit of assessment?

A unit of assessment is really just a category of related subject areas. This allows universities to organise their research units and courses as they see fit, whilst still making sense in the REF system. In practice university researchers submit work to the REF using the most relevant unit of assessment.

How is the REF actually undertaken?

There are three main stages to the REF process: the submission of research, its assessment by expert panels and the publication of results.

Submission stage

This is when universities select the best examples of their research and submit them under the units of assessment that are most appropriate to their work.

Assessment stage

Assessment of research for the REF is undertaken by expert panels. There are two types of expert panel. Most are ‘sub-panels’ assigned to each of the 34 units of assessment. In addition, there are also four main panels, responsible for overseeing the broader implementation of specific REF assessment criteria.

Publication stage

The results of the REF 2021 were published on 12 May 2022. There was a four-month hiatus in the exercise in 2020 because of the coronavirus pandemic, hence the slightly delayed publication of the results.

Who organises the REF?

The REF is administered by different bodies responsible for regulating higher education across the UK:

• Research England
• Scottish Funding Council (SFC)
• Higher Education Funding Council for Wales (HEFCW)
• The Department for the Economy, Northern Ireland

These are the groups that oversee the distribution of public research funding across the UK.

Is REF funding the same thing as Research Council funding?

No – the four UK higher education Funding Councils are different entities to the seven UK Research Councils and REF funding is different to Research Council funding .

Research Councils fund specific projects (including postgraduate research projects) based on their individual merits. Funding Councils provide ‘block grants’ to support departments within institutions based on their REF performance. This combination of Funding Council and Research Council funding is known as the ‘dual support system’.

What are the REF assessment criteria?

Assessment criteria are one of the most important features of the REF. Research submissions are evaluated according to three specific criteria: Outputs, Impact and Environment. These are then combined to provide an Overall result for each department’s REF score.

Output is the simplest of the three assessment criteria. It measures the quality of academic work produced by a university’s researchers. Up to four research Outputs can be nominated for each academic whose research a university submits to the REF.

Examples of Output include publications like journal articles and book-length studies, as well as other fruits of academic research such as important data sets, new technologies and intellectual property.

Why does Output matter to PhD students?

A university’s Output score for a specific subject area can tell you how successful its academics are at generating high-quality publications. This might offer an indicator of the potential for you to take part in cutting-edge research projects (and the publications they can generate). A high Output score may also mean that your PhD will be supervised by academics who are recognised leaders in their fields.

Impact assesses the positive effects of a university’s research beyond the academy. Impact is assessed using submitted case studies that demonstrate the past effects of a university’s research as well as strategies for ensuring present and future impact.

The REF defines impact as consisting of ‘any effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment, or quality of life, beyond academia’. It’s worth noting that Impact applies to all academic disciplines and subject areas.

For example, medical science research might generate important changes to public health policy, whilst research in an arts and humanities subject area might have an impact on educational outreach and or underpin important exhibitions in public libraries and galleries.

Why does Impact matter to PhD students?

A university’s past success in demonstrating the impact of its research can indicate the opportunity for PhD students to take part in high profile projects and activities outside the university and to do work that is of an immediate and appreciable benefit to society as a whole. As well as being immensely rewarding, this will look excellent on your CV, whatever career path you pursue with your PhD.

Environment

Environment is arguably the most important REF assessment criteria from the point of view of prospective research students. It measures the quality of the departments, academic units and research groups in which a university’s research is produced – the ‘environment’ in which you will work as a PhD student.

The quality of a university’s research Environment is assessed based on a range of factors. Of particular importance is evidence demonstrating the ‘sustainability’ and ‘vitality’ of research environments. This can take account of the continuity of research funding as well as the structures in place for the effective support, supervision and training of PhD students.

Why does Environment matter to PhD students?

Of all the REF assessment criteria, Environment is the most directly relevant to prospective PhD students. A university department with a high Environment score will be effectively-organised and sustainably-funded. It is also likely to have a proven track record of supporting postgraduate research projects and good strategies in place to ensure a positive and successful experience for future PhD students… like you!

How are the different assessment criteria used to produce an overall REF result?

The three individual assessment criteria are individually weighted and combined to produce an overall REF result for each subject area:

• Output is worth 60% of the overall score
• Impact is worth 25%
• Environment is worth 15%

Members of the expert panels that assess REF submissions are recruited and appointed at an early stage of the REF process. There are a lot of them, but they fall into two general types:

• Practising Researchers
• Research Users

Practising Researchers are usually other academics working in a field appropriate to their assigned subject area. They provide a form of peer-review similar to that used for academic publications.

Research Users are selected from the audience research in a particular subject area is deemed to be of value to. They may also be academics using research data, or they may be representatives of industry, business or policy groups whose work draws on university research.

Quality profiles

A quality profile is the term the REF uses to describe its presentation of the combined result for REF submissions in each subject area. Each item in a submission is ranked according to its quality.

The quality profile for each set of submissions then gives the proportion of its research that meets each ranking level.

The submissions for each subject area are actually given four quality profiles: one for Output , one for Impact , one for Environment and one for an Overall result.

The exact descriptions of standards are tailored to different assessment criteria, but all are ranked from one star to four star:

• Four star - World-leading
• Three star - Internationally excellent
• Two star - Internationally recognised
• One star - Nationally recognised
• Unclassified

Ready to delve into the data? You can view REF 2021 results by university here on FindAPhD.com.

Ready to do a PhD?

Search our project listings to find out what you could be studying.

Want More Updates & Advice?

Mark bennett.

Mark joined FindAPhD to develop our first ever advice articles in 2013 and now serves as our Director of Audience & Editorial, making sure our websites and information are as useful as possible for people thinking about Masters and PhD study. He has a PhD in English Literature from the University of Sheffield, as well as Bachelors and Masters degrees from the University of Kent and the University of South Wales.

You may also like...

What are the best universities for postgraduate study in the USA? Our guide looks at the latest global league tables to identify top-ranked American institutions and help you discover the best postgraduate opportunities.

Looking for the best universities for PhDs in Physics in Australia? Compare ranking tables from top sources here, along with their methodologies.

Looking for the best universities for PhDs in Environmental Sciences in Canada? Compare ranking tables from top sources here, along with their methodologies.

Looking for the best universities to study PhDs in Environmental Sciences in Germany? Compare ranking tables from top sources here, along with their methodologies.

Looking for the best universities for PhDs in Environmental Sciences in the USA? Compare ranking tables from top sources here, along with their methodologies.

Looking for the best universities for PhDs in Chemistry in Ireland? Compare ranking tables from top sources here, along with their methodologies.

FindAPhD. Copyright 2005-2024 All rights reserved.

Unknown    ( change )

Have you got time to answer some quick questions about PhD study?

Select your nearest city

You haven’t completed your profile yet. To get the most out of FindAPhD, finish your profile and receive these benefits:

• Monthly chance to win one of ten £10 Amazon vouchers ; winners will be notified every month.*
• The latest PhD projects delivered straight to your inbox
• Access to our £6,000 scholarship competition
• Weekly newsletter with funding opportunities, research proposal tips and much more
• Early access to our physical and virtual postgraduate study fairs

Or begin browsing FindAPhD.com

or begin browsing FindAPhD.com

*Offer only available for the duration of your active subscription, and subject to change. You MUST claim your prize within 72 hours, if not we will redraw.

Create your account

Looking to list your PhD opportunities? Log in here .

• Coming soon
• Recent changes
• Academic regulations
• Programme and module design and approval
• Assessment, awards and degree classification
• Personal tutoring, student support and development
• Concerns, complaints and appeals
• Registration and attendance
• Research degree policies and guidance
• Progression
• Supervision
• Research programme regulations
• Regulations for programmes which are not currently running
• Student engagement and representation
• Studies away from the University
• Professional Work Based Learning
• Contingency classification and progression regulations
• Exceptional classification and progression regulations
• Exceptional regulations: Covid-19

Progression Review of research students

This page sets out the Progression Review process which applies to research degree students in all years of their programme, including basic elements, possible outcomes and appeals. Its content is relevant to staff and postgraduate researchers registered for level 7 (masters) and level 8 (doctoral) degrees across all of the UK, China and Malaysia campuses.

Search the manual

1. introduction.

Includes:  basic principles; Progression Review activities throughout the year; progression monitoring; Internal Assessors; maximum time for completion of thesis examination

All postgraduate researchers (PGRs) registered on research degrees lasting more than 1 year full-time or 2 years part-time (e.g. MPhil or Professional Doctorates but not MRes etc.) are subject to progression monitoring and formal review.

The basic principles of Progression Review are common to all postgraduate researchers (PGRs) and all years of programmes who go through such a review. 

In all cases, the purpose of Progression Monitoring and formal review is to ensure that the progress towards meeting the required outcomes at each Stage is sufficient to ensure achievement of the doctoral (level 8) or masters (level 7) outcomes to the required standard, and completion of the thesis examination (including any viva voce examination) within the period of registered study.

For more information about the UNQF, please consult the following:

University of Nottingham Qualifications Framework

Relevant adjustments may need to be made for PGRs studying away from the University or following programmes that involve a significant taught element.

Part-time students

For part-time PGRs, all the processes for the Progression Review will happen every other year rather than yearly and periods of time quoted should be doubled. 

Basic principles of PGR progression monitoring and review

Progress monitoring begins through regular supervisory meetings when the PGR first registers on the programme. These recorded meetings continue throughout the period of registered study and thesis completion period, until thesis submission. Formal records of these meetings should be made, agreed and held securely in the PGR’s supervision records. Progression Review information should be given to the PGR at Induction and then reinforced throughout their registered study.

For more information about responsibilities of the supervisor, please consult the following:

Responsibilities of the Supervisor

In-year progress meetings with supervisory teams can be formal or informal, one-to-one or in a group, as is appropriate for the timing and progress of the PGR. 

For the recommended steps and process in planning for PGR Progression Review, please consult the following:

PGR Progression Review consists of formal assessment of progress against the doctoral or other relevant published outcomes through several components.

• Records of progress recorded in the required minimum number of regular supervisory meetings conducted regularly throughout the period of registered study, including information on: agreed research plans and milestones; progress made; any problems encountered and solutions proposed; and any other pertinent information. As these records form part of the formal assessment of PGR progression at Progression Review, they should contain a record of, for example, any extenuating circumstances, or any concerns about progress, or reasons for congratulation on success. Records of progress meetings should be available to all the team members, including the PGR and all supervisors. 
• Formal Progression Review is assessed against progress towards meeting the required outcomes at each Stage of study in the programme, usually through a written report; 
• Completion of mandatory elements such as training and development, taught modules, laboratory rotations, career planning, engagement with professional development opportunities, and career planning are also reviewed.  

The published criteria for Stage 1, Stage 2 and Stage 3 progression are badged against the outcomes in the University of Nottingham Qualification Framework and the QAA doctoral outcomes. They are designed to enable Assessors to take a consistent and transparent approach in determining the progress of PGRs. 

For more information about Assessment criteria for Progression Reviews, please consult the following:

Formal assessment of progression at Stages 1 and 2 requires a meeting involving an Internal Assessor and should be held as an in-person or virtual event. The School may appoint more than one Internal Assessor if appropriate, for example if the research is inter/transdisciplinary in nature. In such instances all Internal Assessors should be actively engaged in all aspects of the formal Progression Review.

The appointment of the Internal Assessor(s) should be initiated within 3 months of registration and the PGR made aware of this, and the appointment should be confirmed at least 3 months before the formal Progression Review begins. It is strongly recommended that where possible the Internal Assessor(s) are involved in the Stage 3 Progression Review for continuity and quality assurance purposes. The Stage 3 Progression Review can be conducted by the supervisory team but should not be conducted by a single supervisor.

For more information on the role, responsibilities and appointment of the Internal Assessor, please consult the following:

Role and responsibilities of the Internal Assessor

Progression Reviews at Stages 1 and 2 should use the core Review elements below, and may include additional methods of assessment deemed necessary by Schools, to assess progress to date against the Stage appropriate published criteria. When including additional elements Schools must pay due regard to PGR and Assessor assessment load.  

If progress is not sufficient to meet the necessary outcomes within the period of registered study, the aim must be to meet these, and complete the research and thesis examination within a maximum of three years (for MPhil) or four years from first registration (for PhD, MVM, MD, MVS, and professional doctorate) irrespective of programme duration. This maximum duration can only be changed in individual cases if the PGR holds an award with terms and conditions that vary this maximum period and the School approves the arrangement. Note that some awards may have a maximum duration of less than 4 years. 

2.  Timing of formal Progression Review and stages of PGR study

Includes:  for all PGRs; different stages of study; purpose and timing of formal Progression Reviews; 4 year PhD programmes with a substantial taught, training and development or laboratory rotation element in Stage 1; recommended latest timings for Stage 3 Progression Review. 

The latest time at which Stage 3 Progression Review (confirmation that doctoral outcomes can be achieved within 6 months) can take place are in the table below.

3.  Minimum elements of Stage 1 and 2 Progression Review

Includes:  for all PGRs; for doctoral PGRs undergoing confirmation of registration; 4 year PhD programmes with a substantial taught, training and development or laboratory rotation element in Stage 1

Progression Reviews at Stages 1 and 2 should use the core Review elements below to assess progress to date against the Stage appropriate published criteria, to enable the PGR to demonstrate that they have achieved the required outcomes at each Stage.  

When including any additional assessment methods, Schools must pay due consideration to assessment load and the balance of PGR workload for Progression Review against their ongoing research progress. It should not be expected that PGRs stop research / thesis writing in order to devote time to the Progression Review. 

Schools must ensure that the following elements are included in their formal Progression Review process:

For PGRs undergoing Stage 1 and Stage 2 Progression Review:

• A formal summative assessment of PGR progress to date. This is assessed against published criteria/outcomes by:

a) a written report by the PGR on their progress to date and 

b) through questioning at a meeting between the PGR and the Internal Assessor and (if the School wishes, other staff who have not previously been closely associated with the PGR's work). 

The meeting should have a clear agenda. It should begin with a reinforcement of what the meeting will cover and include opportunities to discuss progress to date, the PGR’s future research plans, and the extent of progress towards the doctoral outcomes, the wider research environment and its suitability to support the planned research, and the supervision the PGR receives. 

• Plans for completion and submission of the thesis within the period of registered study, and completion of thesis examination in a maximum of 4 years, bearing in mind that PGRs do not have a dedicated ‘writing up’ period, and thesis writing should be integrated alongside the research. 
• Independent written assessments from the Internal Assessor and supervisory team* on the PGR’s performance to date. The Internal Assessor’s report should include information on PGR performance, progress, attendance and engagement throughout the period of study. Any problems encountered or required support identified by the PGR in discussion or in their written report should be noted in the Internal Assessor’s report, where appropriate. The report from the supervisory team should include any concerns with PGR progress to date, successes or problems encountered or necessary mitigations required to maintain progress. 
• Formative feedback on how improvements might be made on both the written report and the meeting from the Internal Assessor. This can be verbal feedback at the end of the meeting. 
• Confirmation that the required minimum number of supervision sessions has taken place, that the records of supervision meetings are available, and that these record the progress made to date and highlight any problems encountered.
• The Internal Assessor should be able to access evidence that the PGR has attended any modules and passed any assessments that form a compulsory part of their research programme.
• The content of any report submitted to a PGR’s sponsor, if they have one, should also be considered as part of the review. 

Note that the Progression Reviews should also consider and take into account any significant research and/or personal impacts that have /may have significant disruptive impacts on research progress. Consideration of this should include information about how the PGR has managed to progress their research in light of any impacts, the extent to which they have mitigated these impacts and how they have had to change their research plans, activities, and goals/milestones accordingly.

After the meeting with the PGR, the Internal Assessor and supervisors should share their independent reports with the PGR for their comment and response. If the likely recommendation is for reassessment, or if there is disagreement in outcome, the reports should be shared in a meeting with the Independent Assessor and/or the supervisors so the PGR has support when this is communicated. This can be during the verbal feedback after the meeting. 

In light of the meeting, and discussion of the independent reports and the PGR response, the Internal Assessor and the principal supervisor should agree a joint recommendation on the outcome to the Head of School. The agreed joint recommendation will be recorded in the PGR’s record. 

4. Elements of Stage 3 Progression Review

Includes:  consideration of progress to thesis submission; minimum elements

The Stage 3 Progression Review should determine whether the PGR is likely to meet the doctoral outcomes within the next 6 months. It should include a detailed consideration of progress, including the progress made with writing the thesis, and a plan to ensure that the thesis is submitted within the period of registered study. If this is thought to be unachievable at Stage 3 Progression Review (for PGRs on 36 or 42 month programmes), plans must be made for degree completion, including thesis examination, within a maximum of 4 years from initial registration.

It is strongly recommended that the Internal Assessor is also involved in the Stage 3 Progression Review for continuity and quality assurance purposes.

In addition to the minimum elements of Stages 1 and 2 Progression Review, Stage 3 Progression Review should include the following elements:

• a detailed evaluation of how the work to date shows that the PGR has already or will meet /achieve the doctoral outcomes. 
• plans for the PGR to achieve any doctoral outcomes that are not yet met, showing how these will be met within 6 months; 
• detailed information on the progress made on writing the thesis, and plans for its completion (noting that the thesis is expected to be completed within the period of registered study, and a detailed plan for thesis completion and submission). 
• discussion of any exceptional circumstances, unforeseen problems and mitigations that have been necessary to ensure progression and timely completion

Note that if the required outcomes relating to the generation of empirical novel research contributions are unlikely to be met within 6 months, the PGR is unlikely to be able to progress to thesis submission and examination within 6 months.  In this case, the Internal Assessor and supervisors should consider the recommendation that the PGR does not progress and whether an exceptional extension to the period of registered study is required. 

5. The possible outcomes of formal Progression Review

Includes:  outcomes where assessors agree; recommendations on progression; circumstances and outcomes when progression is not recommended

Outcomes where the Internal Assessor and supervisory teams agree:

1. Progression to the next stage of study is recommended. 

In order to make this recommendation there should be records that the PGR has been progressing according to plans throughout the year. Progress must be confirmed as satisfactory in the meeting through assessment against the stage-specific criteria, taking into account discipline / field specific differences in doctoral study and structure. The supervisors and assessors should agree that the PGR will be able to meet the doctoral outcomes in the required time. 

Outcomes of approved progression to the next stage of study are:

a) For Stage 1 PhD PGRs, confirmation of their status as PhD candidates.

b) For Stage 1 and 2 progression, re-registration on the same degree in the following academic session.

c) For all stages, provided the PGR agrees, the assessors can recommend a transfer of registration to another, usually higher, degree (e.g. from MPhil to PhD). In this instance, a transfer form must be completed. If a transfer is recommended at Stage 3, and the review did not involve an Internal Assessor, then an interview with the Internal Assessor is required to confirm the transfer to the new degree. 

d) For Stage 3 PGRs whose progress is satisfactory but who are likely to submit the thesis after the end of their period of registered study, the recommendation will be that on completion of the period of registered study they enter Thesis pending. 

e) For PGRs in Stage 3 who are achieving the necessary progress to meet the doctoral outcomes by the planned date, the recommendation will be maintenance of current status up to the end of their period of registered study. 

For recommendations d) and e), the PGR must have met the requirements for the minimum period of registered study, completed the empirical research and not exceeded 4 years of study. 

2. Progression is not recommended and progress is required to be reassessed. 

For this recommendation there will be information in the supervision records that the PGR and supervisory teams have discussed the reasons for unsatisfactory progress, and that attempts have been made to support the PGR to improve. The progress to date should have been assessed in the meeting with the Internal Assessor against the stage-specific criteria and found to be insufficient, taking into account discipline / field specific differences in doctoral study structure. The Internal Assessor and supervisory teams’ independent reports should agree that the PGR requires additional time and support to be able to demonstrate their ability to progress to the next stage at reassessment. In these cases, supervisory records must be kept, and should be available for the re-assessment. 

a) Stage 1 PGRs on a supportive or corrective plan of action remain on probationary status until confirmation of Stage 1 progression. 

b) At all Stages PGRs are registered/re-registered for PhD in the following academic session with an agreed plan of supportive or corrective action in place for a limited time. 

c) For Stage 3 PGRs who have made insufficient progress in research and writing to allow them to meet the doctoral outcomes within the following 6 months, the assessors may recommend a reassessment after a period of up to 3 months. There should be a plan for supportive corrective action during this time to support the PGR to meet the doctoral outcomes by the end of the period of registered study. 

d) For Stage 3 PGRs who have made insufficient progress in research to allow them  to meet the doctoral outcomes and submit the thesis for examination within the following 6 months, the assessors may recommend an exceptional extension to the PGR’s period of registration for up to one further year. Reassessment for ability to progress to thesis  should be done after no more than 6 months. Thesis completion and examination are still expected within the 4 year maximum period.

e) The maximum number of Progression Reviews for a PGR at any stage is limited to 2. If an Exceptional Circumstances claim is submitted and upheld, the Progression Review may be repeated as a First Sit.

3. Progression is not recommended  - suggested change to registration status.

All Stages: With the agreement of the PGR and on recommendation in the joint report, transfer of registration to another degree (e.g. from PhD to MPhil, or MPhil to MRes). In this instance, re-assessment of progress is not required and a transfer form must be completed. 

The reasons for a recommendation for reassessment or registration on another degree must be explained to the PGR as soon as possible.

In cases of referral for re-assessment, the PGR must have the opportunity to discuss the decision with the Internal Assessor and the supervisory team. If necessary, the PGR should be referred to the School Postgraduate Student Advisor (SPSA), Senior Tutor or other appropriate welfare support officer. The supervisory team should offer the PGR support in addressing the outcome of the review, and if necessary, in producing their response to the assessment in the reports.

6. Resolution of disagreement on Progression Review recommendations

Includes:  process to be followed to reach resolution on the recommendation for progression; timeline for resolution. 

If the independent reports from the supervisory team and the Independent Assessor(s) disagree on the recommended outcome of the Progression Review, the following process should be followed.

The PGR should be referred to the supervisors, SPSA, Senior Tutor or other appropriate welfare support officer for support in addressing the outcome of the review, depending on where disagreement lies, and the required support. They should have the opportunity to discuss the Review with the supervisory team and / or Internal Assessor(s). They should have support in producing their response to the assessment either from the supervisors or from other members of the PGR support team. The PGR should receive the feedback from the review and be given an appropriate time to produce their response.

1. If the Internal Assessor recommends progression but the supervisory team does not. 

The independent and joint reports, and the PGR’s response, should be shared with the Head of School or delegate. The Head of School may call a meeting of the Internal Assessor, supervisors and PGR to discuss the case. The Head of School will then make a decision based on the information in the reports and from the discussion as to whether the PGR should progress, or be referred for reassessment. 

2. If the supervisory team recommends progression but the Internal Assessor does not. 

The supervisory team should set up a meeting for discussion of the proposed outcome to which the Internal Assessor; the PGR should also be invited and have made their response to the independent reports. The group should attempt to resolve the outcome by discussion. If there is information that is relevant to the discussion that was not available to the Internal Assessor in the Progression Review documentation and/or meeting and that might affect the judgement of progress, such as extenuating circumstances, this should be made available in the discussions. If agreement can be reached, then the recommendation will follow the process in Section 5.

If agreement on the outcome cannot be resolved through discussion, the case should be referred to the Head of School as in 1 above.

Such discussions should be held in a timely manner, aiming to reach a resolution and recommended outcome within 1 month of the Progression Review meeting. 

7. Requirements for Progression Review re-assessment and outcomes

Includes:  reassessment of progress; support for PGRs

If progress needs to be reassessed at any stage, the PGR must be given detailed information on the goals and requirements for improved performance, an appropriate and defined timeframe in which to meet these, and support in achieving the required improvement. They should also be made aware of the possible outcomes of the reassessment process.  

The format for re-assessment will be the same as for the initial Progression Review. The PGR should update their written report, focussing specifically on how they have met the agreed objectives. A plan for maintained progress should also be produced and considered at the re-assessment meeting. 

Stage 1 PGRs are re-registered on a continued probationary basis until re-assessment. PGRs at stages 2 and 3 are re-registered/continue to be registered for the PhD until re-assessment. 

When a PGR is referred for reassessment, the following information must be provided to the PGR as soon as possible:

• Feedback on the performance in all aspects of the Progression Review, including clear detailed guidance on the requirements for improvement, with SMART objectives/goals and/or milestones;
• Explicit information on the potential outcomes of re-assessment;
• An explicit time/date for the re-assessment of progress, which should give sufficient time for the required progress to be made. Normally this would not be more than 3 months after the original Progression Review. In exceptional circumstances and with the approval of QSC, the time given for improvement for PGRs at stages 1 and 2 may be up to 6 months; 
• The PGR must receive appropriate academic support and guidance to support them to achieve the required improvements. Academic support is particularly important for PGRs in progressing to Stage 3, to ensure that PGRs get on track to meet the required outcomes, and are prepared for the completion and examination of the thesis; 
• If necessary and appropriate, the PGR should be referred for support outside the supervisory team, e.g. welfare or disability support teams.  

8. Outcomes of Progression Review re-assessment

Includes:  the possible outcomes from re-assessment following satisfactory progress; outcomes on unsatisfactory progress; required evidence of progress. 

If the Internal Assessor and the supervisors disagree on the recommendation after  reassessment, they should follow the process for resolution. This should include the Head of School irrespective of where the disagreement in recommendation lies. 

a) For Stage 1 doctoral PGRs, confirmation of their status as doctoral candidates. 

b) For Stages 1 and 2 progression, re-registration on the same degree in the following academic session.

c) For Stage 3 PGRs whose progress is satisfactory but who are likely to submit the thesis after the end of their period of registered study, the recommendation will be that on completion of the period of registered study they enter Thesis pending. 

d) For Stage 3 PGRs who are on track to achieve the necessary progress to meet the doctoral outcomes by the planned date, and move to thesis submission, the recommendation would be maintenance of current status up to the end of their period of registered study at which point they would, if necessary, enter Thesis pending until thesis submission. 

For recommendations c) and d), the PGR must have met the requirements for the minimum period of registered study, completed the research and have not exceeded 4 years of study. A recommendation for registration for a higher degree cannot be made as a result of progression re-assessment.

If performance in the re-assessment does not meet the criteria and the required progress / improvement has not been achieved as agreed by the supervisory team and Internal Assessor, the recommendation by the School to the University should be that, for all PGRs at all stages

Either 

a) the PGR is required to re-register on another, usually lower degree (e.g. PhD to MPhil, MPhil to MRes). 

b) the PGR’s registration is terminated. 

Termination of registration can only be recommended when supported by evidence that the PGR has received written warnings on lack of progress during the period of study, and the period of supported improvement. 

In these cases, all supervisory records and Progression Review paper work must be submitted to QSC for approval.

9. Considerations of Covid-19 or other major impacts 

Where the recommendations to extend the PGR's registration status relate to research or personal impacts resulting from the Covid-19 pandemic, the PGR can confirm the need for an extension to the 4 year maximum period, by either following the extension to the registered period of study, or the extension to Thesis pending procedure, depending on whether further research is deemed necessary or not.

Other significant impacts on PGR progression, either personal or research-related, such as major disruption in access to facilities, should also be taken into account in Progression Review. When circumstances are out of the PGRs control, these should be considered fairly, with respect and consideration for the impact on the PGR. 

Forms and documentation

Includes: flowchart for progression review and submission; progression review forms; request to register for an additional period of study; request to transfer student

UK campus only  

• Registration during thesis pending period form

* this includes a section for Visa and Immigration team approval

UNM campus only    

Unnc campus only .

• School PGR Progression Review Form (Probationary PhD students)
• School PGR Progression Review Form (excluding Probationary PhD students)

UNUK students   

Student services, unnc students .

UNM students 

Staff       , related content.

Portland Building, University of Nottingham University Park Nottingham, NG7 2RD

Legal information

• Terms and conditions
• Posting rules
• Accessibility
• Freedom of information
• Charity gateway
• Cookie policy

Connect with the University of Nottingham through social media and our blogs .

Browser does not support script.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 30 March 2021

Revising evaluation metrics for graduate admissions and faculty advancement to dismantle privilege

• Andres De Los Reyes   ORCID: orcid.org/0000-0003-0214-983X 1 &
• Lucina Q. Uddin   ORCID: orcid.org/0000-0003-2278-8962 2 , 3  

Nature Neuroscience volume  24 ,  pages 755–758 ( 2021 ) Cite this article

16k Accesses

24 Citations

229 Altmetric

Metrics details

• Neuroscience
• Scientific community

Academics are not immune to the biases contributing to persistent inequalities in society. We face an urgent need to overhaul and dismantle current evaluation practices that uphold inequities at multiple points along the academic pipeline. Graduate admissions and faculty advancement are two arenas of gatekeeping in which a reimagining and redistribution of weighting of commonly used evaluation metrics are warranted. We define and promote the use of dynamic, flexible holistic evaluation models that can be implemented by first recognizing and acknowledging the biases that contribute to racial and ethnic disparities in academia. Leaders of academic institutions must step up to drive adoption of these revised evaluation metrics.

Academic evaluation practices and their shortcomings

The COVID-19 pandemic and the racial justice movement in 2020–2021 have placed societal power structures in stark relief against historical inequalities in access to institutions that facilitate upward mobility. Academic institutions dictate the career opportunities of key stakeholders (e.g., students, faculty, and staff). Thus, these institutions maintain a relatively outsized positioning within economic and social hierarchies that are central to day-to-day life and well-being. The time is ripe to examine practices by the academy that have long disadvantaged marginalized communities. The academy should not and cannot be immune to introspection about its own policies, practices, and power structures. Two sets of power structures require immediate attention, due to their current and continued impact on the academic pipeline: systems regarding admission of students into doctoral programs and criteria for the hiring, tenure, and promotion of faculty. Here we seek to contribute to an emerging dialogue on evaluation metrics for these high-stakes decisions in higher education.

The Graduate Record Examination (GRE), a standardized test historically widely used in graduate admissions decisions in the United States and Canada, is biased toward students from higher socioeconomic status and does not predict completion of the doctorate as effectively as presumed 1 . Issues of access have been at the forefront of discussions surrounding the value of the GRE as a tool for informing graduate admissions 2 . The #GRExit Twitter campaign reveals that graduate programs have been dropping the GRE as a requirement in recent years. Conversations about the equity of the online version of the test necessitated by the COVID-19 pandemic surround the concern that the online testing requirements may further disadvantage prospective students from rural and low-income backgrounds 3 . Even if GRE scores were found to reliably index academic career success, the vast majority of graduate students do not pursue careers in academia 4 , further limiting the utility of this examination for predicting future employment and earning potential.

In addition to the GRE, factors traditionally considered in graduate admissions include grade point average (GPA), personal statements, resumes or curriculum vitae, letters of recommendation, and in-person or virtual interviews. Each of these forms of assessment is subject to its own issues surrounding bias and fairness, particularly those involving subjective evaluations. Empirical evidence suggests that undergraduate GPA, while correlated with graduate comprehensive exam scores and faculty ratings of graduate student performance, is not a very strong predictor of degree attainment or time to completion 5 . Personal statements and letters of recommendation are likewise limited in their ability to predict future academic performance, and they are vulnerable to a number of sociocognitive and rater biases 6 . Most surprisingly, previous research experience also appears to be unrelated to other predictors used in graduate admissions, as well as academic performance in graduate school, although this warrants further investigation, as only a small number of studies have been conducted on this topic 7 . It is important to also keep in mind that barriers to involvement in unpaid research factor into who has access to early research experiences 8 . This emerging literature suggests that many sources of information used in the graduate admission process are subject to bias and are not as objective and fair as widely assumed.

Faculty hiring, promotion, and tenure decisions also reflect policies and practices that create structural disadvantages for those who identify with historically marginalized groups. In particular, these decisions are often biased toward indicators of scholarly productivity with clear underlying power structures linked to them. For instance, hiring and promotion decisions are often biased toward individuals whose research can be framed within mainstream scholarly topics favored by senior faculty from majority backgrounds 9 , 10 . The recruitment of faculty of color who study topics related to diversity, equity, and inclusion (e.g., implicit race bias or health disparities in communities of color) is often relegated to specialty hiring initiatives. These initiatives often have little staying power or long-term funding to support repeated hiring efforts.

We argue that these policies and practices along the academic pipeline collectively create gaping holes, and thus we should not be surprised with the result: academia loses to other industries scores of talented individuals from historically marginalized backgrounds who could otherwise infuse into our scholarly discourse vibrant, progressive, forward-thinking lines of work. Consider, for example, that among grant applicants at the National Institutes of Health (NIH), Black investigators achieved lower funding success rates than their White applicant counterparts; this disparity was in part accounted for by differences in the specific aims of work proposed by Black applicants, which were often focused on topics involving research at the community level 11 .

We fear that a variety of institutional structures in academia have created the very environments academics often claim to be against; that is, where bases of knowledge fail to include sufficient diversity of thought. But there are pathways for developing new evaluative systems to reduce disparities in access to opportunities to flourish in the academy. In this commentary, we propose a rebalancing of key considerations across the academic pipeline, focusing on graduate admissions and on the faculty hiring, tenure, and promotion processes. By applying well-established principles of scholarly discourse to all our evaluative structures, we can fundamentally alter our evaluative processes. We close with concrete recommendations for academics to adopt more dynamic holistic approaches to evaluation to redress historical inequalities in academia.

Rationale and recommendations for implementing truly holistic evaluations

Academia is not a meritocracy.

To reduce the influence of systemic inequalities on important decisions regarding graduate admissions and faculty advancement, we must first disabuse ourselves of the notion that academia is a pure meritocracy. Recent work suggests that above a certain threshold of ‘applicant metrics’, the benchmarks traditionally used to measure research success—including funding, number of publications, or the ‘quality’ of the journals in which a candidate’s publications appear—are unable to completely differentiate applicants with and without faculty job offers 12 . This finding highlights the fact that the faculty hiring processes—and indeed all other types of admissions and gatekeeping practices—are necessarily subjective, as the market is flooded with more qualified candidates than there are available positions at every point along the academic pipeline. A survey of credentials that faculty hiring committees look for in candidates lists “scientific/programmatic and general fit” as one of the most valued attributes 13 . These ‘fit’ criteria are highly subjective and difficult, if not impossible, to quantify.

Need for the recognition and acknowledgement of biases

Indicators of scholarly productivity that are widely used by graduate admissions committees and committees for faculty hiring and promotion are biased, with clear underlying power structures linked to them 14 . These power structures confound our interpretation of scholarly work. The impact of this on graduate admissions is described above. Related to faculty evaluations, in Psychology, the power structures at mainstream academic journals often do not include board members with expertise on scholarly topics shared by faculty from historically marginalized backgrounds 9 . From an historical context, the epistemological perspectives that have governed much of the functioning of our academic spaces have their origins in White European Enlightenment. A key element of this perspective involves its presumed neutrality with regard to the racial and/or ethnic backgrounds of scholars and thus the lack of impact of these backgrounds on evaluations of scholarly merit. Yet, the very presumption that our colleagues’ backgrounds have no impact on how we view their work, or that our backgrounds confer no impact on the conduct of our research, may further the status quo when it comes to faculty hiring and promotion. Thus, it has been argued that if universities truly wish to diversify their faculties, they need to move beyond conventional hiring criteria 15 . This will be increasingly imperative to address disparities in academia that will no doubt be further exacerbated by the current pandemic 16 , which has further revealed stark disparities and inequities for individuals who identify as Black, Indigenous, or people of color (BIPOC) 17 .

Much has been written about how citation counts and journal impact factors are limited in their ability to capture research quality and how producing and incentivizing research that is open, transparent, and reproducible should instead be the goal of science 18 . Still, the unspoken rules of tenure and promotion at most research-intensive universities emphasize the quality and quantity of research, teaching, and service as the only important factors under consideration for evaluation. We argue for a dramatic redistribution and reconsideration of these factors. Note that the evaluation of the quality and quantity of research, teaching, and service need not be abandoned entirely. Rather, they might be weighted differently, leaving room for consideration of additional factors not traditionally evaluated, including commitments to mentoring, community outreach, science communication, and contributions to departmental and institutional diversity, equity, and inclusivity initiatives. As key decision-makers in these contexts, we ought to strongly consider including criteria that might be related to or even embedded within research, teaching, and service, but nonetheless deserve a concrete place in decision-making. These criteria include specific expectations for mentoring, community outreach, science communication, and contributions to diversity and inclusivity. Arguably, these criteria are more subjective than other metrics. However, subjectivity should not preclude their consideration, as we often make important decisions based on subjective criteria (e.g., fit) in academia.

Ideally, unbiased evaluation rubrics would be put in place to replace existing biased ones. Originally (and ironically), GRE and journal impact factors were designed to provide unbiased rubrics, and key elements of these rubrics have the look and feel of indices that lack bias, including standardization and the applicability of the metrics across disciplines and fields. Yet, here too one can point to research indicating that the rubrics we thought were unbiased have turned out to contain biases after all. We previously cited evidence that calls into question such indices as the GRE and publication characteristics (e.g., journal impact factor and citation counts) as unbiased indicators. Consider also evidence of bias within the evaluation metrics regarding scholars’ success at receiving grant funding. In US agencies like the NIH, policymakers have spent considerable attention developing seemingly unbiased rubrics to guide the evaluation of grant applications, including quantitative evaluations of proposal significance, study design, and the strength of the investigative team proposed to carry out the work. Nevertheless, recent work indicates that grant reviewers provide ratings based on these ‘unbiased’ rubrics that not only are unreliable, but are accounted for by factors other than the application’s merit, namely reviewer characteristics 19 . Further, as mentioned previously, these same reviews produce racial disparities in funding success rates that appear to be accounted for, in part, by racial disparities among applicants’ proposal aims and topics of investigation 11 . In sum, although we agree that producing a system predominated by unbiased metrics is a worthy goal, we have to also acknowledge that previous attempts to do so have encountered challenges, if not failures, in producing metrics that lack bias. Consequently, we contend that holistic approaches both acknowledge the fallibility of individual metrics and create opportunities for strategically selecting metrics that reduce the likelihood that biases inherent in any one metric unduly influence decision-making.

Towards truly holistic approaches to academic evaluation

If we are to rely less on biased metrics such as the GRE or journal impact factors to make decisions regarding graduate admissions and tenure decisions, respectively, then what criteria should we use instead? One possible path forward would be to move away from over-reliance on these metrics in favor of a holistic approach that more adequately considers each individual scientist’s personal experiences and contributions to society. In proposing a redistribution and redefinition of factors, we ought to also consider how we weigh these factors when making decisions regarding graduate school admissions and faculty hiring, tenure, and promotion. That is, should we continue to weigh one or more of these factors more heavily, relative to the others? We want to emphasize that some of the best work we can do in this space is to rid ourselves of the need for uniformity in weighting across graduate programs, departments, and institutions. That is, in all likelihood, any standardized shift we would make now might produce some changes in inequities, but would not be the ‘correct’ answer for all circumstances.

Towards implementing such a shift, some might argue that we must wait for an evidence base to accrue on proposed changes to the structures we highlighted previously. For instance, should some of our programs propose to pilot possible weighting strategies before beginning discussions like the one we raise in this commentary? We take a two-fold stance on evidence-gathering. First, waiting for evidence on possible alternative strategies, by construction, quashes discussion of these core issues. This is a particularly pressing concern when it comes to the graduate admission and faculty pipelines, because the current structures we rely on for decision-making stand on flimsy evidentiary grounds. Second, by beginning a discussion on alternative structures, we allow ourselves the ability to discuss methods of evidence-gathering for possible solutions to the pressing problems discussed here. Although the call for holistic evaluations is not new per se, it is important to reiterate this notion, as it has not yet been widely and systematically adopted by the academic community. We suggest that there is a need for both an increase in the number of factors considered during evaluation and a redistribution of the weighting of existing factors. In addition, we suggest that institutions should be given flexibility in implementation of this process, to facilitate empirical evaluations of decision-making processes and identification of processes that prove particularly effective in recruiting and retaining academics from diverse backgrounds.

Recent work demonstrates how decision-makers might implement holistic processes and, in doing so, may reduce key disparities. As with other post-undergraduate programs, medical residency programs frequently encounter racial and ethnic disparities in admissions, with a key bottleneck in the process being those invited to interview for residency placements 20 . To address this disparity in one medical residency admissions process, Barcelo and colleagues evaluated interview invite rates in a diverse pool of 547 applicants to a psychiatry residency program. The authors compared various approaches to evaluating applicants for an interview, including a traditional non-holistic model focused on such elements as applicants’ standardized test scores and involvement in honors societies. The holistic model focused on a diverse set of criteria along a series of domains that reflected not only scholastic aptitude but also community service, clinical experience, and leadership experience, along with considerations of applicants’ personal hardships or barriers they had overcome in the process of making their academic achievements. A third traditional modified model included elements from both the holistic and traditional models and was constructed by the authors to serve as an intermediary between these two models. The holistic model resulted in significant, large increases in invites to under-represented minority applicants (predicted probability of invite = 0.16) relative to the traditional model (predicted probability of invite = 0.08), with relatively little change in the predicted probability across models for non-under-represented applicants. In these analyses, factors such as the holistic model’s increased emphasis on lived experiences and de-emphasis on standardized test scores predicted differences between models in invite rates 21 .

Our goal in introducing this example is to emphasize that waiting for an evidence base to accrue before changes to academic evaluation practices can be implemented would simply result in a perpetuation of the systems currently in place. In fact, the residency study provides a window into how we might approach reconstructing our decision-making practices. Specifically, the holistic model the authors developed included emphasizing criteria that could traverse evaluation of applications across several disciplines and fields (e.g., leadership, lived experience, community involvement) as well as discipline-specific factors that the model emphasized (e.g., clinical experience) or de-emphasized (scores from a standardized medical exam). Further, the authors both tested the degree to which this model reduced disparities relative to a traditional model and examined factors that could account for any model differences they observed.

Now, imagine if hundreds or thousands of programs, across myriad fields and disciplines, developed their own processes for evaluating applicants for interview invitations, with criteria both common across programs and unique to local program needs. Assuming regular or annual application cycles and thus thousands of ‘data points’ per cycle, within a short period academia would inherit a rather large ‘database’ of programs, evaluative models, and outcomes, along with variables to test for prediction of outcomes. With this structural change in how academia approaches this single decision (interview invites for graduate training programs), the variability among evaluative models would allow for scholarly work focused not only on examining whether particular approaches to interview decisions reduce long-standing disparities, but also on predicting important factors going forward, such as disparities in program admissions and the likelihood of positive trainee outcomes, including graduation rates and job placements. Over time, these data would prove critical in identifying evidence-based predictors of trainee and, eventually, faculty performance, thus allowing for refinement in institutional ‘best practices’ in decision-making as they pertain to evaluations of graduate student and faculty applicants.

In raising the potential for flexibility in holistic models, it is important to nonetheless consider the value of standardization, at least within specific models. For instance, with regards to graduate admissions, it has been suggested that the use of structured and standardized materials (personal statements, interviews, letters of recommendation) may be one way to guard against the kinds of cognitive biases that might contribute to racial disparities 6 . We would emphasize that standardization of any element of an application package along these lines need not necessitate that all programs adopt the same set of criteria. That is, different admission committees may vary in the relative weights they assign to personal statements, interviews, and letters of recommendation. Yet, they should endeavor to uniformly assess these elements in a standard way for each applicant in a given admissions cycle.

Barriers to progress

While we argue here that evaluation practices in academia should adopt more truly holistic approaches, we acknowledge that implementing systematic changes to policy will be a time-consuming and challenging endeavor. Faculty leading such efforts will need to seek and obtain approvals from the relevant governance bodies, and arriving at consensus surrounding the adoption of alternate metrics for evaluation will not be an easy task. As such, leaders of academic institutions, including senior faculty members and administrators, must step up to drive these changes and demonstrate through deeds their commitment to dismantling systems that favor the already-privileged.

Members of search and admissions committees must sincerely believe in the value of holistic evaluations for any proposed changes to hold weight. One way to facilitate this is to promote academic leaders with demonstrated commitments to mentoring, community outreach, science communication, and contributions to diversity and inclusivity. Academic leaders who already espouse these qualities will be in a good position to instigate and advocate for change.

Conducting holistic evaluations will no doubt require that committee members put extra time and effort into their evaluations. For graduate admissions, this may require phone calls and meetings with non-academic references for applicants. For faculty advancement, this will require time spent reading the candidate’s publications and becoming familiar with the impact of the scholar’s research by going beyond traditional metrics to solicit input from community stakeholders. As academic evaluation practices currently stand, committee members may over-rely on metrics like GRE scores and h -indices, not because they believe they are particularly valid, but rather for the simple reason that it is less work to focus on these quantitative indices than other more qualitative measures. The extra time and effort that will necessarily accompany holistic evaluations may need to be more adequately and creatively compensated for the individuals who are asked to complete them. That is, holistic review as a service commitment may need to be allocated a specific, valued space in the day-to-day work lives of key decision-makers in academia (e.g., faculty and administrators).

Concluding remarks

Academia’s current decision-making practices linked to graduate admissions and to the recruitment and advancement of faculty perpetuate racial and ethnic disparities. We already have an evidence base indicating that our current practices created these disparities and may be making them worse. We require a new evidence base, one focused on revisions to our approaches and decision-making practices, that takes a holistic view of factors beyond traditional metrics. As disparities exist across disciplines and fields in academia, this should indicate to us that no one model of holistic evaluation will suffice for all decision-making contexts. Where some see challenges in this lack of standardization, we see opportunities. This very challenge may be the one element of structural change that becomes our saving grace.

This moment in history requires structural change focused on three principles. First, we should encourage flexibility in approaches to holistic review. That is, we should allow decision-makers the ability to create holistic models that include elements specific to particular disciplines or institutional programs, as well as elements that cut across disciplines and programs. This variability will facilitate short-term and long-term evaluations of new holistic review models. Second, after some period of time for data collection, we should create task forces or evaluative bodies focused on examining data from holistic review models and determining which factors in these models robustly predict key outcomes, including reductions in disparities at key decision-making moments (e.g., graduate admissions, faculty tenure and promotion). Third, as we develop holistic review models, we should also implement safeguards or explicit components into model development that focus on reviewing whether elements included in these models may introduce additional disparities or exacerbate existing disparities. For instance, our institutions might develop ‘disparity review boards’ for assessing admissions and hiring criteria, much like current institutional review boards (IRBs) for reviewing research protocols. Here, the emphasis would be on evaluating protocols for holistic review models, with a particular emphasis on whether the models include components for which research suggests potential disparities may arise. If a model includes such a component, the review board might make recommendations regarding monitoring this component for any disparities it creates, much like IRB protocols surrounding safety monitoring for clinical trials research.

We see academia writ large at a crucial inflection point. Much of society is seemingly cognizant of and knowledgeable about long-standing racial and ethnic disparities across its constituent institutional structures. We should assume that much of the world will pay close attention to all of these structures and, in particular, to the actions its leaders take to reduce these disparities. With its sizable influence on economic advancement and life satisfaction, academia will see its fair share of this attention on the global stage, and the time to take bold action is now.

Miller, C. W., Zwickl, B. M., Posselt, J. R., Silvestrini, R. T. & Hodapp, T. Sci. Adv. 5 , t7550 (2019).

Article   Google Scholar  

Langin, K. Science Careers https://doi.org/10.1126/science.caredit.aay2093 (29 May 2019).

Hu, J.C. Science Careers https://doi.org/10.1126/science.caredit.abd4989 (24 June 2020).

Larson, R. C., Ghaffarzadegan, N. & Xue, Y. Syst. Res. Behav. Sci. 31 , 745–750 (2014).

Kuncel, N. R., Hezlett, S. A. & Ones, D. S. Psychol. Bull. 127 , 162–181 (2001).

Woo, S. E., LeBreton, J., Keith, M. & Tay, L. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/w5d7r (2020).

Miller, A., Crede, M. & Sotola, L.K. Int. J. Sel. Assess . https://doi.org/10.1111/ijsa.12312 (2020).

Kim, Y. K. & Sax, L. J. Res. High. Educ. 50 , 437–459 (2009).

Roberts, S. O., Bareket-Shavit, C., Dollins, F. A., Goldie, P. D. & Mortenson, E. Perspect. Psychol. Sci. 15 , 1295–1309 (2020).

Clauset, A., Arbesman, S. & Larremore, D. B. Sci. Adv. 1 , e1400005 (2015).

Hoppe, T. A. et al. Sci. Adv. 5 , w7238 (2019).

Fernandes, J. D. et al. eLife 9 , e54097 (2020).

Wright, C. B. & Vanderford, N. L. Nat. Biotechnol. 35 , 885–887 (2017).

Article   CAS   Google Scholar  

Petersen, A. M. et al. Proc. Natl. Acad. Sci. USA 111 , 15316–15321 (2014).

Sensoy, Ö. & DiAngelo, R. ‘Harv. Educ. Rev. 87 , 557–580 (2017).

Google Scholar  

Nature 591 , 7 (2021).

Clark, U. S. & Hurd, Y. L. Nat. Hum. Behav. 4 , 774–777 (2020).

Dougherty, M. R. & Horne, Z. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/9g5wk (2019).

Pier, E. L. et al. Proc. Natl. Acad. Sci. USA 115 , 2952–2957 (2018).

Talamantes, E., Henderson, M. C., Fancher, T. L. & Mullan, F. N. Engl. J. Med. 380 , 803–805 (2019).

Barceló, N. E. et al. Acad. Psychiatry 45 , 34–42 (2020).

Download references

Acknowledgements

A.D.L.R. is supported by the Institute of Education Sciences, U.S. Department of Education, through grant no. R324A180032 to the University of Maryland at College Park. LQU is supported by the National Institutes of Health, U.S. Department of Health and Human Services, through grant no. R01MH107549, and by the Canadian Institute for Advanced Research (L’Institut Canadien de Recherches Avancées). The opinions expressed are those of the authors and do not represent views of these funding agencies.

Author information

Authors and affiliations.

Department of Psychology, University of Maryland, College Park, MD, USA

Andres De Los Reyes

Department of Psychology, University of Miami, Coral Gables, FL, USA

Lucina Q. Uddin

Neuroscience Program, University of Miami Miller School of Medicine, Miami, FL, USA

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Andres De Los Reyes or Lucina Q. Uddin .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Peer review information Nature Neuroscience thanks Melanie Woodin and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

De Los Reyes, A., Uddin, L.Q. Revising evaluation metrics for graduate admissions and faculty advancement to dismantle privilege. Nat Neurosci 24 , 755–758 (2021). https://doi.org/10.1038/s41593-021-00836-2

Download citation

Published : 30 March 2021

Issue Date : June 2021

DOI : https://doi.org/10.1038/s41593-021-00836-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Confronting racially exclusionary practices in the acquisition and analyses of neuroimaging data.

• J. A. Ricard
• T. C. Parker
• A. J. Holmes

Nature Neuroscience (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Thursday 14 May 2009

Phd assessment criteria.

A PhD must show the following … 1. A distinct contribution to knowledge

2. Evidence of the discovery of new facts or the exercise of independent judgement

3. The author’s ability to present well written and suitably documented research

4. That the original work presented in the thesis merits publication, if publication has not been achieved

5. That the author has become competent in independent work or research, and that s/he could repeat the process in a fresh project

6. An understanding of appropriate techniques

7. Critical appraisal and use of related work in the field, from published work and source material

4 comments:

Hello Robert, my first experience when I began my PhD as a mature student with many years of commercial science experience was reading a thesis by a mature African student (a senior civil servant in his home country) which was absolutely atrocious. In my view it actually proved beyond reasonable doubt that he was incapable of abstract conceptual thought. This experience devalued the PhD qualification from the start for me. Eventually I have come to the conclusion that given the financial and social pressures on examiners and lack of repurcussions it is inevitable that standards will slip under the current system. The only way I can see of solving this problem is by introducing the principles of blind assessment which are already accepted in other forms of examination. The viva seems to belong to a bygone era of gentlemanly agreements and supposed fair play which under modern pressures simply isn't working.

The issue of standards is an emotive one. I can only comment on the cases that I have experience of ... which occured in reputable and ranked universities in the UK. Whilst not wishing to glorify "failure" I have been involved in examinations where the PhD was not awarded and a lower qualification of a Masters by Research was awarded subject to revisions being made. These decisions were not taken lightly but fudging the issue or just saying yes because its the path of least resistance never came up in the examiners' conversation. I have also been sent a thesis to examine which I returned with the suggestion that the candidate should be encouraged to make significant adjustments before submitting for assessment ... a form of "pre-viva, think again." Blind review is commonplace in journal publication but the viva format is helpful when looking to explore issues, improve the work and assess whether the candidate actually knows the material. With good examiners it isn't a formality and my own take on the issue is that the decision reached (i.e. pass / fail) reflects on both the individuals concerned (i.e. the examiners and the candidate) and the awarding institution for all time since it is possible for someone to pick it up and pass their own judgement many years later in much the same way that you have done.

Dear Robert In this post, you mention that PhD candidates have to exhibit familiarity with "What we already know". My Q is whether this level of knowledge has to be cleared before joining the course or at some time during the course.

Vedant ... I think you need to make a first attempt at this in developing a research proposal which is usually the first step in being accepted onto a PhD programme ... however, the final version needs to flow from the 3 or so years that you spend doing the research. Hence, the answer is both that you estimate current knowledge and a gap at the outset but that you refine this to a final view in the preparation of your eventual thesis. I hope that this clarifies what I meant. Good luck with your studies Robert