Role of Reinforcement of learning across the continuum of medical education: A Scoping Review

Ayesha Younas¹, Faryal Azhar², Uzma Urooj³

1. Department of Medical Education, Wah Medical College, Wah Cantt-Pakistan.

2. Department of Surgery, Rawalpindi Medical College, Rawalpindi-Pakistan.

3. Department of Gynaecology, Army Medical College, Rawalpindi-Pakistan.

Correspondence to: Dr. Ayesha Younas, Email: ayeshajawwad@gmail.com, ORCiD: 0000-0002-1508-1395

https://doi.org/10.36570/jduhs.2019.3.695

ABSTRACT:

Objectives: The purpose of this study is to perform a scoping review of the literature to evaluate evidence showing the role of reinforcement strategies on student learning in medical education

Methods: scoping review design represents a methodology that allows assessment of emerging evidence, as well as a first step in research development. This scoping review includes research papers describing any teaching learning strategy or educational intervention that used Reinforcement of learning at any time during the learning process. Online databases were used to identify articles in the years 2009-2019, from which 10 publications from Canada and the United States and 6 from other nations were selected. Data collected from the sources was charted with the help of a self-developed form, inclusive of Names of Authors & year of publication, type of article, country of origin, sample size, objectives & key findings of study

Results: This review shows that reinforcement strategies still have a high impact on student learning. Reinforcing the taught material in medical education enhances student learning and retention and a strengthening of knowledge, skills and attitudes contributes to enhanced learning throughout the continuum of medical education & plays a vital role in developing internally motivated lifelong learners.

Conclusion: Evidence from literature suggests that there is extensive application of reinforcement strategies for enhancing learning in the field of medical education. Reinforcement of taught material has a positive impact on student learning & is an effective approach to medical education.

Keywords: Reinforcement; learning; reinforcement of learning; education; modern education; higher education.

INTRODUCTION:

The term Reinforcement was developed by American psychologist Burrhus Frederic Skinner. Skinner is regarded as the father of Operant Conditioning and was  enlisted by American Psychological Association enlisted Skinner as the most eminent psychologist of the 20th century.¹ In 1938, Skinner furthered the work of a prominent psychologist Edward Thorndike (1905) who studied learning in animals using a puzzle box to propose the theory known as the `Law of Effect` and proposed his theory of Operant conditioning.² Skinner introduced a new term into the Law of Effect - Reinforcement. He postulated that Behavior which is reinforced tends to be repeated (i.e. strengthened); behavior which is not reinforced tends to die out-or be extinguished (i.e. weakened).³ This was the basis of the reinforcement theory of motivation which overlooked the internal state of individual, i.e., the inner feelings and drives of individuals, but rather focuses on the effect of external stimuli on behavior.⁴ The main theme behind his theory was that learning occurs through reward and punishment of behavior.  Skinner postulated that Reinforcement is the primary process that shapes and controls behavior and it occurs in two ways, positive and negative.^5,6 Positive reinforcement is the process that enhances the likelihood of the response, by adding something .It can be given in the form of effective feedback by peers and encouraging participation and teacher.⁷ On the contrary, negative reinforcement is one that intensifies the probability of response, by removing or reducing something.⁸ Both types of reinforcement strengthen behavior, or increase the probability of a behavior reoccurring. Skinner`s work was published almost a century ago when behaviorist theories were in vogue.⁹ But with time, the behaviorist paradigm and most theories falling under it have been negated by educational psychologists.¹⁰ Literature however does provide evidence that some concepts derived from these theories have withstood the test of time. Among these is the term Reinforcement, the applications of which, form the basis of this scoping review.

The rationale behind conducting this scoping review is actually to identify literature to show whether reinforcement (i.e. both types of reinforcement; positive & negative) of learned concepts in any domain of learning (i.e. cognitive, psychomotor or affective)¹¹ is still considered to have an impact on student learning. Reinforcement procedures have become commonplace in educational settings as a way to encourage students to read and do their assignments. However, some have been concerned that reinforcement may actually undermine a student`s intrinsic interest and willingness to perform a task once the reinforcement procedure is removed. Similar concerns have been expressed about possible detrimental effects of reinforcement on creativity or originality. Extensive research on these questions has produced inconsistent results.¹² Thus, a scoping review of the literature of past 10 years was conducted to determine that what literary evidence is available to show that reinforcement strategies have a positive impact on student learning in medical education.

Methods

Protocol:

For this scoping review, the PRISMA Extension for Scoping Reviews (PRISMA-ScR) approach has been followed. This reporting guideline is consistent with the JBI guidance for scoping reviews, which highlights the importance of methodological rigor in the conduct of scoping reviews.¹³ A scoping review is conducted in order to systematically map the research done in any area, as well as to identify any existing gaps in knowledge.¹⁴ The PRISMA approach follows a number of clearly defined steps including beginning with a clearly formulated question, using the question to develop clear inclusion criteria to identify relevant studies, an approach to appraise the studies or a subset of the studies, a summary of the evidence using an explicit methodology and interpreting the findings of the review.^15,16

Eligibility Criteria

Search was conducted for different types of research articles. Codes were developed for each article type. Code A is given for original article, code B for Reports, code C for data papers, code D for editorials and code E for review. The details of coding are described in Table 1.  

Table 1: Types of Articles included in review
Code	Type of publication	Description
A	Original Article	A published research presenting any original research related to reinforcement learning.
B	Reports	A publication describing the process, progress, or results of research on reinforcement learning
C	Data papers	Any publication providing facts about the data collected relevant to research on reinforcement learning in education
D	Editorials	Any discussions, commentary and opinion pieces related to reinforcement learning in education
E	Review	Any study providing survey of previously published literature on reinforcement learning

To be included in the review, papers needed to describe any teaching learning strategy or educational intervention that used Reinforcement of learning at any time during the learning process. Peer-reviewed journal papers were included if they were: published between the period of 2009-2019, abstracts written in English, involved human participants and described how the process of reinforcement had either enhanced or decreased the learning process. It is important to mention that papers describing learning strategies for higher education (high school onwards; Respondents age: 13 or above) were also considered, so as to widen the scope of the search. However, the main focus was kept on medical education literature. Quantitative, qualitative and mixed-method studies were included in order to consider different aspects of reinforcement of learning. Papers were excluded if they did not fit into the conceptual framework of the study.

Information Sources:

To start the review, after formulating the research question, it was necessary to determine the inclusion criteria a-priori. Initially, a preliminary search of two bibliographic data bases was done (PubMed & ERIC). However, to further augment results, two other data bases [Psych Info and Google scholar (for grey literature)] were also included. The target was to retrieve around 250 - 300 articles from these data bases. Online literature search was conducted from the 29^th of January and concluded on the 26^th of February 2019. The entire study was then completed in the preceding month.

The criterion for inclusion on further examination was as follows: a research article that either reported on the use of reinforcement learning in any educational environment or any article that provided information about the development of any educational activity employing reinforcement learning

Search Strategy:

Flowchart for search strategy for Pubmed.gov (https://www.ncbi.nlm.nih.gov/pubmed/) is attached in flowchart.

Selection of sources of Evidence:

After applying search limitations (i.e. years &article types) On reviewing the data bases 357 relevant articles from PubMed, 5 relevant articles from Psycho Info, 29 articles from ERIC and 12 in grey literature from google scholar were identified. Abstracts of these 425 articles were studied for further examination. After thorough review, 65 articles from all data bases were further inspected. These were then downloaded and scrutinized thoroughly. On final selection, 16 articles which were included in the scoping review. Out of these articles 14 articles were original researches of quantitative and qualitative types (Article type code A), one was a data paper (Article Type code C) and one was a review (Article Type code E). Article selection flowchart by reviewer criteria is attached. (Figure 2)

Data Charting process:

Data was extracted from included sources of evidence according to a data charting form that was developed to include all required sources of evidence. Data charting form for all the studies included in this review is attached. (Table 2) Table 2: Data charting form for selected articles
S. No	Title of Article /Journal	Article Type	Country of research	sample size	Aims /  Objectives	Key findings related to research question
1	Learning to Become an Expert: Reinforcement Learning and the Acquisition of Perceptual Expertise Journal of Cognitive Neuroscience^17	A	Canada	18	To provide novel ERP evidence demonstrating that actions rapidly acquire value with learning specially in relation to reward and punishments	The computations that underlie human learning and decision-making follow reinforcement learning principles.
2	Observation, Reflection, and Reinforcement: Surgery Faculty Members and Residents Perceptions of How They Learned Professionalism/ Academic Medicine^18	A	Canada	34	To explore perceptions of how professionalism is learned in the current academic environment	Professionalism can be learnt by an active approach through a number of ways including feedback for reinforcement.
3	What Can Medical Education Learn from the Neurobiology of Learning? Academic Medicine^19	B	USA	---	How  understanding the biology of learning can help to improve educational strategies and curricular design in medical education	10 key aspects of learning are identified that can be incorporated into effective teaching paradigms in multiple ways. One of these is reinforcement of learned behavior.
4	Practical skills teaching in contemporary surgical education: how can educational theory be applied to promote effective learning? The American Journal of Surgery^20	E	UK	-----	Exploration of the role of educational theory in promoting effective practical skills teaching	Several aspects of surgical education can be modeled on educational theories including development of expertise after repeated practice and regular reinforcement
5	The effectiveness of using reinforcements in the classroom on the academic achievement of students with intellectual disabilities Journal of Intellectual Disabilities^21	A	Iran	45	To compare the effectiveness of two kinds of reinforcements, on the academic achievement of eighth-grade female students	Both types of reinforcements  play a role in improving student scores
6	The role of repetition and reinforcement in school-based oral health education-a cluster randomized controlled trial. BMC Public Health^22	A	Pakistan	935	To determine the effectiveness of the repeated and reinforced OHE (RR-OHE) compared to one-time OHE intervention and to assess its role in school-based OHE imparted by dentist, teachers and peers	Repetition and reinforcement play a key role in school-based OHE irrespective of educators. The trained teachers and peers can play a complementary role in RR-OHE
7	Is Teaching Simple Surgical Skills Using an Operant Learning Program More Effective Than Teaching by Demonstration? Clinical Orthopedics and Related Research ^23	A	USA	11	To determine whether a group that is taught a surgical skill using an operant learning procedure would more precisely perform that skill than a group that is taught by demonstration alone.	The operant learning (which received reinforcement through using the acoustic stimulus from a mechanical clicker)  group achieved better precision tying the locking, sliding knot than did the control group.
8	Engaging medical undergraduates in question making: a novel way to reinforcing learning in physiology. Advances in Physiology Education^24	A	India	99	To determine whether student learning in Physiology can be reinforced by asking the students to create questions for their opposing teams	Reinforcement of taught topics through the use of a group activity resulted in an increase in student perceptions of their knowledge on the topic as well as communicative and analytical skills.
9	Effects of Reinforcement Method of Dissection Physiology Education on the Achievement in Pharmacology. YAKUGAKU ZASSH^25	A	Japan	15	To study the effects of the reinforcement method of physiology education on achievement in pharmacology	The reinforcement method for education in basic subjects in pharmacy, such as physiology, can improve achievement in more advanced subjects, such as pharmacology.
10	Differential effects of reinforcement on the self-monitoring of on-task behavior. School Psychology Quarterly^26	A	USA	3	The differential effects of reinforcement on a self-monitoring intervention were evaluated.	Self-monitoring alone was effective for 2 students in increasing their on-task behaviors in a general education classroom and self-monitoring with reinforcement was effective for all 3 students.
11	Learning Anatomical Structures: A Reinforcement-Based Learning Approach Medical Science Educator^27	A	Canada	10	To design an anatomical structure identification reinforcement learning task for participants with minimal prior neuroanatomical knowledge	Results show that there is a key role of reinforcement learning approaches to establishing foundational knowledge in the pre-clinical sciences, specifically anatomy, in a time-efficient manner.
12	Enhancing student retention of prerequisite knowledge through pre-class activities and in-class reinforcement Biochemistry and Molecular Biology Education^28	A	USA	79	Does reinforcement of concepts of general and organic chemistry enhance learning of college level biochemistry	This study demonstrates that developing student understanding requires both explicit pre-class review of the information and in-class reinforcement of these prerequisite concepts.
13	Using an online quiz-based reinforcement system to teach healthcare quality and patient safety and care transitions at the University of California International Journal for Quality in Health Care^29	A	USA	500	An online quiz-based reinforcement system to increase resident and faculty knowledge in QI, patient safety and care transitions.	A multi-campus online quiz-based reinforcement system to train residents in patient safety and care transitions was feasible, acceptable, and increased knowledge.
14	Optimizing quality of dental carving by preclinical dental students through anatomy theory reinforcement. Anatomical Sciences Education^30	A	Brazil	330	To evaluate the effect of didactic reinforcement on dental carvings	Theoretical reinforcement of dental anatomy seems to improve the students` carving performance but does not enhance their knowledge about dental anatomy
15	A Reinforcement-Based Learning Paradigm Increases Anatomical Learning and Retention - A Neuroeducation Study Frontiers in Human Neuroscience^31	A	Canada	23	Is there any use of reinforcement learning within the context of pre-class exercises to build foundational anatomic knowledge.	Employing the reinforcement learning paradigm is an effective educational approach for developing anatomical expertise
16	The application of reward learning in the real world: Changes in the reward positivity amplitude reflect learning in a medical education context. International Journal of Psychophysiology ^32	A	Canada	30	To provide support that the reward positivity or reinforcement is reflective of an underlying learning process in a real-world medical education context.	Data shows that reinforcement is an index of a neural learning system, and further validate that this same system is involved in learning across a wide range of contexts.

Data Items:

Data was extracted according to an item list which included the following variables

1. Names of Authors & year of publication (Referenced according to Vancouver referencing style)
2. Article Code (coding done according to type code developed in Table 1)
3. Country where research was conducted
4. Sample size
5. Objectives of study
6. Key findings related to reinforcement

Synthesis of the Results:

The results were synthesized by grouping the studies into various article types according to the codes developed for article types. Out of the 16 studies included in the review, 14 were original research articles (Type A),1 was a report (Type B) and 1 was a Review (Type E). All articles were read thoroughly and their relevance to research question was established. Then the key findings were summarized first in draft form and finally in the form of in the form of Table 2. All studies have been enlisted in a chronological order and their key findings in relation to the research question for this scoping review have been mentioned.

Results

Selection of sources of evidence

Out of the 16 studies that are included in final review, there are 5 each from the USA & Canada, and 1 each from the UK, Iran, Pakistan, India, Japan & Brazil. From 2010 to 2014, only one study for each year was found; however, results showed 2 studies in 2015, 3 in 2016, 2 again in 2017 &3 in 2018. Figure 1 describes the country of origin and year of publication of the articles included in this scoping review.

Figure 1: Country of Origin & Years of Publication of studies included in the scoping review

Characteristics of the sources of evidence:

As mentioned previously, almost all the studies use the reinforcement of learning content as a theme in their research. All of these studies prove that reinforcement is still very much an important concept and varying techniques of reinforcement are being applied in medical education today. Only two of these studies were conducted on high school children and rest were all conducted on students attending medical schools. Six studies were conducted on basic science subjects mainly Anatomy, Physiology, Biochemistry and Dental Anatomy. One study each on preventive dentistry and pharmacology and 3 on students undergoing clinical clerkships in surgery. A study showing reinforcing techniques in professionalism has also been included in the review.

Thus, we can see that reinforcement is a concept that can be applied across the entire continuum of undergraduate medical education and provides promising results at all stages.

Results of Individual sources:

One of the most cited of these articles is from Michael Friedlander¹⁹ which shows how modern studies of neuronal changes and the biology of learning can influence medical education. The authors have enlisted 10 key aspects of learning that can be incorporated into effective teaching paradigms in multiple ways. The 2^nd enlisted aspect is reinforcement and the authors state that the brain`s intrinsic reward system plays a major role in reinforcement of learned behaviors.  A literature review has also been included in the list of included articles. This literature review emphasizes that several aspects of surgical education can be modeled on educational theories including development of expertise after repeated practice and regular reinforcement.²⁰ All other studies included in this scoping review are original researches. Out of these, only two have been carried out on high school students and the rest all on Medical students. One aspect of studies on reinforcement of learning has been the examination of neurological processes involved with reinforcement. Krigolson & Pierce³³ conducted a study to provide Event Related Potential (ERP) evidence demonstrating that actions rapidly acquire value with learning and the computations that underlie human learning and decision-making follow reinforcement learning principles. In the teaching of Dental public health, Haleem & Khan MK²² showed that repetition and reinforcement play a key role in school-based oral Health examinations. A number of different studies including those by Mehta & Bhandari²⁴, Anderson & Hecker³¹, Taylor & Olofson²⁸ Kitayama & Kogota²⁶ proved that the reinforcement method for education in basic subjects can improve learning not only in the same subject but also creates schemas for future retrieval.

Similarly Levy & Pryor²³ showed that surgical residents when received reinforcement through an acoustic stimulus achieved better precision during knot tying for surgical sutures.

DISCUSSION

Reinforcement was divided by Skinner into further types (positive and negative reinforcement)¹² However, for the purpose of this review; Reinforcement of learning is being studied as a whole phenomenon; in its entirety. All the studies that have been found during this search have shown that not only is there a key role of reinforcement learning approaches in establishing foundational knowledge in the basic and pre-clinical sciences, but clinical sciences like surgery also show that employing the reinforcement learning paradigm is an effective educational approach. Studies of neurobiology have also shown that that reinforcement is an index of a neural learning system, and further validate that this same system is involved in learning across a wide range of contexts. The results of this scoping review are similar to a recent study by Matos & colleagues³³ on Spaced education. Spaced education is a novel method that improves medical education through online repetition of core principles often paired with multiple-choice questions.  In this study authors have proved that reinforcement of learning material statistically improves medical retention among interns.  Ann Taylor & colleagues³⁵ used reinforcement of previous learnt material as a learning strategy for Students in a biochemistry course. They reported students saying It`s always good to take a course where much of the curriculum is based on applying previous knowledge. Another study<sup>29</sup> conducted in the university of California showed that Quiz-based reinforcement systems to teach healthcare quality and patient safety and care transitions show promise in fostering active engagement, collaboration & healthy competition among students. Rajiv Mahajan and Colleagues<sup>36</sup> linked lifelong learning with adopting reinforcing teaching-learning methods. They discussed that Medical undergraduates already had prior learning experiences and acknowledging their understanding and experiences, reinforcing them and using appropriate instructional strategy accordingly, helped in cultivating lifelong learning. Positive reinforcement can be used effectively by teachers to improve and enhance student motivation levels<sup>37</sup> and develop internally motivated learners. The effect of reinforcement of learning was also explored in a study<sup>38</sup> on the use of simulation in medical education and authors showed that active participation of trainees could be achieved through the use of open-ended questions and positive reinforcement. Another pertinent argument in favor of reinforcement of learning is that of the spiral curriculum,<sup>39,40</sup> which is in trend in undergraduate medical colleges across the globe. In the spiral curriculum, once learned a topic or subject is continually reinforced and integrated with other concepts at a more complex level. A search of the literature on studies from Pakistan, showed the use of positive reinforcement during micro feedback sessions.<sup>41</sup> Thus, we can see that reinforcement still holds much standing as a learning strategy in medical education across the globe. Its applications are visible not only in a small courses or entire curricula, but also in enhancing learner`s intrinsic motivation for deep and lifelong learning. Reinforcement learning, thus is not a concept that has become redundant with time, rather it is a strategy, that still proves its worth for medical studies in the 21^st century.

LIMITATIONS

This study was conducted initially by a single researcher, and the aim was initially to identify the role of reinforcement in education. However, it was soon realized that the initial aim was too broad and focus was then kept the on use of reinforcement learning strategies in higher education and above only. Literature on the use of reinforcement strategies for improving learning is widely available for both primary and secondary education, however, since these studies were not a part of the inclusion criteria, they were not included.

CONCLUSION

This scoping review provides evidence that certain aspects of the behaviorist paradigm, like the use of reinforcement to enhance student learning have withstood the test of time. Positive reinforcement can provide great benefit to the student & strategies which reinforce learned concepts can help students at all levels of the education spectrum. Further work to identify the most effective reinforcement techniques for undergraduate medical education can be done to fully exploit the potential of reinforcement learning.

FUNDING

No funding by any source was provided to the author for this study

REFERENCES

1) Haggbloom SJ, Warnick R, Warnick JE, Jones VK, Yarbrough GL, Russell TM, et al. The 100 most eminent psychologists of the 20th century. Rev Gen Psychol 2002; 6:139-52.

2) Pavlov I. Burrhus Skinner and Behaviourists. Learn Theor Early Years Pract 2018; 48.

3) McLeod S. Skinner-operant conditioning [Internet]. [Updated 2018 Jan 21; cited 2018 Dec 10] Available from: https://www.simplypsychology.org/operant-conditioning.html

4) Mills JA. A summary and criticism of Skinner`s early theory of learning. Can Psychol Rev Can 1978; 19:215-23.

5) Skinner BF. Reinforcement today. Am Psychol 1958; 13:94.

6) Gewirtz JL, Pelaez-Nogueras M. B. F. Skinner`s legacy to human infant behavior and development. Am Psychol 1992; 47:1411-22.

7) Prakash R, Sharma N, Advani U. Learning process and how adults learn. Int J Acad Med 2019; 5:75.

8) Nevin JA, Mandell C. Comparing positive and negative reinforcement: A fantasy experiment. J Exp Anal Behav 2017;107(1):34-38.

9) Watson JB. Behaviorism. Routledge; 2017.

10) Ekstrand MD, Willemsen MC. Behaviorism is not enough: better recommendations through listening to users. In: Proceedings of the 10th ACM Conference on Recommender Systems. ACM; 2016. p. 221-4.

11) Sonmez V. Association of Cognitive, Affective, Psychomotor and Intuitive Domains in Education, Sonmez Model. Univers J Educ Res 2017; 5:347-56.

12) Domjan M. The Principles of Learning and Behavior [Internet]. Canada: Wadsworth, Cengage Learning; 2010 [cited 2019 Dec 10]. Available from: https:/ /pdfs. Semantic scholar.org/b0db/76855dcca9ae1bc82830e8171140ff118535.pdf

13) Tricco AC, Lillie E, Zarin W, O`Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 2018; 169:467.

14) Peterson J, Pearce PF, Ferguson LA, Langford CA. Understanding scoping reviews: Definition, purpose, and process. J Am Assoc Nurse Pract 2017; 29:12-6.

15) Davis K, Drey N, Gould D. What are scoping studies? A review of the nursing literature. Int J Nurs Stud 2009; 46:1386-400.

16) Brandt B, Lutfiyya MN, King JA, Chioreso C. A scoping review of interprofessional collaborative practice and education using the lens of the Triple Aim. J Interprof Care 2014;28:393-9.

17) Krigolson OE, Hassall CD, Handy TC. How We Learn to Make Decisions: Rapid Propagation of Reinforcement Learning Prediction Errors in Humans. J Cogn Neurosci 2014; 26:635-44.

18) Park J, Woodrow SI, Reznick RK, Beales J, MacRae HM. Observation, Reflection, and Reinforcement: Surgery Faculty Membersʼ and Residentsʼ Perceptions of How They Learned Professionalism. Acad Med 2010; 85:134-9.

19) Friedlander MJ, Andrews L, Armstrong EG, Aschenbrenner C, Kass JS, Ogden P, et al. What Can Medical Education Learn From the Neurobiology of Learning? Acad Med 2011; 86:415-20.

20) Sadideen H, Kneebone R. Practical skills teaching in contemporary surgical education: how can educational theory be applied to promote effective learning? Am J Surg [Internet]. 2012; 204:396-401.

21) Adibsereshki N, Abkenar SJ, Ashoori M, Mirzamani M. The effectiveness of using reinforcements in the classroom on the academic achievement of students with intellectual disabilities. J Intellect Disabil 2015; 19:83-93.

22) Haleem A, Khan MK, Sufia S, Chaudhry S, Siddiqui MI, Khan AA. The role of repetition and reinforcement in school-based oral health education-a cluster randomized controlled trial. BMC Public Health 2015; 16:2.

23) Levy IM, Pryor KW, McKeon TR. Is Teaching Simple Surgical Skills Using an Operant Learning Program More Effective Than Teaching by Demonstration? Clin Orthop Relat Res 2016; 474:945-55.

24) Mehta B, Bhandari B. Engaging medical undergraduates in question making: a novel way to reinforcing learning in physiology. Adv Physiol Educ 2016; 40:398-401.

25) Kitayama T, Kagota S, Yoshikawa N, Kawai N, Nishimura K, Miura T, et al. Effects of Reinforcement Method of Dissection Physiology Education on the Achievement in Pharmacology. YAKUGAKU ZASSHI 2016; 136:1651-6.

26) Otero TL, Haut JM. Differential effects of reinforcement on the self-monitoring of on-task behavior. Sch Psychol Q 2016; 31:91-103.

27) Anderson SJ, Krigolson OE, Jamniczky HA, Hecker KG. Learning Anatomical Structures: a Reinforcement-Based Learning Approach. Med Sci Educ 2016; 26:123-8.

28) Taylor ATS, Olofson EL, Novak WRP. Enhancing student retention of prerequisite knowledge through pre-class activities and in-class reinforcement. Biochem Mol Biol Educ 2017;45:97-104.

29) Shaikh U, Afsar-manesh N, Amin AN, Clay B, Ranji SR. Using an online quiz-based reinforcement system to teach healthcare quality and patient safety and care transitions at the University of California. Int J Qual Heal Care 2017; 29:735-9.

30) de Azevedo RA, Correa MB, Torriani MA, Lund RG. Optimizing quality of dental carving by preclinical dental students through anatomy theory reinforcement. Anat Sci Educ 2018; 11:377-84.

31) Anderson SJ, Hecker KG, Krigolson OE, Jamniczky HA. A Reinforcement-Based Learning Paradigm Increases Anatomical Learning and Retention-A Neuroeducation Study. Front Hum Neurosci 2018;12.

32) Krigolson OE, Pierce LJ, Holroyd CB, Tanaka JW. Learning to Become an Expert: Reinforcement Learning and the Acquisition of Perceptual Expertise. J Cogn Neurosci 2009; 21:1833-40.

33) Matos J, Petri CR, Mukamal KJ, Vanka A. Spaced education in medical residents: An electronic intervention to improve competency and retention of medical knowledge. Vrana KE, editor. PLoS One 2017 ;12:e0181418.

34) Williams CC, Hecker KG, Paget MK, Coderre SP, Burak KW, Wright B, et al. The application of reward learning in the real world: Changes in the reward positivity amplitude reflect learning in a medical education context. Int J Psychophysiol 2018; 132:236-42.

35) Taylor ATS, Olofson EL, Novak WRP. Enhancing student retention of prerequisite knowledge through pre-class activities and in-class reinforcement. Biochem Mol Biol Educ 2017; 45:97-104.

36) Mahajan R, Badyal DK, Gupta P, Singh T. Cultivating lifelong learning skills during graduate medical training. Indian Pediatr 2016; 53:797-804.

37) Abdel Meguid EM, Khalil MK. Measuring medical students` motivation to learning anatomy by cadaveric dissection. Anat Sci Educ 2017; 10:363-71.

38) Jones F, Passos-Neto CE, Braghiroli OFM. Simulation in Medical Education: Brief history and methodology. Princ Pract Clin Res 2015; 1:56-63.

39) Harden RM. What is a spiral curriculum? Med Teach 1999; 21:141-3.

40) Howe A, Campion P, Searle J, Smith H. New perspectives-approaches to medical education at four new UK medical schools. BMJ 2004; 329:327-31.

41) Baseer N, Mahboob U, Degnan J. Micro-Feedback Training:Learning the art of effective feedback. Pakistan J Med Sci 2017; 33:1525-7.