RESEARCH ARTICLE


Validation of the Indonesian Version of the Face, Legs, Activity, Cry, Consolability (FLACC) Scale in Postoperative Cleft Lip and/or Cleft Palate Patients



Salshabia Gabrielle Raissa Tasman1, Farah Asnely Putri2, Tantry Maulina2, *
1 Faculty of Dentistry, Universitas Padjadjaran, Bandung, Indonesia
2 Oral and Maxillofacial Surgery Department, Faculty of Dentistry, Universitas Padjadjaran, Bandung, Indonesia


Article Metrics

CrossRef Citations:
0
Total Statistics:

Full-Text HTML Views: 447
Abstract HTML Views: 368
PDF Downloads: 131
ePub Downloads: 71
Total Views/Downloads: 1017
Unique Statistics:

Full-Text HTML Views: 261
Abstract HTML Views: 231
PDF Downloads: 91
ePub Downloads: 44
Total Views/Downloads: 627



Creative Commons License
© 2022 Tasman et al.

open-access license: This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: https://creativecommons.org/licenses/by/4.0/legalcode. This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

* Address correspondence to this author at the Oral and Maxillofacial Surgery Department, Faculty of Dentistry, Universitas Padjadjaran, Bandung, Indonesia; Tel: +31 6 34454998; E-mail: tantry.maulina@fkg.unpad.ac.id


Abstract

Background:

The face, leg, activity, cry, and consolability (FLACC) scale is a validated pain measurement instrument that is used on postoperative patients with limited verbal ability, including postoperative cleft lip and/or cleft palate patients.

Objective:

This research aimed to test the validity and reliability of the Indonesian version of the FLACC scale as a measuring instrument for pain intensity experienced by postoperative cleft lip and/or cleft palate patients.

Methods:

The procedure was initiated by a back-translation process of the FLACC. Once the back-translation process is completed, a calibration process of the field researchers was conducted. Twenty-eight participants that went through a cleft lip and/or cleft palate surgery at the Unpad Dental Hospital were then enrolled. Two calibrated field researchers measured the postoperative pain intensity in three different time points, shortly after the patients regained full consciousness (T0), four hours (T1), and eight hours after the first measurement (T2). The collected data were analysed by SPSS version 23. The Spearman correlation analysis was performed to test the validity, while a Cronbach’s alpha value was calculated to test the reliability.

Results:

Based on the results of the Spearman correlation analysis, the Indonesian version of the FLACC scale was considered to be valid as the r values of each sub-scale were all higher than the r table value (r value > 0.317). Reliability was marked by the obtained Cronbach’s alpha value of 0,875.

Conclusion:

The Indonesian version of the FLACC scale was considered to be valid and reliable to be used as a pain measurement tool in postoperative cleft lip and/or cleft palate patients.

Keywords: Cleft lip, Cleft palate, Postoperative pain, Orofacial pain, FLACC scale, Validity, Reliability.



1. INTRODUCTION

Cleft lip and cleft palate are the largest groups of craniofacial malformations in humans, with a prevalence of 1 in 1000 births. These malformations occur because of a disruption in the development process on the orofacial part in the early weeks of pregnancy [1, 2]. Cleft lip and palate disorders are known to be more common in males than in females with a ratio of 3:2, with Asian and Native American (Indian) populations showing the highest prevalence [3, 4]. Considering the long term impacts of these malformations, it is crucial that the management of cleft lip or cleft palate is performed in the early age of its sufferer. Previous studies revealed that cleft lip or cleft palate has the potential to cause nutritional intake disorder, speech delays, hearing loss, and psychological impacts such as the loss of self-confidence. Not to mention the psychological impact experienced by the patient’s parents [5-7].

The management of cleft lip and/or cleft palate consists of several stages, including an interventional surgery that is ought to be performed in the early stages of the patient’s life [8, 9]. To determine the timing of the cleft lip and palate surgical intervention, the ‘rule of ten’ is usually applied, it includes parameters at 10, such as; age more than 10 weeks old, 10 pounds of body weight, haemoglobin at least 10 g/dL, and white blood cell count 10.000 mm3 [10]. Surgery, as known, has several postoperative consequences, including postoperative pain [11], which in the case of postoperative cleft lip and cleft palate patients, require special attention, considering that patients are unable to express the pain that they experience verbally and appropriately [12]. The clinician's failure to assess pain level may result in inadequate treatment and medication [13]. Therefore, in order to have a valid pain measurement on postoperative cleft lip and/or cleft palate patients, the usage of a validated and reliable pain measurement tool that is based on behavioural observation instead of the ones that is based on verbal communication is considered to be more appropriate [14].

The Face, Legs, Activity, Cry, Consolability (FLACC) behavioural pain scale was developed to provide a simple and consistent method for doctors and nurses to identify, record, and evaluate pain in patients who are unable to communicate their pain state or pain level verbally, due to their medical condition, or cognitive impairment. The scale evaluates pain severity with a maximum score of 10 [15-17]. The validity and reliability of the FLACC scale have been proven in previous studies [18-20]. Yet, according to our literature study, no previous study has utilized the scale to evaluate postoperative pain experienced by cleft lip and/or cleft palate patients. Additionally, no validation of the Indonesian version of the scale has been recorded. Given the high prevalence of cleft lip and palate surgery in Indonesia, having a valid and reliable behavioural pain measuring tool is considered important for improving pain management.The aim of the research was, therefore, to test the validity and reliability of the Indonesian version of the FLACC scale for postoperative cleft lip and/or cleft palate patients.

2. MATERIALS AND METHODS

The current study was conducted at Unpad Dental Hospital in Bandung, Indonesia, from January to March 2021. Twenty-eight (16 male; 12 female) patients (mean age: 11 months) who were about to undergo cleft lip surgery and fulfilled the inclusion criteria were recruited. Prior to the research, ethical approval from the Health Research Ethics Committee, Faculty of Medicine, Universitas Padjadjaran (No: 715/UN6. KEP/EC/2020) was obtained. All research procedures were carried out following the Declaration of Helsinki and all research participants’ parents gave their consent to participate in this research and for their data to be used in future scientific publication as the result of the current study.

2.1. Sample Size Determination and Selection

As there is no standardized samples size formula for validation study, the number of samples was calculated using the 1: 5 comparison scale, whereas for each question category in the questionnaire/ scale, the data from five participants were used [21, 22]. Therefore, considering the scale consisting of five questions/ categories, 25 participants were considered to be sufficient. To be able to answer the research questions, several inclusion and exclusion criteria were set. The inclusion criteria for the sample in this research were 1) Patients were under 36 months old of age, 2) Unable to express pain verbally, 3) Were scheduled for labioplasty and/or palatoplasty surgery, and 4) Agreed to participate in the study and signed informed consent. Additionally, patients who had an injury from another body part(s) that could be a source of acute pain other than the acute orofacial pain due to the cleft lip and/or cleft palate surgery should be excluded from the study.

2.2. Face, Leg, Activity, Cry, and Consolability (FLACC) Scale

The FLACC scale was first developed in 1997 by Merkel et al. and has been used to evaluate pain levels experienced by children aged from 2 months to 7 years that underwent an operative procedure. FLACC is an acronym for Face, Legs, Activity, Cry, and Consolability. This scale is commonly used to evaluate postoperative pain intensity in several previous studies [17, 20]. The scoring for each category consists of the number from 0 to 2, with a maximum score of 10. The interpretation of the FLACC total scoring results is as follows; 0 = calm and comfortable, 1-3 = slightly uncomfortable, 4-6 = moderate pain, and 7-10 = feels very uncomfortable or has severe pain or a combination of both. This scale can be used on both sleeping and awake subjects. Observations were made for two to five minutes for subjects who were awake and five minutes or more for subjects who were asleep. When making observations, the condition of the legs and body should not be covered. If necessary, it is possible to re-adjust the subject position. In the measurement process, it was necessary to evaluate the subjects’ muscle tone and body tension [23].

The validation and reliability evaluation of the FLACC scale was initiated by translating the FLACC scale from English to Indonesian by a native English speaker who is also fluent in Bahasa Indonesia. Once the scale was translated, the Indonesian version of the FLACC scale went through a review process by three experts within the field of orofacial pain, psychology, and linguistics. Afterward, the Indonesian version of the FLACC scale was back-translated to English and re-evaluated by the same experts (Table Sl). Back-translation is a method that has been widely accepted because it is considered to be effective in maintaining the content and meaning of the entire literature between the original version and the translated version [24]. Prior to the patient evaluation process, two of the authors and one field researcher were calibrated for the FLACC scoring. Five patients (data were not included in the study) who went through cleft lip surgery were evaluated during the calibration procedure. All three evaluators simultaneously evaluated the pain for three evaluation points (similar to the design of the study). Any scoring discrepancies during the calibration process were discussed and resolved. Although three evaluators were calibrated, only two evaluators performed the scoring for each patient in the current study, mainly due to the availability of the evaluators.

2.3. Data Collection

The first pain measurement was done using the FLACC scale after the patients had finished the cleft lip and/or cleft palate surgical procedure and the anaesthesia effect had worn off. For the first measurement, as all patients were awake, the observation was conducted for five minutes. As for the subsequent measurement, the observation period was five minutes for those who were awake and more than five minutes (if needed) for those who were asleep. No observation period exceeded seven minutes of time. Pain intensity was measured three times, shortly after the anaesthesia effect had worn off (T0), four hours after the first measurement (T1), and last, eight hours after the first measurement (T2). The research procedure was declared complete once the third measurement was done. Data collection was carried out by two field researchers who had been calibrated prior to the start of the study.

2.4. Statistical Analysis

The data obtained were then analysed by using the Spearman correlation analysis to test the validity of the FLACC scale, while the reliability of the Indonesian version of the FLACC scale was evaluated by calculating the Cronbach’s alpha value. For the validation evaluation, the component of the FLACC scale is considered to be valid if the correlation coefficient (rs) obtained is higher than the r table [25]. As for Cronbach's alpha value, the accepted reliability result is if the value is ≥ 0.7. The closer to 1, the higher the reliability of the scale [26]. Primary and secondary data were analysed using the IBM SPSS statistic version 23. A product of IBM (Internasional Bussiness Machine) Corporation in 2009 and this version was released in 2015 [27].

3. RESULTS

The participants of the current study were cleft lip and cleft palate patients aged less than 36 months that underwent corrective surgery and experienced postoperative pain. Most of the participants were male (57.14%) where 16 participants, regardless of sex, went through cleft lip surgery. Demographic characteristics, as well as clinical characteristics of research participants, are shown in Table 1.

In order to set determine the appropriate r table value, the degree of freedom has to be set. The degree of freedom (df) was then set at 26, considering that the number of subjects participating in the research were 28 (degree of freedom = number of sample - 2). The r table value for a degree of freedom of 26 with a significance level of 0.05 is 0.317. The results of the Spearman correlation analysis showed that the rs value for every component of the Indonesian version of the FLACC scale was greater than 0.317 (Table 2).

The reliability test was performed by analysing the internal consistency of each scoring category using Cronbach's alpha test. The Cronbach's alpha value showed a result of 0.88 (original value was 0.875).

Table 1. Distribution of research participants based on demographic characteristics and clinical characteristics.
Variable Category n Significant difference
Sex Male 16 p = 0.45
insignificant
Female 12
Age 0-12 months 14 -
13-24 months 14
Surgery Characteristics Cleft lip surgery 16 -
Cleft palate surgery 10
Cleft lip and palate surgery 2
Table 2. The FLACC scale correlation Spearman results.
Correlation test components Aspect of the questionnaire
F L A C C
rs value* 0.599 0.671 0.808 0.749 0.821
p value** < 0.01 < 0.01 < 0.01 < 0.01 < 0.01
Conclusion Valid Valid Valid Valid Valid
* r table value = 0.317 ** significance value is set at p < 0.05.

4. DISCUSSION

Using a behavioural-observation-based pain rating scales helps doctors and other health workers evaluate the pain intensity experienced by pediatric patients with limited cognitive ability or pediatric patients who can not communicate their pain verbally due to their medical-related condition [28]. There are several instruments that can be used to assess the intensity of pain in children, such as the Neonatal Infant Pain Scale (NIPS), the Douleur Aiguë Nouveau-né (DAN), the Neonatal Facial Coding System (NFCS), the Face Legs Activity Cry Consolability (FLACC), the Evaluation Enfant Douleur (EVENDOL), the Children and Infants Post-operative Pain Scale (CHIPPS), and the Crying, Requires increased oxygen administration, Increased vital signs, Expression, Sleeplessness (CRIES) [29, 30]. The FLACC has been acknowledged for its validity and reliability for the above mentioned purposes, which is proven by the results of the validity and reliability tests shown in previous studies [12, 16, 31]. The FLACC scale is considered to be fairly easy and practical to use [32].

Validity and reliability tests are the two main criteria to determine the quality of an instrument [33, 34]. The validity test is intended to test whether the measuring instrument used in research is able to measure the variable that is intended to be measured in the most accurate way. There are several types of validity, namely face validity, content validity, construct validity, and criteria validity [26, 35]. The validity test carried out in this research is the construct validity because we measured a specific construct, in this case, is postoperative pain [36]. Construct validity is considered the most valuable type of validity, yet the most difficult one to perform [37]. In the current study, the Spearman correlation analysis results showed that the rs value of each category of the FLACC scale was greater than 0.317. Therefore, the Indonesian version of the FLACC scale is considered valid. The results of the validity test revealed that each assessment category showed a strong correlation, where the highest rs value was shown by the “consolability” section, with an rs value of 0.821. The significance level of all assessment categories showed a p-value of less than 0.01, which indicates that all assessment categories were highly statistically significant.

The validity of the FLACC as a valid pain intensity measurement scale in postoperative patients has been reported in previous studies, including in post-craniotomy patients. In this study conducted by Suraseranivongse et al. (2015), the FLACC is found to be valid and reliable in measuring postoperative pain, especially for patients who are intubated [38]. This scale is also reported by Bai J et al. to have a significant validity value with a moderate to strong correlation coefficient when used as a pain intensity measuring instrument in post-cardiac surgery patients, especially in critically ill children [20]. Based on the results of these previous studies, it can be concluded that the FLACC scale is a valid pain measurement scale when used to assess pain intensity in postoperative children or children with cognitive impairment. Additionally, the FLACC scale is considered to be more effective when used in children under five years of age considering that their verbal skill is not yet well developed [17].

On the other hand, reliability is related to the consistency of a measurement tool [39]. There are three main components to the reliability, namely internal consistency or what is also known as homogeneity, stability, and equivalence. Cronbach's alpha is a reliability test that is most used to find the value of internal consistency. The calculation of the Cronbach's alpha value will result in a value that lies between 0 to 1, whereas the acceptable Cronbach’s alpha value of an instrument to be considered as reliable is ≥ 0.7. The closer to 1, the higher the reliability is [26, 37]. Considering that the Cronbach's alpha value of the Indonesian version of the FLACC showed a value of 0.875, the reliability of the scale in its current version is considered to be highly reliable. A similar result of the reliability evaluation of the FLACC scale is shown in a study conducted by Voepel Lewis et al. [16] on subjects who suffered from critical pain and, therefore, could not describe the intensity of experienced pain. In the study, the FLACC scale was considered to be highly reliable with a Cronbach's alpha value of 0.882. Another study that used the FLACC scale to evaluate pain in children aged six months to five years of age who experienced acute pain in an emergency also showed the FLACC scale to be reliable [15].

Currently, this pain intensity measurement scale is a pain scale that is widely recognized and one of the most used pain assessment instruments in hospitals. The FLACC scale is also considered as one of the easiest pain rating scales to use, especially by nurses that are not skilled and/or inexperienced [13]. The FLACC scale has been translated and validated in several languages​​, including French, Chinese, Portuguese, Swedish, Italian, Brazilian, and Japanese [16, 24, 31], and is considered the best pain intensity measurement tool for the measurement of postoperative pain in children compared to other scales [40-42].

The current study evaluated the validity and reliability of the Indonesian version of the FLACC scale by evaluating the construct validity and internal consistency. In regard to the current sample size, a further study involving more samples would be a solid addition to the current result. As for future direction, it is hoped that the current study result can be used as a solid foundation for the utilization of the FLACC pain scale in postoperative pain management at Unpad Dental Hospital.

CONCLUSION

This research concluded that the Indonesian version of the FLACC scale was proven to be valid and reliable in measuring pain intensity in postoperative paediatric patients with cleft lip and/or palate disorders.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

The research was approved by the Health Research Ethics Committee, Faculty of Medicine, Universitas Padjadjaran (KEPK-FK UNPAD), Bandung, Indonesia (Number: 715/UN6.KEP/EC/2020).

HUMAN AND ANIMAL RIGHTS

No animals were used in this research. All research procedures on humans were followed following the ethical standards of the committee responsible for human experimentation (institutional and national), and with the Helsinki Declaration of 1975, as revised in 2013 (http://ethics.iit.edu/ecodes/node/3931).

CONSENT FOR PUBLICATION

All participants’ parents signed an informed consent regarding their participation in this research and their agreement regarding the publication of the data gained in the current research.

AVAILABILITY OF DATA AND MATERIALS

The data supporting the findings of the current research are available from the corresponding author [T.M.] on reasonable request.

FUNDING

The current research was funded by the Ministry of Research and Higher Education research grant, Indonesia (Number: 1207/UN6.3.1/PT.00/2021).

CONFLICT OF INTEREST

The authors declare no (potential) conflict of interest, financial or otherwise.

ACKNOWLEDGEMENTS

All authors would like to acknowledge the doctors and nurses of the inpatient ward of Unpad Dental Hospital and the staff of The cleft lip and cleft palate foundation in Bandung, Indonesia, for their kind assistance and help during the research period.

SUPPLEMENTARY MATERIAL

Supplementary material is available on the publisher’s website along with the published article.


REFERENCES

[1] Worley ML, Patel KG, Kilpatrick LA. Cleft lip and palate. Clin Perinatol 2018; 45(4): 661-78.
[2] Beaty TH, Marazita ML, Leslie EJ. Genetic factors influencing risk to orofacial clefts: Today’s challenges and tomorrow’s opportunities. F1000 Res 2016; 5: 2800.
[3] Crockett DJ, Goudy SL. Cleft lip and palate. Facial Plast Surg Clin North Am 2014; 22(4): 573-86.
[4] Wehby GL, Cassell CH. The impact of orofacial clefts on quality of life and healthcare use and costs. Oral Dis 2010; 16(1): 3-10.
[5] Al-Namankany A, Alhubaishi A. Effects of cleft lip and palate on children’s psychological health: A systematic review. J Taibah Univ Med Sci 2018; 13(4): 311-8.
[6] Fakuade BO, Efunkoya AA, Adebayo AM, Adisa AO. Psychosocial impact of cleft lip and palate children on their parents. Niger J Basic Clin Sci 2018; 15(1): 50.
[7] Berger ZE, Dalton LJ. Coping with a cleft II: Factors associated with psychosocial adjustment of adolescents with a cleft lip and palate and their parents. Cleft Palate Craniofac J 2011; 48(1): 82-90.
[8] Peterson P, Nyberg J, Persson C, Mark H, Lohmander A. Speech outcome and self-reported communicative ability in young adults born with unilateral cleft lip and palate: Comparing long-term results after 2 different surgical methods for palatal repair. Cleft Palate-Craniofac J 2021; 10556656211025926.
[9] Pai BCJ, Hung Y-T, Wang RSH, Lo L-J. Outcome of patients with complete unilateral cleft lip and palate: 20-year follow-up of a treatment protocol. Plast Reconstr Surg 2019; 143(2): 359e-67e.
[10] Dawjee SM, Dawjee M. Introducing the Alveolar Moulding Obturator (AMO): an appliance for the early management of Cleft Palate Neonates. S Afr Dent J 2019; 74(3): 141-3.
[11] Reena , Bandyopadhyay KH, Paul A. Postoperative analgesia for cleft lip and palate repair in children. J Anaesthesiol Clin Pharmacol 2016; 32(1): 5-11.
[12] Crellin DJ, Harrison D, Santamaria N, Babl FE. Systematic review of the Face, Legs, Activity, Cry and Consolability scale for assessing pain in infants and children: Is it reliable, valid, and feasible for use? Pain 2015; 156(11): 2132-51.
[13] Makhlouf MM, Garibay ER, Jenkins BN, Kain ZN, Fortier MA. Postoperative pain: factors and tools to improve pain management in children. Pain Manag (Lond) 2019; 9(4): 389-97.
[14] Fox MA, Ayyangar R, Parten R, Haapala HJ, Schilling SG, Kalpakjian CZ. Self-report of pain in young people and adults with spastic cerebral palsy: Interrater reliability of the revised Face, Legs, Activity, Cry, and Consolability (r-FLACC) scale ratings. Dev Med Child Neurol 2019; 61(1): 69-74.
[15] Kochman A, Howell J, Sheridan M, et al. Reliability of the faces, legs, activity, cry, and consolability scale in assessing acute pain in the pediatric emergency department. Pediatr Emerg Care 2017; 33(1): 14-7.
[16] Voepel-Lewis T, Zanotti J, Dammeyer JA, Merkel S. Reliability and validity of the face, legs, activity, cry, consolability behavioral tool in assessing acute pain in critically ill patients. Am J Crit Care 2010; 19(1): 55-61.
[17] Redmann AJ, Wang Y, Furstein J, Myer CM III, de Alarcón A. The use of the FLACC pain scale in pediatric patients undergoing adenotonsillectomy. Int J Pediatr Otorhinolaryngol 2017; 92: 115-8.
[18] Pedersen LK, Rahbek O, Nikolajsen L, Møller-Madsen B. The revised FLACC score: Reliability and validation for pain assessment in children with cerebral palsy. Scand J Pain 2015; 9(1): 57-61.
[19] Cascella M, Bimonte S, Saettini F, Muzio MR. The challenge of pain assessment in children with cognitive disabilities: Features and clinical applicability of different observational tools. J Paediatr Child Health 2019; 55(2): 129-35.
[20] Bai J, Hsu L, Tang Y, van Dijk M. Validation of the COMFORT Behavior scale and the FLACC scale for pain assessment in Chinese children after cardiac surgery. Pain Manag Nurs 2012; 13(1): 18-26.
[21] Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth 2017; 11(Suppl. 1): S80-9.
[22] Maulina T, Nadiyah Ridho S, Asnely Putri F. Validation of Modified Dental Anxiety Scale for Dental Extraction Procedure (MDAS-DEP). Open Dent J 2019; 13: 358-63.
[23] Zieliński J, Morawska-Kochman M, Zatoński T. Pain assessment and management in children in the postoperative period: A review of the most commonly used postoperative pain assessment tools, new diagnostic methods and the latest guidelines for postoperative pain therapy in children. Adv Clin Exp Med 2020; 29(3): 365-74.
[24] Matsuishi Y, Hoshino H, Shimojo N, et al. Verifying the validity and reliability of the Japanese version of the Face, Legs, Activity, Cry, Consolability (FLACC) Behavioral Scale. PLoS One 2018; 13(3)e0194094
[25] Siegle D. Educational research basics. UConn 2018.
[26] Heale R, Twycross A. Validity and reliability in quantitative studies. Evid Based Nurs 2015; 18(3): 66-7.
[27] Kirkpatrick LA. A Simple Guide to IBM SPSS Statistics-Version 23. 0: Cengage Learning 2015.
[28] Srouji R, Ratnapalan S, Schneeweiss S. Pain in children: Assessment and nonpharmacological management. Int J Pediatr 2010; 2010: 1-11.
[29] Gall O, Champigneulle B, Schweitzer B, et al. Postoperative pain assessment in children: A pilot study of the usefulness of the analgesia nociception index. Br J Anaesth 2015; 115(6): 890-5.
[30] Beltramini A, Milojevic K, Pateron D. Pain assessment in newborns, infants, and children. Pediatr Ann 2017; 46(10): e387-95.
[31] da Silva FC, Santos Thuler LC, de Leon-Casasola OA. Validity and reliability of two pain assessment tools in Brazilian children and adolescents. J Clin Nurs 2011; 20(13-14): 1842-8.
[32] Lempinen H, Pölkki T, Kyngäs H, Kaakinen P. Feasibility and clinical utility of the Finnish version of the FLACC pain scale in PICU. J Pediatr Nurs 2020; 55: 211-6.
[33] Herr K, Coyne PJ, McCaffery M, Manworren R, Merkel S. Pain assessment in the patient unable to self-report: Position statement with clinical practice recommendations. Pain Manag Nurs 2011; 12(4): 230-50.
[34] Sullivan GM. A primer on the validity of assessment instruments. J Grad Med Educ 2011; 3(2): 119-20.
[35] Taherdoost H. Validity and reliability of the research instrument; How to test the validation of a questionnaire/survey in a research. Electron J Stat 2016.
[36] McLennan KM, Miller AL, Dalla Costa E, Stucke D, Corke MJ, Broom DM, et al. Conceptual and methodological issues relating to pain assessment in mammals: The development and utilisation of pain facial expression scales. Appl Anim Behav Sci 2019; 217: 1-15.
[37] Bolarinwa OA. Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Niger Postgrad Med J 2015; 22(4): 195-201.
[38] Suraseranivongse S, Yuvapoositanont P, Srisakkrapikoop P, Pommul R, Phaka W, Itthimathin P. A comparison of pain scales in patients with disorders of consciousness following craniotomy. J Med Assoc Thai 2015; 98(7): 684-92.
[39] Morton NS. The pain-free ward: Myth or reality. Paediatr Anaesth 2012; 22(6): 527-9.
[40] Crellin D, Harrison D, Santamaria N, Babl FE. Comparison of the psychometric properties of the flacc scale, the MBPS and the observer applied visual analogue scale used to assess procedural pain. J Pain Res 2021; 14: 881-92.
[41] Tamvaki E, Efstratiou F, Nteli C, et al. Validation of the greek version of comfort-B, FLACC, and BPS scales in critically ill children and their association with clinical severity. Pain Manag Nurs 2020; 21(5): 468-75.
[42] Merkel SI, Voepel-Lewis T, Shayevitz JR, Malviya S. The FLACC: A behavioral scale for scoring postoperative pain in young children. Pediatr Nurs 1997; 23(3): 293-7.