YT YouTube2Text

YouTube2Text → Transcripts → ¿Que tan confiable es un instrumento de investigación?

¿Que tan confiable es un instrumento de investigación? — Transcript

by Roberto Joel Tirado Reyes · 7,790 words · 1,197 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00the generation and application of
0:02knowledge in public health, management,
0:05care education, and mental health. He
0:09has developed academic production
0:11derived from research projects and
0:12thesis direction at the bachelor's,
0:14master's, and doctoral levels, which
0:16contributes to the training of
0:17specialized human resources and the
0:19strengthening of the scientific
0:21community. This work incorporates
0:25methodological, quantitative, and
0:28epidemiological tools, as well as
0:30intervention strategies in health
0:32promotion, particularly in programs
0:34such as Prevenimss, favoring the link
0:37between research, teaching, and social
0:39impact through the improvement of
0:42lifestyle in the population served.
0:45Welcome, Doctor. And we can begin with
0:49this participation. Very well, thank
0:52you very much, Doctor. A pleasure to
0:55greet you through this medium, and
0:58thank you very much to you for the
1:00invitation, as well as to your team,
1:03your academic body, and all the
1:05professors who made this meeting
1:07possible.
1:10>> I am going to start by sharing my
1:13screen. Could you please help me by
1:18letting me know when the presentation
1:23is visible. Give me just a minute and I
1:28think you will let me know there. I
1:32already have very well, correct. So,
1:36let's go ahead and get started. In this
1:41dissertation that we are going to have
1:44today, we will address some important
1:47aspects regarding the design and
1:50validation of instruments, as well as
1:52some key concepts. Dr. Herrera was
1:57mentioning to me that it was important
1:59to define if there was a method or a
2:04design. We will also be discussing how
2:08we are going to reach a consensus on
2:10some aspects related to the design and
2:12validation of instruments. I want to
2:16tell you that there is no single recipe
2:18for the design and validation of
2:20instruments. Yes, many of the aspects
2:24that we are going to address I will
2:27support with articles and publications,
2:30but also some of them are left very
2:33much to what is conventional among
2:35researchers, to the consensus among
2:38researchers. And well, at the end, I
2:42will give you a general review and I
2:44will show you an application that we
2:49have been working on for the design and
2:51validation of instruments, right? So,
2:54let's get started, let's get into the
2:56subject matter. Eh, as I mentioned, the
3:00purpose of this presentation is to
3:02identify the design and validation of
3:04an instrument based on a theoretical
3:05foundation. Yes. And well, we are going
3:10to analyze some aspects, and the
3:12objective will be for you to be able to
3:15distinguish the validity and
3:16reliability of the internal structure
3:19of an instrument. And to identify when
3:23to select, adapt, or design an
3:25instrument. Uh, something that
3:28sometimes creates confusion among
3:30doctoral students, master's students,
3:31and even among us as professors as well
3:33. how to choose and how to tie
3:38everything together in the design of an
3:41instrument regarding the very specific
3:43aspect of content validity. We are
3:48going to look at the Delphi method,
3:50modified Delphi, the content validity
3:53index by item and by dimension, the
3:56factorial analyses—exploratory and
3:58confirmatory—some estimators, how to
4:01adjust the model, and, well, here I
4:04present the route we are going to
4:06follow, right? From the initial idea
4:10until we carry out, beforehand, the
4:13design and validation of instruments.
4:16So, I am going to continue and start
4:20with this section. I am going to move
4:25this part a bit so it doesn't interrupt
4:27us. Usually, when we are in these
4:32stages of instrument design and
4:34validation or are developing a project,
4:37certain concepts and questions arise
4:40for the thesis student, and one of them
4:43is the selection, adaptation, and
4:46design of an instrument. We will select
4:51an instrument when we already have the
4:56construct or concept fully defined. And
5:02if this is already well-defined and the
5:05phenomenon, event, or problem—
5:07whatever you want to call it—has been
5:09studied extensively, well, if it has
5:12been fully studied, we will simply
5:14select an instrument that measures that
5:16variable of interest or those variables
5:19of interest to us. Yes. And well, it is
5:23assumed that this instrument has
5:24already been adapted to a population in
5:26context. Yes. and that instrument
5:30already has some approved metric
5:32properties. But when we are working on
5:37a project to design and validate an
5:39instrument, other concepts also come
5:42into play, such as adaptation and
5:44design. In the specific case of
5:50adaptation, it may be the case that the
5:53instrument we are trying to use is an
5:59instrument, for example, that is built
6:02in another language, perhaps in Chinese
6:04, German, French, yes? And then if we
6:08have the need to employ or use this
6:11instrument in our project, well, we
6:13would have to keep in mind that it is
6:16not enough to just do a translation or
6:19a transliteration of the instrument
6:22into the language in which we want to
6:25operationalize or implement it; it
6:27implies that we carry out an adaptation
6:30of the instrument, and for that, a
6:34series of important steps are required.
6:37Yes. And so I was telling you that
6:39translating does not equate to
6:41validating. To be able to validate, we
6:46would have to follow a series of steps,
6:49because it often happens that, perhaps
6:52in the transliteration from the
6:54original language to Spanish, for
6:57example, some concepts may not
6:59previously exist in the Latin or
7:02Spanish context. And so, the meaning is
7:07modified, and that can miscalibrate the
7:11instrument or cause us to obtain
7:14erroneous metrics. And so, what would
7:18be the path, the route to follow? Well,
7:22it would be for us, basically, to have
7:26a process in which a person who speaks
7:29the native language and who speaks
7:32Spanish audits our instrument; and also
7:35a person whose native language is
7:38Spanish and who has learned the
7:40language of the instrument makes that
7:43cross-check as well, so that we can
7:47validate and culturally adapt that
7:50instrument. Of course, there are other
7:52steps, I will explain them later on.
7:54This is just an introduction. And well,
7:59when we also, for example, identify a
8:02very particular situation and we want
8:05to propose an instrument to evaluate
8:08that event, that phenomenon, and there
8:11is no way to evaluate or measure it,
8:14then we will have to start from scratch
8:18to design the instrument. And so I want
8:22to make it clear here that to be able
8:25to carry out this analysis and also the
8:27adaptation, well, we have to have a
8:29theoretical source, right? I didn't
8:33mention it as much in the adaptation
8:35phase or the cultural adaptation of the
8:37instrument because it is assumed that a
8:39previous researcher already designed
8:41and validated it, and for this, they
8:43had to have a theoretical basis. But in
8:47the specific case, if we want to start
8:50from scratch to design and build an
8:54instrument, the first thing we have to
8:57do is dive into the literature and
9:00identify if the construct, the concept
9:03we want to measure, is fully developed,
9:07right? Or not. If it is not fully
9:10developed and there is confusion, then
9:12we would have to go directly to the
9:14population with the experts. Yes. And
9:18sometimes the experts, let me tell you,
9:20are not going to be people who have a
9:21master's or a doctorate. For example,
9:25it is often the case in some works by
9:27some thesis students who, for example,
9:30have culturally adapted instruments to
9:33Nahuatl, to Mayo Yoreme, for example,
9:36to Rarámuri, just to mention a few.
9:41And then, for example, imagine that we
9:42are designing and validating an
9:44instrument to understand or measure the
9:49perception of childbirth in a community
9:51, for example, or perhaps in Cora
9:52Nayarit. Well, the experts who attend
9:56births there would be the midwives, and
9:58they won't have a master's or a
10:00doctorate, right? But they are indeed
10:04the experts at the population level,
10:06because they will allow us to validate
10:09those concepts we are trying to define,
10:11and that's where the qualitative part
10:14comes into play, right? which sometimes
10:17generates controversy because some
10:20researchers say that the design and
10:22validation of instruments is a mixed
10:24design. There is a qualitative phase,
10:26there is a quantitative phase. There is
10:32still a lot of controversy in this area
10:34, but yes, for example, if we start
10:36from scratch, obviously we will need
10:39some purely qualitative aspects, and
10:41perhaps at first we will use a
10:43questionnaire. Yes. And then, based on
10:49this, we will carry out a survey or a
10:52semi-structured questionnaire directed
10:54at the population, and we will have to
10:57have those approaches with the experts
10:59of that community as many times as
11:01necessary. Theoretical foundation is
11:06required, it is a bit more demanding,
11:08but also when we do this, we can also
11:11end up making some mistakes. And what
11:15are some mistakes that we could
11:16potentially make? Well, not clearly
11:20defining the items. Yes. For example, I
11:25was telling you about people whose
11:27native language is the one in which the
11:29instrument is going to be adapted or
11:31designed, and perhaps that person, well
11:33, in Mexico there is the INPI and there
11:36are people who have certification in
11:38Mexican languages. So, as a quality
11:42criterion, it would be that the person
11:44who is going to collaborate with us has
11:46that certification. Yes. Also, basing
11:51all our energy and dedicating all
11:54validity only to Cronbach's Alpha or
11:58McDonald's Omega is also a mistake. And
12:02also overestimating the model, yes?
12:06When we want to force the model because
12:08they are already asking us for the
12:10thesis, or the professor is asking us
12:13to publish the article, yes? Or also
12:17piloting the instrument in a meager way
12:21, yes? And, for example, not having a
12:26consensus on the number of interviews,
12:29the number of interviewees that we
12:32require. And I think that more than
12:35interview, the correct word would be
12:37surveyed. Yes, because surveying is not
12:39the same as interviewing. And well,
12:42also not documenting the decision
12:44criteria we had, right? And well, there
12:48is no single formula, as I told you, to
12:52be able to carry out these steps. Yes.
12:57Next, I would like to present some
13:00concepts that we need to discuss a
13:04little before we dive fully into
13:08today's topic. Other concepts will come
13:14up that we will try to define. And I am
13:19leaving you some reference articles
13:20here for you to look up. Articles that
13:23I believe are necessary and that I
13:25always ask my students to consider for
13:27the class. Validity and reliability can
13:32sometimes be seen by some people as
13:34synonyms; however, that is not the case
13:37. Validity has to do with the evidence
13:40that supports the instrument. It is
13:44constructed based on the structure, the
13:45relationship of the variables, and the
13:47theoretical aspects. Yes. Contextual
13:51aspects of the population to which we
13:53are going to direct the instrument are
13:56at play, and reliability has to do with
13:58the degree of consistency, the
14:00stability, and the metric properties of
14:03the instrument that can be estimated.
14:08Imagine, for example, that you step on
14:11a scale and maybe you have a BMI of 24,
14:14and then you step on the scale and you
14:17weigh, what would you say? 65 kg. Well,
14:22you would be happy, right? But you get
14:24on it again for some reason, you have
14:26your keys in your pocket, you set them
14:28aside and get on it, and now you weigh
14:3090. What would you think? Because that
14:34scale lacks consistency, it lacks
14:36stability, right? Think about taking a
14:40fasting blood glucose test and maybe it
14:42comes out at 200, and then you have
14:45doubts, you take it again and now it
14:47comes out at 90; well, that device
14:49lacks stability, it lacks precision,
14:51right? The same thing happens with
14:54instrument metrics. It is important
14:57that we understand these aspects, yes,
15:00in order to perform correct
15:01measurements. I also leave you two
15:05articles there that I think are basic
15:07and will be able to help you. I leave
15:10them there for you to read and consider
15:13. They provide good contributions for
15:15the concepts I am discussing with you.
15:18And well, what is the path to follow?
15:21So, I was saying that there is no
15:24specific path based on the analysis of
15:27the literature. A project or thesis
15:33project to design and validate an
15:35instrument, or to culturally adapt an
15:38instrument, has different steps and
15:40sequences. Uh, I'm leaving you two
15:45articles that can help you there to
15:48provide light and direction in these
15:50steps, but notice that by analyzing,
15:53doing an integrative analysis, a
15:55systematic review of the steps, I think
15:58most of the articles reviewed have
16:01these steps that I present to you here.
16:05Number one is for you to ask yourself
16:08the question and define the construct.
16:11Yes. And for you to understand what
16:13dimensions that construct you are going
16:15to measure has. Review the literature
16:18of your conceptual framework, right? If
16:21there are other instruments similar to
16:23yours, right? What need is there for
16:26you to design, validate, or culturally
16:29adapt an instrument? Yes. Uh, generate
16:33your bank of items. And so here I am
16:36going to give you a tip. Previously
16:40they were called "reactivos," but the
16:42word "reactivos" has sort of become
16:44outdated in the Spanish language. But
16:48notice that the word "reactivo" in its
16:51etymological context comes from the
16:54fact that it makes you react. So, if a
17:00question you formulate does not make
17:03the person you are surveying react,
17:05well, it won't give you the information
17:08you require, or if it makes them react
17:11negatively, but you require a positive
17:14response, it will also generate some
17:16problems. So, it is important to write
17:20your items. It is always recommended
17:23that you do it in a positive way. Yes,
17:27you will find literature and also
17:30instruments in which some authors
17:33invert the scales. Well, that will
17:38imply a bit of confusion at times when
17:40it comes to interpretation. Yes, I
17:44suggest that you keep your items
17:46aligned. Yes. And well, once you have
17:51your bank of items, now comes the whole
17:54thing about how you are going to
17:56validate these items. And then some say
18:00, "I am going to use Delphi, the Delphi
18:02method." And some say, "No, I am going
18:06to use modified Delphi." Or I am going
18:09to use, for example, uh, Aiken's V. Or,
18:11"You know what? I am going to measure
18:15the content validity index for items or
18:17I am going to measure the content
18:19validity index by dimensions because I
18:21want to do an integrative evaluation."
18:25And well, some also say, "No, I want to
18:27see which items are essential in my
18:29instrument and I am going to calculate
18:31the CVR." And well, that's where the
18:34controversy begins, right? And, excuse
18:38the expression, but that confusion and
18:40mental diarrhea, that mental
18:42gastroenteritis starts to appear in the
18:44student, right? And also among us
18:47professors, sometimes we are not clear
18:49about where we are heading or how we
18:51are going to do it. I'm going to give
18:54you some suggestions, and I'll leave
18:55some articles here as well that will
18:57help you. Yes. And well, the next step,
19:01once you have this phase, is the pilot
19:03test, piloting your instrument to see
19:05if it is understood. There, you have to
19:08look for the response rate and the
19:10dropout rate of your items, whether
19:12they are understood or not. You will
19:16run an exploratory analysis, then a
19:17confirmatory one, and then you might
19:19say, "Hey, why are you asking for an
19:21exploratory one if I already have a
19:23theory and I want to confirm it?""Well,
19:27exploratory analysis helps us from a
19:29probabilistic point of view to
19:32determine how many dimensions the
19:34statistical software proposes from a
19:37statistical-probabilistic aspect." And
19:41well, you can then force the model and
19:44extract and rotate your dimensions
19:46according to your theory. Yes, don't
19:49get confused by this part; I will
19:51address it later in more detail. And
19:53well, some estimators, right? Some
19:55researchers also use some default
19:58estimators, and there are some oblique
20:02ones and some orthogonal ones. I'll
20:06give you tips on which one to use. Yes.
20:08And finally, the report. Are there more
20:12stages? Yes, yes, there are other,
20:15there are more stages. If I have time,
20:17I'll share them with you too. I will
20:18try to cover them. But basically, this
20:22is as far as most of the instruments
20:25you will find published go, yes? And
20:28that you will see in the papers. Yes.
20:32And well, now I'm going to—let's go
20:34back to the beginning. Let's go back.
20:36That's it, we've already covered the
20:38generalities. Let's go step by step so
20:40you don't get confused. Yes. The first
20:44thing we have to work on is the content
20:46validity of your instrument. And well,
20:51the whole thing about whether you went
20:53to the population and it wasn't defined
20:55, or it was defined, that was work you
20:57already did. Yes, now you've formed
21:01your item bank, and so now then. Which
21:05method do I use? Well, look closely,
21:10the Delphi method is widely used and
21:12helps us to reduce, for example,
21:14domains, but from a qualitative point
21:17of view, it allows us to establish
21:19agreements between experts and judges.
21:23And then someone out there said, "Well,
21:26yes, but that's just qualitative.""
21:29Someone out there invented the modified
21:32Delphi, where you can now measure
21:34quantitatively, and you can also use
21:36both. Yes. And it's a refined version
21:41of the instrument that registers
21:43changes from a quantitative point of
21:44view. Well, yes. Now, uh, there is also
21:49, as I mentioned, the content validity
21:52index per item, when what you're trying
21:54to do is evaluate each of the items,
21:56but you also have the other option.
22:01When you want to evaluate your
22:03instrument comprehensively by scale,
22:06there is the content validity index by
22:09dimension, which allows you to average
22:12out a global content validity. But
22:17there is also the modified Kappa, which
22:19is the degree of agreement or
22:22concordance estimated by probability.
22:25Yes. And so this allows us to adjust
22:29agreements, but from a probabilistic
22:31point of view, not from the point of
22:34view of chance or a hunch, right? Or
22:39because remember that one researcher
22:41may have one opinion and another a
22:43different one, and then you estimate
22:45the degree of agreement with Cohen's
22:47Kappa index, but you don't take the
22:49random point of view, and that can
22:50affect your results, right? And well,
22:55the one that is used the most, the one
22:58that is best known—I want to avoid
23:00brands, but it could be like the
23:03Coca-Cola of instruments—is Aiken's
23:06V. Yes, it is an ordinal assessment of
23:08the items where judges grade, in a
23:11gradual way, the clarity, relevance,
23:14and coherence of the items, and it
23:16allows you to establish scores, right?
23:21But when you also work with dichotomous
23:23scales, you can use an evaluation where
23:26you assess which items are essential
23:28and which are not. And for that, there
23:33is the Lawshe index or CVR, yes? Which
23:37attempts to measure and establish which
23:39items or reagents are essential for
23:41your instrument, right? So, uh, these
23:46do not compete with each other. Uh, we
23:50will see later what suggestions I have
23:52for you. I would suggest, well, I'd
23:54better wait because I'll give you
23:56suggestions when you work with scales.
23:59I'll show them to you there, and I'll
24:02suggest how to use, for example, Delphi
24:05and Aiken, or, for example, CVR and
24:07Delphi. I'll show you that later on.
24:11I'll leave this literature down below.
24:13Uh, it will be very useful to you.
24:15>> This is Hurtado 2025, Galet and S 24.
24:20Here are the DOIs. I recommend them to
24:23you. Read those. They are basic for you
24:25to understand these concepts. Yes. Now
24:30then, once we have already defined
24:32these aspects of content validity, we
24:35now move on to what is construct
24:37validity. We are moving up one more
24:40step, right? We are moving up another
24:44step in the process, and now we are
24:47going to look at exploratory and
24:49confirmatory factor analyses, when to
24:52analyze each of them, what the criteria
24:55are, and what these analyses seek. Well
25:01, like any statistical test, analyses
25:03are models, and to run statistical
25:06tests, I think your statistics teachers
25:08have already told you that there are
25:11criteria you must meet, right? Assumed
25:15criteria that you must meet in order to
25:17perform a statistical test. Well, in
25:21the case of exploratory factor analysis
25:23, in order to perform it and carry out
25:25a dimension reduction, exploratory
25:27factor analysis attempts to perform a
25:29reduction of dimensions. I want to make
25:32that clear. When you, for example, have
25:36a scale and that scale has, for example
25:38, 20 items, theoretically you are
25:41dimensioning your scale into 20
25:43dimensions, right? And so what
25:48exploratory factor analysis tries to do
25:51is for you to reduce those dimensions
25:55so that you can visualize and measure
25:58the same construct from different
26:01angles, so that you can measure it in a
26:04comprehensive way without leaving out
26:08important aspects. And maybe that
26:11doesn't mean much to you right now,
26:13does it? But I will give you an example
26:14, look, so you can understand it better
26:16. Suppose we are designing and
26:20validating an instrument, and it occurs
26:23to me that the instrument could be, for
26:25example, about Mexican food, right? So,
26:30for example, you generate some
26:32questions, but when you are making the
26:34questions, you realize that food from
26:36the north is nothing like food from the
26:38south, nor is the north like the center
26:40. So, imagine that you only consider,
26:46for example, the diet of the south in
26:49the instrument, and you don't
26:51contemplate the north; well, that will
26:54leave out other aspects of food
26:57evaluation in the scale. And so, when
27:02you want to measure, you are going to
27:05leave out important aspects within your
27:08instrument, right? So, roughly speaking
27:11, that is how I explain why it is
27:13important to dimension instruments. Yes
27:16. And well, in order to carry out those
27:20dimensions, those dimensions—I hear a
27:22microphone open somewhere. I am going
27:26to ask you, please, to close your
27:29microphone so that we can communicate
27:32better, right? And if there are any
27:34questions, you can interrupt me as well
27:36without any problem. I was telling you
27:39that in order to conduct a factor
27:41analysis, certain assumptions must be
27:43met, specifically the KMO, the
27:45Kaiser-Meyer-Olkin test, and Bartlett's
27:47test, which is based on chi-square. And
27:52so, we will look for parameters that
27:55fall above .70 for the KMO, ensure
27:58Bartlett's test is significant, and
28:01verify that there are factor loadings
28:04in the communalities above .30. Yes.
28:09And well, that is what we are going to
28:10look after. If we meet the Bartlett and
28:13KMO criteria, we have satisfied the
28:15first assumption and can then proceed
28:18to build the analysis. This means that
28:22your matrix, the database you have,
28:24indeed allows you to carry out or run
28:27an exploratory analysis with the data
28:30you have, right? But the matter of
28:34extraction and rotation also comes into
28:37play. Well, extraction methods allow us
28:41to extract the latent variables, the
28:44dimensions, and the subscales of our
28:46instrument. And well, it is important
28:50here that you, for example, take into
28:53account that one of the most used is
28:55principal components or principal axis
28:58factoring, because it is more robust,
29:01it is the most classic, and it seeks
29:03latent constructs. But there is also
29:06the minimum residuals method, which is
29:09very useful in contexts such as
29:11psychometrics, but with ordinal data.
29:15There is also maximum likelihood, but
29:17this will require you to have normality
29:20. And I will stop there, because it
29:23turns out that when you have to do an
29:26analysis of this type, as I told you,
29:29you must meet assumptions, and one of
29:31those assumptions is normality. And I
29:36won't mention brand names, but some
29:39widely used software, like SPSS,
29:42doesn't allow you to generate a
29:45multivariate normality test, like
29:48Mardia's test, and so that is a
29:51limitation, right? Some software like
29:56JASP or Jamovi do allow it, but we must
29:59also recognize that SPSS, for example,
30:02has the bootstrapping function, which
30:05allows you to generate subsamples until
30:08you reach normality to meet that
30:10assumption, right? But it doesn't
30:13calculate Mardia's test for you. Yes.
30:15And the Kolmogorov-Smirnov test or the
30:18Shapiro-Wilk test is a bivariate
30:20normality test. So, that must be made
30:23clear, right? Don't let them trick you
30:25with that. Well, the principal
30:29component analysis method is sometimes
30:32confused with exploratory factor
30:34analysis; however, this is an
30:36extraction method that also allows for
30:39modeling latent constructs. We'll leave
30:44the extraction methods there to move on
30:46to rotation. And rotations allow us to
30:50integrate the items further. I will
30:53speak to you about it in a general way
30:54because later I will explain each one
30:56of them. Yes, Varimax is an example of
31:00an oblique rotation, and Oblimin is an
31:03example of an orthogonal one. I will
31:06tell you later when to use them and
31:09with what criteria. And well, here I
31:11leave you some decision criteria that
31:13you have to look for. Yes, look for
31:16factor loadings above .30, preferably
31:19up to .50. Explained variances of 60%so
31:25that you can explain your constructs
31:27very well and ensure this is consistent
31:31with your theoretical design. Yes, be
31:35very careful to look for cross-loadings
31:37between the items. Yes. And here I also
31:41leave you some articles that will be
31:43very useful to you. I am giving them to
31:45you here so you can look for them by
31:47Tabachnick. They will help shed more
31:50light on the subject. These are
31:52articles that we usually read when we
31:54are talking about factor analysis. And
31:59well, then there is also confirmatory
32:01factor analysis and model fit. And well
32:05, here for example, I am going to give
32:07you a tip as well. When you work with a
32:10Likert-type scale, okay? Always work
32:15with weighted least squares. Watch out,
32:20hey, SPSS does not have this option,
32:25and sometimes you have problems fitting
32:28the model. Because of this, JASP has
32:31them, Jamovi also has them. Yes. And
32:35well, if you are going to work with
32:37dichotomous items, also use tetrachoric
32:39correlations, okay? And also weighted
32:42least squares, okay? And that will help
32:45you to be able to fit your model, right
32:47? Uh, I am going to give you some
32:51suggestions; here are some articles
32:54that will also help you so that you
32:57know what the minimum model criteria
33:00are that you must write about in your
33:03thesis and some best practice criteria.
33:07Yes. If it is possible to use an
33:10independent sample from the exploratory
33:12factor analysis, do it. Yes. Uh, do not
33:17modify the model just to improve the
33:20indices, because sometimes we go
33:22directly to the errors and we want to
33:25model the variances, the covariances of
33:28the errors to lower the factor loading
33:31and decrease the chi-square, right?
33:36There are even parameters that some
33:38mathematicians will tell you, if you
33:40load them, look for the highest ones,
33:42and they'll even give you an average of
33:44how much your chi-square will drop for
33:46you to keep in mind, but this part is
33:48not highly recommended. Yes, you can do
33:51it. You can also remove items, but
33:55always be careful not to affect your
33:57theory, so that when removing an item
33:59or a dimension, you aren't just
34:01removing a theoretical item or
34:03dimension because the mathematical
34:05model is suggesting it, while leaving
34:07out the theoretical aspects. Yes, that
34:11is extremely important, right? And well
34:14, here I will clarify, I'll give you
34:16some insights and tips. Take it, if you
34:18want, take a screenshot. And I will
34:21also share this presentation with Dr.
34:23Herrera so that if anyone wants to
34:26consult it later. Extraction methods by
34:29principal components, here they go.
34:33What do these do? Well, they seek, the
34:36principal component analysis. It seeks
34:40to find linear combinations, right?,
34:42among the original variables that
34:43explain most of the total variance. Yes
34:47, it is the most used one, and well, it
34:50provides data, and we will see that the
34:53factors exceeding unity are the ones
34:56that explain the most total variance,
34:59and the first factor is the one that
35:02does this, followed by the second and
35:04the third. Always locate them. The
35:07software will show you which ones are
35:09the first. Yes. Try to identify them so
35:12you know what the total explained
35:15variance is that it gives you. Review
35:18the correlation matrices. Yes, check
35:21your items. And very importantly, when
35:24you do these analyses, it is important
35:27that you check your database well, that
35:29you don't have missing data, because
35:32that can also give you errors. It
35:36commonly happens that maybe you are
35:38tired and the response option was 1, 2,
35:413, or 4, and since you did it in the
35:44early morning, you put 11 or skipped a
35:47cell, and that will also generate
35:50errors when running the analyses. In
35:54the extraction methods, right? In
35:57factor analysis, I was telling you that
36:00it is the principal components one.
36:04This one, and it is the one I told you
36:06is usually confused with exploratory
36:08factor analysis, seeks to maximize
36:10explained variance, and you will use it
36:13when the objective is to reduce
36:15dimensionality, what I told you, to
36:17build dimensions. Yes. In the case of
36:22exploratory factor analysis, it is also
36:24a common model that distinguishes
36:26between shared variance and specific
36:28variance. It is ideal for discovering
36:32latent structures, principal components
36:34, right? Which look for shared variance
36:38, and that will be very useful when the
36:41variables have a strong conceptual
36:43relationship. Yes. And keep this in
36:47mind because we are going to use it
36:49later on. Yes. When they have
36:51conceptual variance or share conceptual
36:53variance, keep that in mind because we
36:55will use it later. The maximum
37:00likelihood part, yes, well, it is also
37:02very good, it is the ideal one, but I
37:04told you that you have to meet the
37:07assumption of multivariate normality
37:09and that sometimes you cannot calculate
37:11the Mardia test, right? Or another test
37:15that allows you to carry out this
37:16analysis. Yes, here I leave you some
37:20articles that will also help you
37:22understand a little more about what we
37:23are discussing regarding the
37:25exploratory analysis. Um, some key
37:28recommendations, look closely here,
37:31take note of it and take a screenshot
37:34if you want, but when you are working
37:37with ordinal scales, right? It is
37:40important that you use polychoric
37:42correlations and robust extraction
37:44methods. Yes. And when these are small
37:50samples, watch the criterion. Honestly
37:55here, yes. Many speak about sample size
38:01, but many of these aspects are by
38:03consensus, by convention among
38:06researchers. Some professors or your
38:09professors will tell you," You know
38:11what? Your scale has 10 items, so I
38:15need at least 10*10, 100, you need 100
38:18interviewees. "And some will tell you,"
38:22No, you know what? With 50, because
38:24with 10%or 50%it is more than enough. "
38:28And that's where the controversy comes
38:30in. Well, always try to go for the
38:34intermediate or the maximum part. But
38:39let me also tell you that, if you have
38:42a very large sample and you exaggerate,
38:46what will happen is that, in the
38:48adjustments, in the condition indices,
38:51the P-value of the multivariate
38:54chi-square will come out significant,
38:57and you will be struggling there to
38:59adjust your model. Well, I missed a
39:04call, my apologies. So it is indeed
39:06important to reach a good consensus
39:08with your thesis advisor on this part.
39:10Yes. And well, um, the adequacy of what
39:16the data is. It is very important here
39:22that you meet the KMO and Bartlett
39:25assumptions. Yes, watch them very
39:27closely, make sure you meet them to be
39:29able to perform these analyses. Yes.
39:31And well, as with everything in life,
39:33practical rules provide guidance, but
39:35the decision must be based on the
39:37quality of your data and the coherence
39:39of the model. Yes, that's why I told
39:44you to be careful when establishing
39:47your sample, so you don't fall too
39:50short, or water down the soup too much,
39:54only to struggle later with the fit and
39:58have your professor ask for a
40:01multivariate chi-square with a
40:04non-significant p-value. And well, some
40:10students then say," Professor, why is
40:13it forced that I must have a
40:15non-significant multivariate chi-square
40:18in my model? "Well, because look, I'll
40:22put it to you this way. Imagine that we
40:27have, I don't know, a photograph, and
40:30you try to model or make a copy of that
40:34photograph and you have both. So, the
40:39more this copy resembles the original,
40:43the more similar they will be. And so,
40:49what we look for is that they are not
40:51different, because in modeling, what we
40:54seek is to create a representation of
40:56reality. That's why we look for those
41:01aspects, and some professors get a bit
41:03strict with the student and we ask for
41:06the model to fit and for the p-value to
41:09not be significant in the multivariate
41:11chi-square, right? Now, pay attention
41:16here, look, regarding rotation methods
41:19in factor analysis, I told you that
41:21rotation reorganizes the factors to
41:24facilitate their interpretation by
41:26adjusting the factor loadings. Yes? And
41:31so, there are orthogonal methods and
41:33there are oblique methods. Orthogonal
41:37methods, keep in mind, keep the factors
41:40independent. When you are building an
41:44instrument, it is important as a
41:47researcher that you are clear about
41:51whether the constructs or dimensions
41:55you have are correlated or not, because
41:59if they are correlated, you shouldn't
42:02use Varimax. Yes. And in most of the
42:07articles you will read, the papers,
42:09they report that a rotation was done
42:12with Varimax, yes? Or Equamax or
42:16Quartimax, which are variants of
42:18Varimax, yes? And so they never report
42:22whether the constructs are independent,
42:26but then some reviewer of the article
42:28or some professor in your thesis
42:31defense, if they want to trip you up or
42:34put you to the test, they will ask you
42:37this question. Are your constructs,
42:42your dimensions, independent? And if
42:47you say no, they will say," Why did you
42:49use—like the cat meme, right? Why did
42:53you use the orthogonal method? "Yes.
42:58And here I will give you an example.
43:00When analyzing a cognitive skills test,
43:02you could assume that the memory and
43:04reasoning factors are independent.
43:07Right? Or, for example, when you are
43:10working with an inventory, inventories
43:13classify, so the dimensions are not
43:15correlated. In that case, it is advised
43:19that you work with an orthogonal method
43:21. And when do I work with an oblique
43:25one? Well, you should work with an
43:29oblique one when your factors are
43:31indeed correlated. Yes. And you can
43:35apply Promax or Oblimin, right? When
43:39these are correlated—in most of the
43:42instruments we work with in health
43:45sciences, the dimensions are correlated
43:48. So, in that case, you should apply
43:52oblique, but it can also be the case
43:56that. Yes, here I leave you some
43:58supporting articles for what I am
44:00telling you from Share. Here is the DOI
44:04and, well, this name is a bit strange,
44:07but I’ll leave it here for you to
44:10check it out too. Yes. And I’ll give
44:14you an example: in a study on
44:16psychological well-being, the factors
44:18of self-esteem and personal
44:20satisfaction could be correlated, and
44:22so it would be correct that if you do
44:24an oblique rotation, you would be doing
44:27it with a good criterion, a good
44:28precedent. So, if the constructs are
44:32related or correlated, an oblique
44:35solution is usually more conceptually
44:37defensible. But what if I have doubts
44:40about whether they are or not, what do
44:42I do? Because it can also be the case
44:45that you have doubts about whether or
44:46not they are correlated. Well, if they
44:50are correlated or not and you have
44:52doubts, then run both. Run both. If
44:57there is uncertainty about independence
45:00, run both and analyze them. Use
45:02correlations between factors as a guide
45:04. Review the matrix. Uh, if the
45:07correlations between the factors are
45:09greater than point 30, pay close
45:10attention, here is the tip. There, if
45:14the correlations between the factors
45:17are indeed greater than 30, then opt
45:20for an oblique rotation, since the
45:22factors are probably related. If there
45:27are no significant correlations between
45:29the factors, then opt for an orthogonal
45:31one and the problem is solved. Yes. So,
45:35just check that. When you are going to
45:38do it and have doubts, do this. That is
45:41why, in some papers, some professors
45:43and some researchers report both and
45:46establish the criteria for why. When
45:50the correlations between the factors
45:52are greater than point 30, well then it
45:55means they are correlated, and then it
45:57is better if you opted for an
45:59orthogonal one. Always ensure your
46:04variable, your latent variable with its
46:08items, has loadings as close to one as
46:11possible; but between one dimension and
46:15another, the lower they are, the more
46:19it will justify using oblique or
46:22orthogonal rotation. This is the key,
46:26right? I’ll leave it here for you to
46:28have. Uh, it's not, it's not about
46:33hiding information. This is something
46:36that is sometimes complicated to
46:37understand. I was telling you about the
46:41mental diarrhea that this situation
46:42generates. And well, I’ll leave you
46:44some minimum criteria that you should
46:46report here. I won't go on too long
46:48because I’ve already exceeded the
46:50time for the talk. Uh, you tell me how
46:53much time I have left because I think
46:57I’m already deep into my time. Uh,
46:59minimum criteria to report. Are there
47:01more? Yes, there are more. But the
47:04minimums you should report in your
47:06thesis or paper, I’ll leave you other
47:08articles here that would also be
47:09convenient for you to read so you can
47:11report them in your thesis. Yes. Work
47:15suggestions according to the type of
47:17instrument. I’ll leave these for you
47:18here too. Look at what we were already
47:20discussing. I’ll summarize it for you
47:22here and also read this article by
47:24Pérez Ortiz and Tajari. They will give
47:27you more clarity. Yes, these
47:29suggestions are very good. If it is a
47:33Likert scale, I’ll tell you that if
47:36you are working with a Likert scale,
47:39use expert judgment, conduct a pilot
47:42study, perform exploratory and
47:44confirmatory factor analysis, and use
47:47the index per item. Also this
47:52integrative one by dimensions, uh,
47:55calculate Aiken's V, and also Alpha and
47:59Omega. If it is a dichotomous scale,
48:03watch out, use expert judgment here as
48:06well, conduct an essentiality review of
48:09which items you require using the CVR.
48:15Do your analysis with tetrachoric
48:16matrices, apply your exploratory and
48:18confirmatory factor analyses. Yes. And
48:22I suggest you use the modified Kappa,
48:25okay? If it is a questionnaire and
48:29there are well-defined domains and a
48:32measurement or screening purpose—
48:34which is being used a lot right now,
48:37screenings—I suggest you use this,
48:40validate your dimensions, do not assume
48:42a single factor. Use the Delphi, uh,
48:47and modified Delphi by dimensions. And
48:52if it is an inventory, and it has a
48:54solid theoretical structure, profiles
48:56or trait consistency subscales, then I
48:59suggest you use exploratory factor
49:01analysis, confirmatory, and reliability
49:04by scales. Yes, calculate McDonald's
49:08Omega, the Alpha, and adjust your model
49:11, okay? and look at the discriminant
49:15aspects, convergent and discriminant,
49:17that is, what we were talking about
49:19earlier, regarding whether it is
49:21orthogonal or oblique. Yes. And well,
49:27this would basically be what most
49:31papers report, and you could already
49:35publish your results. Some have added
49:41more, for example, and here I leave you
49:43the literature, what I already
49:45mentioned, if you want to measure
49:47reliability and it is a scale, use
49:49Cronbach's alpha. If it is a
49:53dichotomous scale, use the
49:55Kuder-Richardson, okay? And so you will
49:59work in that way, calculating the
50:01reliability. Yes. Another aspect that
50:06they also ask for, or that you could
50:08further advance in the design and
50:10validation of your instrument, is
50:12stability. Up to the previous point,
50:16right? Up to here, it would be
50:18measuring the metric properties
50:20specific to the instrument. And from
50:24this stability phase on, what we try to
50:28do here is measure the results, right,
50:31or external validity, the metrics that
50:34the instrument yields, and here
50:37everything related to the items and
50:40dimensions. Yes. Content, construct,
50:44and reliability validity. They evaluate
50:48all aspects related to the theory of
50:51the instruments, the consistency, the
50:53modeling, but starting from the next
50:56step, which is stability, we evaluate
50:58external validity. And there, what we
51:02try to do is see, for example, what I
51:04told you about the scale and the
51:06glucometer, right? If our instrument
51:09has stability, right? And so we can do
51:13that by considering, for example, in
51:17criterion validity, for instance, if
51:21there is a pattern test, a gold
51:25standard, with an R, with a Pearson's R
51:29, right? We could see, if I am
51:33measuring, for example, or proposing an
51:35instrument that measures university
51:38quality of life, but there is already
51:40another instrument similar to mine,
51:42then I can apply both and do a
51:44correlation to see if they have
51:46congruence between them. Is there a
51:50correlation between the results? Do I
51:53reach similar metrics, not identical,
51:56but similar? Yes. Or else, for example,
52:00through Cohen's kappa index if my
52:03instrument yields categorical data, or
52:06also if they are ordinal, you could use
52:09a...Wow, the name of the statistical
52:14test just slipped my mind. But, right
52:18now I'll remember, Kendall's tau-b,
52:21right, which is an ordinal correlation,
52:23and tau-c you could use as well. Well,
52:29if there is no reference test, no gold
52:32standard, then in that case, you could
52:35work with linear or logistic regression
52:38. You can also work with neural
52:42networks, right? To establish
52:44predictive validity. And what you are
52:48trying to do here is, for example,
52:50demonstrate if this instrument you are
52:53proposing really measures what you are
52:55claiming it does. But there is no gold
52:59standard, because it often happens that
53:01it's a new instrument with nothing to
53:04compare it to; and so, over time, we
53:06perform predictive validity using a
53:09regression or a neural network. Yes.
53:13And finally, another proposed step is
53:16determining the cutoff point, the
53:18sensitivity, and the specificity of
53:21your instrument. This part is rarely
53:26put into practice by authors in their
53:28papers, but it would be very good to
53:31also report the sensitivity and
53:33specificity of our instrument. Remember
53:38that sensitivity is the ability of the
53:40test or instrument to detect people who
53:43have the condition when they actually
53:46have it. Yes. And specificity is when
53:50people do not have the condition and
53:52the instrument correctly detects them
53:54as not having it. Yes. And maybe that
53:57didn't tell you much, so I’ll give
53:59you an example, right? The sensitivity,
54:03for example, of a glucometer, is the
54:06device's ability to detect a person who
54:10is diabetic when they are indeed
54:12diabetic. And the specificity is the
54:16ability of that same device to detect a
54:19person who is not diabetic when they
54:22are not diabetic, right? So, we do this
54:25using what are called ROC curves. There
54:30are other methods, yes, there are more,
54:33but the one most used, the most
54:36prevalent, and the best known is the
54:39ROC curve, the diagnostic performance
54:43curve, which also allows us to
54:45determine a cutoff point, and from that
54:49, we define the score—just like when
54:52you give an exam to your students and
54:56say: if this exam has 10 questions and
55:00you get five wrong, you didn't pass,
55:04but if you get six or above, you passed
55:07, based on a minimum criterion. Yes.
55:11And then you go up from there, right?
55:13Seven, fair; eight. Very good. Nine. Um
55:17, let's see, I think I got confused, I
55:19pulled a Peña Nieto, right? Anything
55:21less than five is a failing grade. Six
55:24is sufficient, seven is fair, eight.
55:28Wait, help me out there. The
55:32Alzheimer's is already kicking in. It's
55:37good, right? And then very good and
55:39excellent. My apologies, but it seems
55:42to be failing me too. I didn't take my
55:44medicine this morning. And I'll leave
55:46you some other literature here as well,
55:48okay? For you to consult. And well,
55:51I'll close the presentation. I'll close
55:53the presentation here of what we saw.
55:57Content validity. This is the first
56:01phase, construct validity. When you
56:05apply exploratory factor analysis,
56:08pilot testing, reliability when you
56:11measure alpha, omega, KR20, if it's
56:15dichotomous, stability when you try to
56:18compare scores, right? Criterion
56:23validity when you compare your
56:26instrument with another, with a
56:28standard test, a gold standard, and
56:31performance when you provide a score to
56:33interpret your instrument, because it
56:36happens a lot that researchers don't
56:39give you a cutoff point and you
56:41struggle to interpret it. Well, here I
56:46leave you the references and, well,
56:49I'll also quickly show you an
56:51application we've been working on to
56:54help. In this application, we try to
56:59replicate all those processes in one,
57:03and this application allows you, for
57:07example, to load your database and make
57:11a diagnosis of what is multivariate
57:14normality with Mardia, look for
57:17Mahalanobis to identify scores that
57:21don't have, well, congruence. It also
57:27allows you to see correlations, it
57:29allows you to see item-total
57:31correlation. Yes, the application also
57:37allows us to generally generate what is
57:40V-Aiken, and we are adding that to the
57:42application. In fact, I'm working on
57:47adding Delphi, adding modified Delphi,
57:50adding Kappa, and the others I
57:52mentioned. I already have the beta
57:56version, I'll share it with you later
57:58so you can pilot it, we are going to
58:00pilot it. And well, it also allows us
58:03to perform confirmatory factor analysis
58:05. And here we have also created a
58:09section so that if you have worked with
58:12JASP, JASP gives you the syntax, you
58:14copy and paste that syntax literally
58:17into our application and voilà, you
58:20can actually verify the calculations
58:22you made with JASP in our application.
58:28And well, making adjustments also
58:30allows us to calculate reliability,
58:33stability, ROC curve, and a scale. Yes,
58:38that's what we've been working on
58:41recently and well, we've also had
58:43progress, this is what the platform
58:45looked like at the beginning, right?
58:48And the first one, or the previous one,
58:51was how the latest version looks.
58:53Notice how, uh, it even allows us to
58:56generate the sedimentation chart, right
58:59? And the PAT charts that it provides.
59:05Yes, we will be able to replicate those
59:07there as well. And well, I've reached
59:09the Q&A section. Yes, I've reached the
59:13Q&A section and I invite you, while you
59:16prepare your questions.

About this transcript

This page contains the full transcript of ¿Que tan confiable es un instrumento de investigación? by Roberto Joel Tirado Reyes, generated from the public captions YouTube serves with the video. The transcript has 7,790 words across 1,197 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.