YT YouTube2Text

YouTube2Text → Transcripts → ANAB Webinar | AI and the Revised ISO/IEC 17024: What Certification Bodies Need to Know

ANAB Webinar | AI and the Revised ISO/IEC 17024: What Certification Bodies Need to Know — Transcript

by ANAB - ANSI National Accreditation Board · 8,229 words · 1,236 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:05Good afternoon and welcome. I'm David
0:08Feist and I will be the facilitator of
0:10this webinar today. Our presenter is Dr.
0:13VJ Krishna. Thank you for joining for
0:17our complimentary webinar on AI and the
0:21revised ISO IEC7024
0:24standard 2026 version. Please note we're
0:28recording this webinar so it will be
0:31available for viewing in the future. A
0:34few few logistics before as we get
0:38started. Uh due to the high attendance
0:41uh everyone will remain muted uh for the
0:44duration of the webinar. Please enter
0:46your questions into the Q&A box. We've
0:48planned time at the end of the webinar
0:51to answer any questions which are asked.
0:55Now, there is a disclaimer. The content
0:58that we deliver in this webinar is not
1:00guaranteed to fully prepare attendees
1:02for any specific application for
1:04accreditation. You're responsible for
1:07determining the applicability of this
1:09information presented to your specific
1:11circumstances. The examples provided in
1:14this webinar are for illustrative
1:16purposes only and are intended to
1:18facilitate your understanding of the
1:20concepts discussed. They're not to be
1:22interpreted as templates, prescriptive
1:25models, or representations of compliance
1:28with any specific standard. Now, on to
1:32our presenter today. Dr. VJ Krishna is
1:36the vice president of credentiing at
1:38ANAB, the ANC National Accreditation
1:41Board. VJ is an IIAF expert to the ISCO
1:45ISOCCO working group 30 revising this
1:49standard. VJ has been an active
1:52participant in the working group 30
1:55meetings to date and is a recognized
1:57world expert in this subject.
2:00VJ if you would like to take control.
2:05>> Thank you David. Uh thank you everyone
2:09for attending this uh webinar.
2:13So let me get started with uh the
2:17specific uh section on AI in the revised
2:2217024 standard.
2:25And for those of you who are not
2:27familiar with uh the 17,000 series
2:31standards, this is the first time uh any
2:34of the 17,000 series ISO cascus standard
2:38has specific requirements for the use of
2:42AI in the standard itself. uh which has
2:46uh implications both for the
2:50accreditation body as well as for the
2:52certification body because when you have
2:55requirements in the standard per se, the
2:59certification body is required to
3:02provide evidence and demonstrate
3:04compliance with those requirements and
3:08it becomes the responsibility of the
3:10accreditation body to evaluate those uh
3:13requirements.
3:14uh and then make a determination
3:16regarding compliance.
3:19The first thing uh I want to start off
3:22is uh just because the revised standard
3:26has a section on the use of AI
3:30the adoption of AI by a certification
3:33body does not become mandatory. So a
3:36certification body can decide whether
3:39they want to use AI, what stages they
3:42want to use. Uh in no way does this new
3:47section require
3:50CBS to start using AI. But one of the
3:53caveats that I would like to throw is
3:56even if you are a certification body and
3:59you are not actively using AI
4:03but it is certainly a possibility that
4:07some of the existing tools that you use
4:10might have AI embedded in them. So I
4:14would caution you from coming to the
4:18conclusion that you just don't use AI.
4:21uh but to look in terms of your existing
4:24platform technologies
4:27and develop a deeper understanding of
4:29how AI gets integrated in some of the
4:32existing tools. So the standard makes AI
4:35adoption as optional. But what the
4:37standard says is if you use AI then
4:42there are mandatory AI governance, AI
4:45accountability
4:47uh essentially to make sure that uh a
4:50certification body is using AI in a
4:53responsible manner. It has clear
4:56evidence to show that uh the use of AI
5:01is based on the specific requirements
5:04that are articulated in the standard. So
5:08I don't want anyone to think that the
5:09standard is now saying you have to use
5:11AI but if you use AI governance and
5:15looking in terms of specific uh ways in
5:18which AI should be used becomes
5:21mandatory.
5:24Now there are various definitions of AI.
5:28Uh even several ISO uh standards have
5:34somewhat uh different definitions of AI,
5:38AI systems, machine learning. When the
5:41working group 30 was looking in terms of
5:44how to define AI for the purpose of
5:48certification of person, we looked at
5:51existing ISO definitions. uh but we also
5:54wanted to make sure that that definition
5:57was relevant in the context of
6:01certification of persons. So this is the
6:05definition that the new standard has
6:073.31
6:09which essentially says autonomous
6:11systems that perform functional
6:13functions normally associated with human
6:16intelligence such as reasoning,
6:18learning, selfimprovement. So when you
6:22think of an AI tool, it could be an
6:25agentic AI, it could be an LLM platform,
6:28anything that you are using, if that is
6:33used or associated with functions that
6:37are typically performed by human that is
6:40what would fall within the scope u uh of
6:44an AI definition in this standard. So if
6:47you're looking at u say a Microsoft word
6:50or any other technologies it's not
6:52really doing things that are associated
6:55with uh a human in terms of their
6:58reasoning learning but if you jump on a
7:03LLM and then ask it to develop a scheme
7:06or develop an exam an assessment then
7:09essentially you are creating or you are
7:12using leverage to perform functions that
7:14are not really associated with ment. So
7:17this is really important to recognize
7:19the AI definition in the context of
7:23certification of persons.
7:26Now when we look in terms of AI use
7:29across certification life cycle, this is
7:32specific to certification of persons. Uh
7:35under 17024
7:37there are various touch points where AI
7:41could be used.
7:43It could be used in scheme development
7:45as part of job analysis in terms of
7:48looking at various job descriptions
7:51looking in terms of literature as to how
7:54the work is typically performed with the
7:57objective of developing a scheme based
8:00on clause 8 or 170 or 24. Uh AI could
8:04also be used for exam development
8:08uh generating exam items. uh it could
8:12also be used to review uh exam items
8:15developed by uh subject matter experts.
8:19It could be used uh in assessment and
8:22delivery like remote proctoring,
8:25practical exams, uh simulations that are
8:29done by AI agents or in an AI
8:33environment where AI is essentially
8:36performing as the examiner.
8:39uh AI could also be used uh in terms of
8:42communication with uh the applicant um
8:46the candidates uh certified individuals
8:50uh typically via chat bots.
8:54And it could also be used uh in a
8:57postcertification and decision support
9:00in terms of handling complaints in terms
9:02of handling
9:04summarizing data doing a lot of analysis
9:08uh of exam data analysis of the delivery
9:14uh in order to have better informed
9:16decision. So certainly AI has a lot of
9:19capabilities that helps for a
9:22certification body to gather evidence
9:25regarding the performance of the the
9:28certification assessment exam. So it's
9:31it's a really good tool for uh checking
9:34the health of an assessment.
9:37Uh similarly uh AI could be used uh in
9:41terms of decision support.
9:44Now the important aspect is to make sure
9:49that there is essential oversight and
9:52risk management.
9:54Uh which essentially means that you are
9:58looking at AI more as a tool as an agent
10:03versus
10:05um a standalone decision maker.
10:10When you look at the specific
10:11requirements for the use of AI, they can
10:14be grouped under these six broad
10:17requirements. And these requirements are
10:20deliberately spelled at a very high
10:23level at a very high construct level for
10:26the simple reason that uh when the
10:29working group met uh they wanted to make
10:32sure that these requirements are to the
10:35extent futurep proofed to the extent
10:38they provide various tools to emerge and
10:41it is just not possible especially for
10:44something as evolving and changing as AI
10:48to be prescriptive and to come down to
10:51the level of uh specific variable or
10:54detail. So think of these as broad
10:57constructs that should inform uh your
11:00decision making as part of a
11:02certification body. So I will uh look in
11:05terms of each of these requirements more
11:08uh as we go through this uh this
11:10webinar. uh essentially it is uh to
11:13verify, manage, monitor AI, validate
11:17outcomes with subject matter expert,
11:20personal competence, human oversight,
11:23validity, reliability and fairness and
11:26disclosure. And as you can see from this
11:30infographics, the central part of this,
11:34the way in which I look at it is human
11:37oversight,
11:38human in the loop. AI serves as a tool
11:42but uh it does not replace humans. Uh
11:46humans are still the makers of
11:48certification decisions who have full
11:51responsibility in terms of the outcomes
11:54of those decisions to make sure that the
11:58process uh remains fair, transparent uh
12:02and provides evidence that humans
12:06have been the decision maker. So if you
12:09look at these requirements um uh three
12:13of these requirements are very closely
12:16or directly tied to the human oversight
12:20uh which starts with personal
12:22competence. Making sure people who are
12:26responsible or tasked with uh oversight
12:30have the competence uh to be able to
12:33ensure oversight in a responsible
12:35manner. subject matter experts uh as
12:39part of the validation pro uh pro
12:41process and evidence of human oversight.
12:45So among all of these requirements, the
12:48key piece is competence and competence
12:51of people who have the responsibility
12:53for oversight, competence for people who
12:56are responsible for making sure that AI
13:00does what it is supposed to do uh within
13:04the boundaries that is defined by the
13:07certification body. So I will spend a
13:10little bit more time looking at um uh
13:13each of these specific requirements and
13:16what does that really translate. So if
13:19you are a certification body personel
13:22things that you should start looking at
13:23start considering
13:26uh putting together in terms of an
13:28evidence checklist would be something uh
13:31that I want to spend uh some time going
13:34over.
13:36So the first thing is AI verification
13:38process
13:40which essentially means that uh the AI
13:44system that you're using uh is
13:47performing as expected
13:50uh before and during the use uh to make
13:52sure that the intended results uh is um
13:55is captured. And what that essentially
13:58translates is in terms of as a
14:01certification body you are defining the
14:04parameters for expectation. So as an
14:08example if you have you know people
14:10coming to an office you would set an
14:12expectation that you have to come to
14:15office twice a week or thrice a week. So
14:19just like you said expectations for
14:21humans, you would look in terms of what
14:24is the expected outcome for an AI
14:27system. So for for example, if you're
14:31using an AI system to review a scheme,
14:35what is the level of accuracy that you
14:37would expect from an AI scheme reviewed
14:41by an AI agent?
14:44and then measuring the expectation with
14:48the performance of the AI system which
14:52will then help you to you know have an
14:55objective evidence in terms of whether
14:58the verification process is working or
15:00not. The next important aspect is look
15:04in terms of the overarching AI
15:07management framework. And to me it would
15:10seem like this is a really a good
15:13starting point for certification bodies
15:16as they start looking in terms of using
15:18AI to start very very broad in terms of
15:23looking at governance, looking in terms
15:25of policies, looking in terms of
15:27controls which would then start
15:30percolating down in terms of specific
15:33tools that you use, specific functions
15:36that you assign to any of these
15:39uh AI models or workflows but setting
15:42the broader context in terms of what the
15:45framework looks like. Do you have a
15:48policy in terms of how AI should be used
15:51for instance by employees? Do you have a
15:54policy how uh AI should be used by exam
15:57writers by subject matter experts? And
16:00how do you make sure that these policies
16:01are actually translated and implemented?
16:05Sometimes it may not be enough just to
16:07have policies on its own. It would be
16:10required for a certification bodies to
16:13essentially verify and confirm whether
16:16the people who are actually using uh
16:19some of these AI tools are implementing
16:22those policies and abiding by those
16:24policies. So that could be an important
16:27piece of evidence that you would need to
16:29look in terms of. So if you have uh a
16:32group of uh experts that are involved
16:35with writing exam questions, you might
16:37have a policy and you should have a
16:40policy to begin with which articulates
16:43how AI should be used by for instance by
16:46these um exam uh item reviewers or exam
16:50item writers.
16:53But it would not be enough for you to
16:56just ask them to sign on these policies
16:59or acknowledge these policies. But what
17:01would be your mechanism to verify that
17:03these policies are actually being
17:05implemented? So think in terms of uh
17:08starting off broad in terms of framework
17:11uh typically the hardware which would be
17:14your management system policies and
17:17governance and then start looking in
17:19terms of policies uh tactics and
17:22operational elements of AI use.
17:26Now we all you know are using AI in some
17:30form or the other and one of the things
17:33that should not come as a surprise to
17:36any one of us is the rate at which uh
17:39you know AI is evolving and changing uh
17:42and things that would be deemed as not
17:46accurate
17:48uh AI newer models are now able to
17:51improve significantly on things which
17:53were more deficient. So the standard
17:57essentially requires certification
17:59bodies to not just have a one-time
18:03verification process but have an ongoing
18:06AI monitoring system which would be you
18:09know over time detect issues make sure
18:13uh there is just like the AI models are
18:15continuously learning and improving the
18:18certification body is continuously
18:20learning and evolving in terms of how AI
18:23is being used and keeps space with the
18:26changes and evolution that's happening
18:28in the AI field itself. So it cannot be
18:32a static process in terms of one-time
18:35policy or one time review. It has to be
18:38something which is more dynamic
18:40proactive uh in order to make sure that
18:44uh these policies are reviewed and the
18:47monitoring is on an ongoing basis.
18:50The second set of requirements
18:54essentially talks about validation
18:56outcomes with subject matter experts.
18:59And what that means is when uh I started
19:04I talked about various touch points
19:07where AI could be used. So AI could be
19:09used for a job analysis. AI could be
19:12used for exam item writing. AI could be
19:15used for psychometric analysis. So for
19:19each of those situations, think in terms
19:22of validation through subject matter
19:24experts, which would mean that you could
19:26certainly use AI to do a job analysis,
19:29to do a background search and it's
19:32extremely efficient uh tool to get that
19:36done in a you know relatively uh quick
19:40period of time. So it's able to scan the
19:43universe of what you are looking to do
19:46in terms of deconstructing the job. But
19:49what the standard says is okay you have
19:51done that now show me the evidence that
19:54that was taken by the subject matter
19:57experts they reviewed it and they made
19:59sure that this output was fit for use.
20:02So as a certification body you would
20:04have to have that documented evidence.
20:07It could be typically in minutes in
20:09discussions. Um and it would not just be
20:14uh acceptance but also in terms of how
20:17do you make sure that uh it is an active
20:20involvement where the subject matter
20:22experts are the one who are making the
20:24final decision. So think of each of
20:27those inputs that you could use uh AI
20:31for as part of the certification
20:33process.
20:35And then think in terms for each of
20:37them. How do you make sure that the
20:40subject matter experts are involved and
20:42the expertise of the subject matter
20:45experts would also be critical. So it
20:47would be specific to the domain
20:49expertise for the job and also uh in
20:54terms of governance policy and technical
20:57expertise to make sure that there is
20:59confidence uh in the outcomes
21:03and as I mentioned uh look in terms of
21:06how do you collect documented
21:09evidence for each of these and this will
21:13help you to be proactive. So as you
21:16start looking in terms of using AI
21:19probably uh it's it's good to look in
21:22terms of um uh AI processes that you
21:25think would be the first you want to try
21:28based on the specific risk involved
21:31based on the the factor that would have
21:34the maximum impact. So think in terms of
21:37for each of them how are you going to be
21:40looking in terms of evidence that the
21:42subject matter experts were involved
21:44they had an active role and they
21:47essentially made sure that uh the AI
21:50output was something that they reviewed.
21:53So think in terms of uh output of AI
21:56systems becoming the input of subject
21:59matter experts and then they essentially
22:02making a final determination as the
22:05output for each of these uh inputs by AI
22:09and the the critical thing in uh
22:12certification accreditation is evidence
22:15evidence evidence uh and for each of
22:18them uh you have to have documented
22:21evidence to show that uh it was an
22:24active process and certainly things that
22:27are consistent across the standard in
22:30terms ofmemes being competent
22:33uh free of conflict uh of interest. Um,
22:37if you havememes who have known
22:40conflicts with certain uh AI vendors or
22:45certain conflicts that could impact
22:47their judgment or impartiality,
22:51that's something that you have to be
22:53cognizant of. firm and theme validation
22:56certainly helps uh technical error
23:00prevention making sure that the content
23:03uh is relevant and hidden bias that AI
23:07outputs could have and all of us
23:10recognize that um AI um can make
23:14mistakes uh it's prone to mistakes and
23:17this is something which uh even when you
23:19use uh any of these large language
23:22models uh they put out as disclaimers
23:26that AI can make mistakes. So it's
23:28really important for certification
23:31bodies to uh look at it from a
23:35perspective of risk. Um and the higher
23:39the risk, the higher it impacts uh the
23:43competence assessment of individuals,
23:46the more should be the involvement of uh
23:50uh the subject matter experts and human
23:52in the leaf. So it's always good to do a
23:54kind of risk analysis.
23:56Uh and the revised 17024 standard even
23:59in part of the management system which
24:01we will not be talking today essentially
24:04puts a lot more stress uh and focus on
24:07risk management.
24:09And that's a framework that you should
24:12be looking um to adopt as you look at
24:15various AI uh agents or tools that you
24:20intend to use uh now or sometime down.
24:23So there are different aspects uh of the
24:26certification process that comes up with
24:29different uh risk elements.
24:31And so looking in terms of risk
24:33mitigation strategy would be something
24:35that will help you to not eliminate but
24:40at least mitigate and make an informed
24:42decision whether you're able to live
24:44with the residual risk.
24:47This is again to me really really an
24:50important aspect of the new standard uh
24:54the competence of personnel uh in terms
24:57of uh AI oversight.
25:00And when I look at competence of persons
25:04uh uh to me uh you know you need to have
25:08uh technical understanding in terms of
25:10how these AI models work uh how these
25:14algorithms are written something uh
25:17which could look under the hood or
25:19understand the black box in terms of
25:23technical
25:24um requirements of how AI is used. uh
25:29just like in any domain specific that uh
25:33you use experts
25:35you should think of um AI competence uh
25:40similar to that. Now again the standard
25:42is not prescribing in terms of what that
25:45competence should look like.
25:47But the idea is for you as a
25:50certification body to look in terms of
25:52what is it that you want from the
25:55experts and does that expertise provide
25:58you confidence that uh the experts have
26:01done their best to help you as
26:04certification body uh identify how to
26:07use this AI responsibly. And you know I
26:09would think of it uh pretty similar to
26:12when you go to to a doctor uh based on
26:16the needs you would decide the level of
26:19expertise that is required. If it's
26:21something which is very specific
26:23high-risk specialized you would go to a
26:25specialist but if it's something which
26:28is um routine you would look at uh an
26:32internist or a physician. So think in
26:34terms of the AI expertise that is
26:37required and there are different types
26:39of AI expertise required. The technical
26:42uh expertise that talked about um
26:45expertise relating to risk management
26:49expertise related to governance uh AI
26:53standards expertise related to privacy
26:57expertise related to confidentiality
27:00data. So there are different types of
27:01expertise that is required. So it's
27:04really important for you to think about
27:06it
27:08which would be more in terms of legal uh
27:11ethical awareness uh bias data
27:15protection explanability
27:17and so the idea is to make sure that uh
27:20again I'll keep using the word using AI
27:23responsibly
27:25being aware of the risk that is in
27:28involved. You don't want to be in a
27:30situation that uh you made decisions
27:32which uh was not well informed. Uh and
27:37for that to happen certainly uh you need
27:41broad expertise uh in this specific
27:44domain and making sure that as
27:48certification body you have evidence of
27:50training
27:52documented qualifications of personnel
27:55and making sure that that evidence goes
27:58beyond just training and experience. the
28:01evidences in terms of demonstration of
28:04how people who are supposed to make
28:06these decisions were actually actively
28:09involved. So I spoke about some uh
28:12earlier in my previous slide some
28:13mechanism by which that can be done but
28:17it's really really important to make
28:18sure that uh there is evidence and it
28:22becomes even more critical because
28:24personally I can foresee um there could
28:28be challenges to certification decisions
28:31there could be challenges to how a
28:33certification body has handled a
28:35complaint
28:37um or an appeal or process uh especially
28:42if uh AI was used uh because uh to me it
28:48would seem like people expect uh these
28:51decisions to be made by humans um and so
28:54there is likely to be a greater
28:57challenge um uh when these decisions are
29:00made by AI agents. So it's really really
29:04important for legal protection uh to
29:07make sure that the certification body is
29:10aware of the legal requirements. You
29:12cannot uh claim ignorance in a court of
29:15law. So you have to have documented
29:18evidence of legal compliance. You have
29:20to have documented evidence that the
29:23subject matter experts were involved to
29:25make sure that the use is responsible.
29:28Again, it's not really meant to scare a
29:30certification body away. uh it's really
29:33meant to make sure that we have these
29:35real guards in place to make sure that
29:38there is confidence in the system and
29:41confidence
29:43among the users whether it's regulators
29:46team owners
29:48employers that EI was used responsibly
29:51and I think all of us are here to make
29:53sure that there is confidence in the
29:56certifications and accreditations that
29:58are offered
30:00based on 7924 for. So think in terms of
30:03yes we uh intend to use it's a great
30:06tool but we want to make sure we have
30:08expertise to do that
30:11uh in terms of oversight. So the first
30:14the earlier part of it says yes like you
30:17know you have to have people with
30:19required competence technical competence
30:21govern competence legal and statutory
30:24awareness uh competence. So 65D says
30:28okay you have done that now provide
30:30evidence that there was actual home uh
30:33human oversight of AI
30:36which essentially comes down to some of
30:38these aspects of u AI outputs uh making
30:44sure that uh humans retain the
30:47responsibility responsibility is not
30:50delegated to AI independently to make uh
30:54decisions
30:55And I would certainly urge uh
30:58certification bodies to think in terms
31:01of clearly defining responsibilities. Um
31:05uh so who would be responsible for the
31:10expected proper use of any of these AI
31:13that you have for various functions. So
31:16not just the person who's doing it but
31:20is the responsibility clearly
31:22articulated and defined. So it might be
31:25something that you might want to look at
31:28some of these um position descriptions
31:31that you have for staff managing various
31:34AI or subject matter experts using uh
31:37AI. So that's something that I would
31:40advise you to think in terms of if it is
31:43part of their defined responsibilities
31:46because then it puts the spotlight on
31:49them and makes them accountable for
31:52making sure that it was used in
31:54accordance with the CB's policy
31:59and also I talked about it earlier uh in
32:03terms of monitoring which has to be uh
32:06continuous
32:08an intervention which essentially means
32:11that um uh humans are actually involved
32:16with review. It's a rigorous process of
32:19AI outputs and humans provide those uh
32:24uh uh you know flags or issues and
32:27correct where it is required. So think
32:29of it as uh you know one of the common
32:32analogies that goes around uh in terms
32:35of AI is think of it as a doctoral
32:37student. So think of uh US certification
32:41body who has this doctoral student
32:43working for you. So you have to
32:45essentially uh continuously monitor the
32:48work that is done uh so that you're able
32:51to provide those interventions and make
32:54sure that you're able to use this very
32:57smart and intelligent tool. So that
33:00would be a good piece of evidence for
33:03you to have uh in terms of examples
33:06where the output uh of AI was reviewed
33:11by humans and then they corrected them
33:14or changed them or red flagged them. So
33:17those pieces of evidence are good to
33:19capture because that would show uh
33:22evidence of oversight.
33:24uh sometimes it's really hard to look at
33:27these requirements and think in terms of
33:28okay how do I really provide oversight
33:31so these would be good uh examples that
33:35you can capture as part of your meeting
33:38minutes you can uh capture as part of
33:42your actual workflows which would be
33:44okay we were creating an exam these were
33:46the exam outputs by um an AI agent and
33:50then we hadmemes review it and these are
33:53the changes. Just like you have track
33:55changes, various mechanisms could be
33:58used which highlights and captures the
34:01corrections or provides evidence of
34:04human oversight. So that's really a
34:06powerful way to show active oversight uh
34:10of AI
34:12and it's really important to recognize
34:14that there is no doing away with
34:16accountability. The certification bodies
34:20are accountable for their decisions.
34:23uh humans are the one that ultimately
34:27bear the responsibility
34:29and so the more transparent you are uh
34:32the more defensible is going to be your
34:35outlook um which is what all of us
34:39really want. So think in terms of these
34:41requirements uh uh as providing very
34:46grounded approach for certification
34:50bodies to be able to come uh and defend
34:54the use of AI in a responsible manner.
34:57And so having this requirements and a
35:00standard uh thus elevate the importance
35:03of AI for the use of certification of
35:06persons. At the same time it also
35:08provides you with the very defensible
35:12criteria. So these requirements in the
35:15standard was not just one or two people
35:17coming together and saying this is how
35:19AI should be used uh in personal
35:21certification. This is something which
35:25was you know an output of AI experts
35:30output of various uh you know
35:34participants who participated in the
35:37working group at ISO the voting process
35:40which essentially involved all the I
35:43think it's 162 countries that were
35:45involved in the process. So it's a
35:47global consensus standard which says
35:50okay we believe that these are the
35:53requirements for responsible use of AI.
35:56So when you take that in case you have
35:59to defend what you do it provides a very
36:02very strong framework that you're using
36:05to defend the certification decision. At
36:08the same time you have to have evidence
36:10to support uh each of these uh
36:13requirements in the standard.
36:16The the next requirement essentially is
36:20looking in terms of validity,
36:22reliability and fairness.
36:25And these are really really at the heart
36:27of certification. And those of you who
36:31are already accredited or familiar with
36:34the 1724 uh existing standard
36:38validity, reliability and fairness.
36:40These are the three words which um you
36:43know drive every certification decision.
36:46So validity essentially means
36:49what is the evidence that uh the
36:52intended outcomes are being met
36:55accurately. So does your AI do what you
36:58claim it to do? Uh and so for that you
37:01need to first define what the intended
37:04outcomes are and then measure it in
37:06terms of the AI outputs.
37:09The reliability of AI outcomes
37:12essentially would mean that is the AI
37:16consistent in terms of making sure that
37:19uh uh in terms of assessments in terms
37:23of what it does uh it is performing
37:28consistently. uh and there is one of the
37:31things that you will find is when you
37:33start using AI the way in which I look
37:36at it there are different ways in which
37:39you could be using AI one of it would be
37:42more across the board which you don't do
37:45it often but you do it intermittently on
37:49an ongoing basis like a scheme
37:51development so you could use AI for a
37:54scheme development
37:57where uh you know you have used it and
38:00then you have a time where that scheme
38:03is used to create an assessment, you
38:05administer it and then you come back and
38:08redo the scheme or validate the scheme.
38:10Then there are essentially AI uh
38:16agents or AI tools that are actually
38:19used uh consistently
38:23for test administration
38:25or for exam writing where the rate of
38:29improvement or the rate of changes
38:31becomes uh something that impacts um uh
38:35candidates uh over a period of time. So
38:39if you are using an AI for remote
38:42proctoring
38:43the system is learning uh and becoming
38:46better and better. So which would mean
38:49that um people who take test later on
38:53might end up with having a different set
38:56of requirements for uh how those test
39:01takers should be performing in an AI
39:04proctor environment or if an AI is being
39:08used for a practical examination. the AI
39:11model is learning and then how it treats
39:14the first candidate might be different
39:16from how it treats the second candidate.
39:18So what you have to think about is uh
39:22how do you make sure that you have
39:23systems in place to either lock down uh
39:27a a model for a period of time where
39:30you're testing uh in order to make sure
39:33that there is consistency of decisions.
39:36Um again there are different ways in
39:39which uh it could be done and that's
39:43where you look in terms of the technical
39:45experts who can help you figure these
39:48things out in terms of making sure that
39:50no testicle is disadvantaged because of
39:54changes uh in how an AI model is
39:58performing. So it's really important to
40:01look in terms of validity, reliability
40:04and the big one in terms of fairness
40:08and the standard uh we spent a lot of
40:12time looking in terms of the fairness
40:14issue and as most of us uh are aware of
40:20that a lot of training data is based on
40:24the existing data that has existed
40:28for the P period when you know things
40:32started getting digitalized
40:35and so there is a huge concern that uh
40:39some of the existing data that was used
40:42to train AI models might put certain
40:45groups at a disadvantage
40:48and so it's really important as
40:50certification bodies uh to be cognizant
40:53of that and this is not something new in
40:57a sense but it's something that you
40:59would have to focus looking in terms of
41:02putting AI uh as uh an intermediary
41:07variable uh to see if that is what is
41:11causing any changes in the outcome.
41:14Typically many certification bodies
41:16after they have administered the test
41:18they will try to do an analysis to see
41:21how the test has performed across
41:25various groups across various
41:27geographical boundaries to see if there
41:31is difference in terms of how the test
41:33is performing across these groups. And
41:37what typically they would do is if they
41:40find that the test is performing
41:42exceptionally well for one group uh the
41:46pass rate in the other group is
41:48extremely low. It could be geographic or
41:51it could be based on
41:55uh an ethical identity uh a group. um
41:59the certification bodies would go deeper
42:01and look in terms of what is causing
42:04those differences. Uh and if the
42:06differences on account of competence,
42:08the differences on account of training
42:10that can justify it then
42:14the certification body has done its due
42:17diligence. So the instrument per se is
42:21not discriminating or disadvantaging a
42:24group but based on other factors those
42:28results uh appear different.
42:33So when you look in terms of doing
42:35similar
42:37analysis of test, uh
42:41you should also look in terms of um is
42:45this also caused uh by any fluctuations
42:50in terms of how AI is interpreting the
42:54responses or how AI is interpreting how
43:00[clears throat] someone from a specific
43:02group uh has performed on the test. So
43:05be mindful of that. Uh but the idea is
43:08to make sure that uh uh AI is uh you
43:15have evidence to support its fairness
43:17and the CP has done its part in terms of
43:21bias uh mitigation.
43:25So the last of these set of requirements
43:30is in terms of disclosure and
43:32acknowledgement
43:34and it is up to the certification body
43:38to decide its disclosure policy.
43:44You can decide based on what you want to
43:47disclose, how much you want to disclose.
43:50Uh all of that is certification decision
43:53just like u some of the certification
43:55bodies uh would uh have the list of
43:59people who were certified disclosed uh
44:03provide other information about people
44:06who hold the certification through a
44:08badge uh and that uh micro credential or
44:12a badge could unpack some of the
44:14competence. So there are specific
44:16requirements. So analogous to that uh
44:20for the use of AI the certification body
44:22can choose what it wants to disclose in
44:25a responsible manner. But what the
44:27standard specifically focuses here is in
44:30terms of uh AI usage in communication
44:34interactions. So when an AI chatbot uh
44:38is interacting with an applicant, a
44:41candidate during a live exam or someone
44:44is certified
44:47that must be disclosed uh and uh consent
44:53and acknowledgement must be obtained.
44:56And this is something which probably uh
44:59you know most of us might have
45:01encountered with the advancement of some
45:04of these technologies. It becomes so
45:06hard to you know decifer whether the
45:09person that you're instructing is a live
45:12agent or an AI bot. Uh it could be uh
45:16through a voice agent and some of these
45:18voice agents sound exactly like human.
45:21their understanding is top class and so
45:24does um chat uh and messaging system. So
45:29the standard says when AI chatbot or AI
45:34is used in terms of communication
45:36interactions with the applicant with uh
45:40the candidate certified individuals then
45:44that must be disclosed and the person
45:48should acknowledge uh uh as part of an
45:51explicit uh consent to that. So these
45:55are specific requirements. So what does
45:57this really mean? uh for certification
46:00bodies looking for transition to the
46:032026 version of the standard start off
46:06with looking in terms of AI tool
46:09identification. So this is my
46:10suggestion. Make a [snorts] a directory
46:14or a list uh of okay you know right now
46:17as a certification body these are the AI
46:20tools that we are using. So list them
46:23make it into a list that's active. Uh so
46:26when you add on new tools new AI agents
46:30new users of existing agents keep
46:32updating them. That way at least you
46:34know what are the tools that uh as a
46:38certification body you're using.
46:42The second thing is look in terms of uh
46:45establishing your governance policies.
46:48So and that's something that uh you know
46:5172 requires to make sure you have
46:54policies procedures processes in place.
46:57So think in terms of starting off with
46:59broad framework uh management system
47:02policies and one of the things which I'm
47:04not covering here today uh is also in
47:07the revised standard is as part of the
47:11management review you also have to look
47:13in terms of how the AI performance is
47:17happening uh the output of AI is being
47:20reviewed to make sure that it conforms
47:22to the expected policies and governance
47:25procedures in place. So think in terms
47:28of overarching governance. Um think in
47:31terms of yes we're going to use this
47:33trade. Uh do you have competence uh
47:37among the staff among the volunteers
47:39among the subject matter experts that
47:40you are using to use it in a responsible
47:43manner. So you know training that you
47:45provide uh document that keep evidence
47:48of that training keep uh evidence of
47:52experience monitoring
47:54so that uh you have evidence to show
47:59that uh you have trained people for the
48:02specific task that uh they are
48:05responsible for in terms of use of AI.
48:08So think in terms of uh documenting some
48:11of these training uh and look in terms
48:14of uh transparency and accountability
48:17who at the end of the day within the
48:19certification body is responsible. So
48:22chart out responsibility roles within
48:25the certification for specific uh AI
48:29tools for specific AI functions uh and
48:33then establish a hierarchy of uh
48:36responsibility and accountability within
48:39the certification body that will then
48:41make sure that there is active
48:43oversight. You're using this tool which
48:46is fabulous but you're using it
48:49responsibly responsibly and making sure
48:52that the stakeholders have confidence in
48:55the system that uh uh is out there. So
48:59with that, I think um [clears throat]
49:02David u I will turn it over to you to
49:06talk through additional uh resources.
49:10And uh
49:11>> thank you VJ.
49:13>> Great presentation. We've had quite a
49:15few questions come through and we're
49:17going to get to those after we discuss
49:20just some quick additional resources uh
49:23that we have. If this topic fascinated
49:27you, uh VJ and Dr. Adrien Cadell uh
49:30hosted a workshop on 170 to 24
49:33transition. We are we are placing that
49:37in an ondemand training that is
49:40available for purchase now. Uh it is
49:43being released uh this month. And to
49:46give you an idea
49:49um of what we're looking at here
49:53um
49:55see
49:57this is an example of
50:01what that uh
50:04course will look like. Um, it's broken
50:09down into 10 different sessions. And as
50:12you go through the sessions, it will
50:16come across like this.
50:34So that gives you an idea of what this
50:37session could be like and there are as I
50:40said 10 different sessions in 17024 here
50:43along with questions along with
50:45knowledge it'll talk about what you're
50:48going to learn in this session key
50:50takeaways some terms and definitions so
50:53that gives you an idea of uh what you'll
50:57see here let me share the get back to
51:01sharing the presentation here.
51:05Okay.
51:08All right. And let's get to where we
51:10were in the presentation.
51:13Um,
51:16we're at the point where we're ready to
51:19take some questions and we've had quite
51:21a few come in. So for example,
51:30the first question BJ is for slide eight
51:34would we see one artificial intelligence
51:37tool for each of these requirements or
51:40some combination of AIS?
51:44>> Yeah. So typically you could and it it
51:47would depend in terms of whether you are
51:49looking to have dedicated AI agents for
51:53each of these tasks. So typically uh the
51:57way in which these systems work is you
52:00could have uh a large language model
52:03that sits at the core of these AI
52:07functionalities
52:09and from that large language model you
52:12could decide oh I'm going to have
52:14different agents
52:17uh and that could be one way of doing it
52:20and these days uh you know having
52:24multiple agents is becoming the now. Uh
52:27so you know when you think of uh say
52:30exam development
52:32even within that functionality you could
52:35have multiple uh agents. One one agent
52:39could be looking in terms of developing
52:42uh core questions. Um some uh other
52:46agent might be looking at looking at the
52:48stem of each of these questions. So
52:50there are multiple ways in which it
52:52could be done. Similarly for a scheme
52:56development, there could be one agent
52:58that does the entire scheme or it could
53:01be multiple agents which look at
53:03specific knowledge domains and then they
53:06get specialized.
53:08In a sense, it's not really different
53:10from how humans work in real life. firm.
53:15So when you look at a job analysis,
53:17typically you would bring in these
53:18subject matter experts who would have
53:20different type expertise come together
53:23and then make uh a decision about uh
53:26what that scheme should look like. So
53:28it's possible to to do that. Uh but my
53:34my understanding or what I've looked at
53:36is uh you would start off with a
53:39foundational model. uh and that model
53:42could be used if you don't want to go
53:45the agentic AI route, you could just use
53:48the model for a lot of functions which
53:50would then uh save you from going the
53:53the agentic route but it is certainly an
53:57option for uh the CBS to consider.
54:03>> Okay. Thank you VJ. You noted that this
54:06is the first ISO 17,000 series standard
54:09to address AI. Are other ISO 17,000
54:13standard working groups formulating
54:15similar requirements? If so, is there a
54:18general timeline that these changes are
54:20anticipated to roll out?
54:23>> Yeah. And I think this like the way in
54:25which uh at the ISO level this is
54:27working is uh ISO has left it to each of
54:32the working group to make that
54:33determination whether they want to to
54:36include AI specific requirements or not.
54:40I'm aware of at least uh another 17,000
54:43series standard that came off for
54:45revision which was revised and published
54:47where the working group decided that
54:50they did not want to include uh
54:52specifically anything with AI. Um, and
54:57so across the 17,000 series standards,
55:01uh, I'm not sure how that's going to
55:04unfold, but my personal hunch is, um,
55:09uh, if the standard does not come out
55:11with these requirements, uh, I would
55:14think at the global ACI level, uh, there
55:17could be a mandatory document or
55:20requirements that comes and addresses
55:22these issues because I
55:25you know we see the use of AI across
55:28product lab management system. So it's
55:31not something that's only unique to 17
55:34or 24. Just like the scheme uh if you
55:38look at 1724 that's the only standard
55:42which has specific requirements for the
55:45scheme inbuilt in the standard. other
55:47standards. Every standard uses a scheme
55:49but it is not uh uh you know spelled out
55:53as requirements in the standard but then
55:56you ended up with an if MD25 document
56:00which uh spells out the requirements for
56:03a CV. So it it's a possibility but I
56:07don't foresee my personal take is I
56:10don't think uh there is going to be uh
56:14all the standards coming out now with AI
56:17uh requirements ISO has given it uh
56:22consideration and has allowed each
56:24working group to decide based on uh you
56:28know the context for that and to me it
56:30would seem like like it's really really
56:33critical ical for persons and so we made
56:36a conscious choice to have it.
56:40>> Okay. Um, another question with AI being
56:44performed on cloud-based systems, how
56:46can you ensure that confidential data is
56:49protected when we're using AI for
56:51auditing workflows?
56:53I I think that that that's that's a
56:55really great question in terms of uh
56:58what is uh you know stored in systems
57:02where CBS would not have access to uh to
57:06some of the some of the data that's
57:09managed uh through you know not cloud
57:12certainly is uh what every uh
57:15organization doing these days but I
57:18think like the CBS would have to look
57:22deeper or in terms of the contractual
57:24relationship they have with any of these
57:27providers. Um to see in terms of how
57:31much of it uh uh is covered as part of
57:36confidentiality
57:37and to me it would seem it's not
57:39significantly different from what is
57:42currently happening because right now a
57:44lot of data for candidates uh and test
57:47takers is is uh stored in the cloud. um
57:51uh you know and uh it may not have been
57:53through AI. So I think the same
57:55challenges uh of data storage, data uh
58:00retrieval and confidentiality would have
58:02to be followed. Uh so I don't think it's
58:05a AI specific question. It's just uh the
58:08way in which uh uh candidate data is
58:11stored uh on cloud uh currently.
58:15>> Okay. Um
58:18there's a thank you here VJ for
58:20presenting specific to AI. Can we create
58:23a foundational workflow diagram or
58:27evidence for a schematic use of AI in
58:30certification?
58:33So uh is that I'm not sure we is is that
58:37we to uh ANAB or is that but I think
58:40like certainly uh one of the things that
58:43ANAB is committed to uh is we want to be
58:47able to disseminate uh uh every bit of
58:51learning that we have uh as we start
58:54implementing these requirements. So we
58:56have our assessors who would be going
59:00through additional training. we come
59:02together, we look at it. Uh we are also
59:05uh very actively involved
59:07internationally looking in terms of how
59:09different countries whether it's Europe,
59:10Asia, Pacific is dealing with it. So
59:13what we will certainly do uh is we'll
59:16try to bring all of that uh uh lessons
59:20learned and shared with the community
59:23because I think it's it's good for all
59:25of us. Certainly uh we will be committed
59:28to that but if there is specific uh
59:31schematic representation how we could do
59:34that we can certainly look into it.
59:39>> VJ there's an interesting question uh
59:42specific to AI tools not developed by
59:45the CBS. uh do since the CBS are not
59:49involved in the building or training of
59:52the artificial intelligence tool are
59:55they allowed to use that artificial
59:57intelligence gadget?
59:59>> Yeah. Yeah. Yeah. Certainly like you
1:00:02know I don't think any certification
1:00:03body is going to building uh an LLM like
1:00:07like you know right now we have only you
1:00:09know few companies a handful of
1:00:11companies globally that can do that. Uh
1:00:14but the idea is like how do you take
1:00:17that tool with all the good bad and ugly
1:00:20which is part of that learning uh and
1:00:24then look in terms of how do you build
1:00:27and train it because the tool comes with
1:00:29certain functionality. So right now you
1:00:31can just go to uh any LLM and say create
1:00:35an assessment tool for vibration
1:00:38analyst. It'll create that. Uh but what
1:00:41we are looking at it is you look at you
1:00:43take that and then how do you establish
1:00:46parameters in terms of use how do you uh
1:00:51create lockdowns and how do you
1:00:53essentially create uh uh a controlled
1:00:57environment where the the big model so
1:01:01think of it as you know you got this um
1:01:05uh person who got uh a degree a
1:01:09bachelor's degree from a university,
1:01:11right? So, you have no real control
1:01:12about what this person really learned.
1:01:15Uh but once you bring them within your
1:01:18organization, how do you train them
1:01:20based on corporate governance, based on
1:01:24organization policies,
1:01:26uh procedures, cultures so that the
1:01:29behavior of that person within the
1:01:32organization is what you have control
1:01:34over and the person employee behaves
1:01:36based on the company's policies,
1:01:38company's expectations. So think of it
1:01:41as you know you are going to get an LLM
1:01:44which was created outside. So once you
1:01:46bring it in, how do you uh I don't know
1:01:50whether to use the word indoctrinate or
1:01:52how do you condition that uh to make
1:01:55sure that uh that system is living
1:02:00governed based on CB's policies and
1:02:03procedures and all the learning and
1:02:05there are different ways in which that
1:02:07can be done.
1:02:10>> Thank you VJ. Um, if you had a question
1:02:14that we didn't get to or you weren't
1:02:17able to ask, please visit our website
1:02:20when this recording posts, uh, and you
1:02:23can submit your questions there and we
1:02:25will do our best to answer them. Um, if
1:02:28you're interested into the 17024
1:02:31transition on demand course, um, we
1:02:35encourage you to look at that on our
1:02:36site. uh once you register and it opens
1:02:39you will have availability to that for
1:02:4230 days. Again VJ thank you very much
1:02:45for an informative presentation.
1:02:48That concludes our webinar for today.
1:02:51Thank you.

About this transcript

This page contains the full transcript of ANAB Webinar | AI and the Revised ISO/IEC 17024: What Certification Bodies Need to Know by ANAB - ANSI National Accreditation Board, generated from the public captions YouTube serves with the video. The transcript has 8,229 words across 1,236 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.