YT YouTube2Text

YouTube2Text → Transcripts → GMT20251209 170003 Recording gallery 3840x2160

GMT20251209 170003 Recording gallery 3840x2160 — Transcript

by Advertising Research Foundation · 6,389 words · 898 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hello, I'm Earl Taylor, the chief
0:01knowledge officer at the Marketing
0:03Science Institute, a division of the
0:04Advertising Research Foundation, and
0:07it's my pleasure to welcome you here
0:08today to our webinar on how generative
0:11AI is transforming the consumer insights
0:14industry. I'll introduce our speaker in
0:16just a moment, but as I hope most of you
0:17know, MSI is that unique organization
0:20that brings together leading
0:22forwardinking companies and top
0:24marketing academics around the world to
0:26develop new marketing knowledge. And we
0:28do that in a variety of ways. We poll
0:30our members about the most important
0:32challenges they're facing. We
0:34selectively fund research on those
0:36topics with our marketing academics. And
0:38then we share the results of that
0:40research with you in a variety of ways.
0:42And we're delighted today to have Najora
0:45with us. He is the chair of the
0:47marketing research and education program
0:50at the University of Wisconsin's Madison
0:52School of Business. uh he is also the
0:54academic executive director of the
0:58marketing leadership institute there.
1:00His research focuses on uh artificial
1:02intelligence, large language models,
1:04digital marketing effectiveness,
1:06consumer choice among other topics. Uh
1:09he's widely published and in fact the
1:11research that he'll be sharing with you
1:12today is getting a lot of currency. It
1:15also appears as an MSI working paper
1:17earlier this year and we'll be sharing
1:19that with you after the event. So we're
1:22delighted to have Nirage with us today
1:24to tell us how generative AI is
1:26transforming the consumer insights
1:28industry. Uh please do submit questions
1:31you have for NJ at the Q&A function at
1:34the bottom of the screen and there will
1:35be a short discussion uh at the end of
1:37the presentation. So NJ
1:39>> thank you Earl. Thank you for uh
1:41inviting me to share
1:44uh this work I've been doing in in the
1:47journey of AI consumer insights
1:50intersection and
1:52uh I'm happy happy to be to be here. So
1:57the let me let me start off by just a
1:59big picture in terms of how generative
2:01AI is affecting
2:03uh the business in in in general and uh
2:07a few years ago the predictions were
2:10that advertising product development
2:13sales all of that was going to be
2:15affected and uh coding was another use
2:19case that that uh and it's it's all
2:21coming true right coding is definitely
2:23one of the most disrupted areas and in
2:26terms of both augmentation and and uh
2:29automation. So digging into into
2:31marketing the one area that I'm I'm
2:33really interested in is is marketing
2:35research and then u areas that happens
2:38to be one of the areas where you know
2:41Genai has seen a bunch of action. So
2:43personalization, content creation,
2:46social listening, all of those have
2:47clearly been affected. But inside
2:49generation also is is has seen a
2:52significant play.
2:55So the the
2:57I've always been very very interested in
2:59in in Jerai for the last three to five
3:02years and and it's uh so I have been
3:06teaching marketing research and and one
3:09of the ways to sort of figure out how
3:10geni is going to sort of intersect with
3:12marketing research was try to write a
3:13paper which is what we did and this was
3:15a paper that got published earlier in
3:18the year and and uh most of what I will
3:21talk about today is bu built upon upon
3:23this work that was uh joint work with
3:26Ishita and Yohi both are at WMadison
3:30Ishta's faculty and Yohi is is a very
3:34bright uh PhD student we have here
3:38so the way I've I've always taught
3:40research um over the last 20 years or so
3:44and most of us have uh also sort of talk
3:47in terms of exploratory research
3:50descriptive and causal right the the so
3:52you could do a depth interview which is
3:54exploratory in nature or you could do a
3:56survey or you know AB tests which are
3:58more descriptive or causal. This tends
4:00to be more qualitative and descriptive
4:03and causal tends to be more more
4:06quantitative.
4:07And once you've sort of figured out what
4:10what type of uh research you want to do
4:12then you get on to the task of designing
4:15the the study sample selection called
4:18data collection data analysis. So one of
4:21the things I'm sure most of you have
4:23seen now is for something like depth
4:26interviews in qual research you can now
4:29very easily generate a discussion guide
4:31with with [clears throat] generative AI
4:33you can ask it for who do I who should I
4:35talk to for to for for my research
4:37questions you can even generate
4:39synthetic respondents which I'll talk a
4:41fair bit about today and certainly you
4:43can have generative AI summarize the
4:46data extract themes and then even
4:48visualize it Um so the whole research
4:51pipeline is is getting fundamentally
4:53disrupted and and um the the key things
4:57for me are research is a getting faster
5:01and b it's getting cheaper.
5:04We can be a lot more efficient with
5:06research today because of of generative
5:08AI. And my hope is that because we can
5:11think do things the wrote things the
5:13more mundane things with the help of
5:15generative AI that researchers can be
5:18more uh impactful in terms of answering
5:21the research question generating
5:22insights that actually matter. So the
5:25focus would be more answering the
5:27questions than than you know writing a
5:30survey for example where generative AI
5:32gives you a fantastic start. So that's
5:34sort of the my first big theme here is
5:36like faster cheaper or for better better
5:39consumer insights.
5:41So going back to the paper that we
5:44wrote, we were interested in studying
5:47both qualitative and quantitative
5:49research and how generative AI has a
5:51play in those. Where does it add value?
5:54And in order to try to see how well
5:56generative AI works uh for both those uh
5:59use cases, we wanted to have a ground
6:01truth. And ground truth in this case was
6:04a quall study and a survey that a
6:07Fortune 500 company had done in 2019.
6:11And so what we were going to try to do
6:13then is use generative AI to replicate
6:16those the study and see what insights do
6:19we do we get from that process of
6:21replication. So that's most of what I
6:24will I will share with you um today. So
6:28the qua research in this case was a
6:31friendsgiving study. So this company has
6:34a big play in Thanksgiving and then so
6:36they also wanted to see for this newer
6:39emerging area where it's like
6:41Thanksgiving but a smaller in scale and
6:44a different kind of target audience. So
6:46you want we they wanted to sort of get a
6:48sense of how that that um would would
6:51look like. So the focus of this study
6:53was a depth interview.
6:56And so what we did was we already had
6:58the human data then we got generated the
7:01LLM data and then we compared the two.
7:05And one of the results that came out of
7:07this comparison is the LLM generated
7:10answers were actually deeper and they
7:14were also more insightful.
7:16And if you're wondering like what what
7:19was the source of of this this
7:21evaluation? These were independent
7:23crowdsource workers who look at uh the
7:27the LLM answers and the human answers
7:29and then give give it give an evaluation
7:32for the themes that were coming out from
7:34from LLM versus the human uh answers and
7:38then what what we found um uh was some
7:41encouraging results right this is this
7:43is like LLM is not just giving you
7:45something useless it's it seems to be
7:47giving you something that is appears to
7:49be meaningful then we followed that up
7:51with um uh in a collaboration with a
7:54company called CNR research. They have
7:56experts who do this qual stuff, right?
7:58They have uh expert moderators, they
8:00have experts analysts. So, we also
8:02engage them to then try to assess um how
8:06well do does the LLM arm do compared to
8:10the human arm of of this experiment that
8:12that that we did. And the main takeaways
8:14were there are unique insights that the
8:18LLM brings to the table and there are
8:20unique insights that the humans bring to
8:21the table. They definitely overlap a
8:24fair bit but there also like uniqueness
8:26to it. So our main story we try to tell
8:28in in the paper is that these LLMs offer
8:33complimentary insights that to to the
8:35what the humans were bringing to the
8:37table. The other big thing which which
8:40I'm pretty pretty high on is is the
8:42ability of LMS to to analyze the data on
8:46the analysis front. LM are just simply
8:49fantastic and then and um that that's
8:52been a pretty pretty powerful use case
8:53for for them.
8:56The other thing we did here was to have
8:59the LLM moderate itself. So you ask LLM
9:03a question, give it a persona for a
9:06person who's 32 years old, lives in
9:07Chicago, is a host for this
9:10Friendsgiving event. We ask it the same
9:12question that we asked of humans. It
9:14gives an answer and then we evaluate
9:16that answer on depth, relevance, depth,
9:18clarity, insightfulness.
9:20And if it was answer was below a
9:23threshold of 80, an arbitrary number we
9:26picked, then we said then we probe a
9:28little bit more. not we, the LLM probes
9:30a little bit and then um so it asks it
9:33elaborate a little bit further. So this
9:36is pretty cool, right? I so the LLM is
9:38actually working harder to give you a
9:41better answer that you are interested in
9:43for the research question that that you
9:45have. This idea of LLM moderated um
9:52interviews is is getting big traction in
9:55the industry. So um because I'm so
9:58interested in this area I've been I've
9:59been tracking bunch of startups and then
10:01who are who are impressive I mean they
10:03really are doing impressive work and I
10:05mentioned couple here outset and next
10:07intelligence um uh do AI at scale it
10:13depth interviews that is AI enabled at
10:16scale that was previously infeasible
10:20right so we were typically doing 20 qual
10:24interviews in service of like a thousand
10:25person person quant study that that was
10:28sort of the norm. Now you can do 100
10:31interviews in two days that would have
10:34previously taken you weeks and would
10:36have taken you a lot more money. Right?
10:38So that's sort of the idea. So companies
10:40like um Vox Popme does that. They have
10:44use cases where they show that the
10:45analysis is much quicker. It's better.
10:49The existing research that you have that
10:51was gathering dust somewhere can now be
10:53analyzed and mined for more information.
10:55So uh all all kinds of interesting stuff
10:58happening in the qual space.
11:01So in the paper we also uh leave the
11:05reader with a road map for how qual
11:07research uh can work in collaboration
11:09with generative AI. I'm not going to go
11:11into details here, but but the sort of
11:14the gray portions are where you see
11:17significant um action. And and it to me
11:21sort of the big takeaway here is because
11:23you can do uh these this this research
11:27faster at scale
11:30and and and much more efficiently that
11:34allows you to sort of cycle through more
11:36studies quicker.
11:38So the sort of knock against the
11:40research typically has been that you
11:42cannot quite deliver on time. Now we
11:45have a sort of a system in place that
11:48allows you to do more frequent research
11:51and hopefully align better with the
11:53decision cycle. So that sort of um is is
11:56exciting.
11:58The other piece that we worked on was
12:01was qualitative research uh I'm sorry
12:03the quantitative research which was
12:05surveys. The context here was a product
12:07that this company ended up launching um
12:10it was a frozen food not frozen
12:13refrigerated food for for for dogs and
12:16then uh before they launched it they did
12:19this study and the study asked pe people
12:21to look at things like um ask
12:24respondents to give the purchase
12:25likelihood the liking and attitudes
12:28towards the product. Um, so very
12:31traditional study that you would do
12:33before you launch a new product. And
12:36this was an online survey. And we took
12:39the data, we then um used the LLM to
12:43create personas for each respondent in
12:46the data and replicated the answers.
12:50And what we were interested in was to
12:53try to find out how the LLM answered
12:55compared with the uh human answers and
13:00then and uh are there ways to improve
13:02upon that. So let me let me walk you
13:05through some of the the the main
13:08results. So on measures like purchase
13:11likelihood, liking and uniqueness um on
13:14a one to five scale
13:18uh where you you have the human is like
13:21the light gray and the LLM is is is the
13:23is is the blackish um bar here. What you
13:28see is both in terms of direction and
13:31magnitude, the LMS do a pretty good job
13:34to characterize the average purchase
13:36likelihood, average liking, average
13:38uniqueness. The LLMs are doing a a
13:40pretty pretty nice job. This is also
13:43true for things like um so these are
13:48seven different attitudal questions that
13:50were asked in survey. So A1 for example
13:53was um uh has this this dog food has
13:59more benefits than other premium dog
14:01food. Um uh A7 the last one for example
14:04was it makes me feel good about what I'm
14:07feeding my dog. So so these are very
14:09typical kinds of attitudal questions
14:11that that that you would you would ask
14:13of um of of you know in a study like
14:17these. And then the again the
14:19directionally we're doing well magnitude
14:22wise we we we're capturing the data
14:24pretty well. The place where LLMs do
14:28poorly is you probably can't see this
14:30but here uh here and here you see that
14:34the these uh confidence intervals are
14:37are are narrow. What that's really the
14:39reason that's happening is LLMs
14:41typically tends to not capture the
14:44distribution of the of the um responses
14:49responses. It typically gives fours and
14:51fives when there were like ones, twos,
14:52and threes in the data. So, it's a
14:54pretty well-known limitation of LLMs is
14:57is that they don't capture diversity
14:59really well. So, what the way we got
15:02around that was by injecting what we
15:05call in context. we don't call it that's
15:07a wellestablished term it's called in
15:08context learning and a tool called rag
15:12um and and by bringing in rag and and
15:17deploying that in this context we got
15:19answers that improved sort of the
15:21heterogeneity in the answers the data
15:23were a little more dispersed and the way
15:26typically rag works is is the basic idea
15:28is the following uh it's you give the
15:32LLM some additional information that may
15:34be relevant that it allows you to answer
15:37questions better. So this is a way to
15:41bring in your proprietary information
15:43that that you may have within your
15:45companies. Uh this could be other
15:47service, it could be other qual research
15:49or related category that you want to
15:52bring into this this this context. And
15:54for us what we used for rag was another
15:57qual study that was also a dog food
16:01related study and that helped the
16:03quality of answers actually actually
16:05improved the answers became more
16:07internally consistent they also became
16:09more and more heterogeneous.
16:11So while a vanilla LLM did not give give
16:16um great answers in terms of
16:18heterogeneity but injecting additional
16:21propri propri proprietary firm specific
16:24information did help.
16:27So this this again happens to be a u
16:32highly promising area for for for
16:35marketing research and I want to mention
16:36two companies here. First is um
16:38Evidenza. So, Evanza is a company in the
16:42synthetic space that is uh doing more
16:45work in the B2B space which you think
16:47about it makes sense in the following
16:50way because B2B
16:53marketing research is difficult. The
16:55respondents are difficult to reach.
16:56They're expensive. So often times what
17:00companies do is they talk to the
17:02salespeople who have been talking to the
17:04customer. So, in some ways, the
17:06salespeople become the voice to the
17:08company. That makes sense. But, but
17:11wouldn't it be nice if we could do
17:13better than that? Wouldn't it be nice if
17:15we could use synthetic respondence,
17:17which is what evidence is is trying to
17:19do to um have the LLM give you uh have
17:23you fill in some information that you
17:25have voids for that you don't know the
17:27answers to?
17:30The other option is not do any research
17:31at all. So, um I I I like that. basic
17:35premise and I think it's promising and
17:37Evidenza has uh done some some of their
17:40own benchmarking against humans and um
17:43they they claim that it does well in one
17:45study that I that I've I've seen but
17:47again this is very early this is like
17:49the first innings of the synthetic play
17:52and we need to do as as academics I
17:54think we are seeing a fair bit of fair
17:56bit of work in that space what we are
17:58trying to do is is um test that idea the
18:01replicability of of Ellens's doing what
18:04the humans
18:04uh do the other one uh I want to briefly
18:08mention is called expected parent very
18:10cool name um so this is um uh work this
18:14is work from MIT and and this is um a
18:19expected parent is launched by a
18:20professor there um and he and his uh
18:23wife they've started this this company
18:25and they have tools that that are that
18:27are established in fact um uh John
18:30Hartford is just like the the the author
18:32for this paper he has a couple of highly
18:34influential papers that I if you get
18:36interested in this area I I I would
18:38recommend that that you read those those
18:39two papers. I've certainly read them and
18:41learned a fair bit from from them and um
18:44uh so yeah fair fair bit of work that's
18:47that's happening in this space
18:50uh not only us not only companies like
18:52Evanza and and and uh and expected
18:55parrot but also um Colombia has been
18:58very active in this space. So Olivier um
19:00and um his his colleagues uh have have
19:04have uh compiled this data set that has
19:08uh 2,000
19:10uh a twin here stands for digital twin.
19:13The 2K stands for 2,000 respondents. 500
19:16stands for 500 um questions. So they
19:20they have very rich
19:23data set for 2,000 people, 500 questions
19:25that they collected over a long period
19:26of time. And the data are out there. So
19:29you could actually use the that data to
19:33come up with your own synthetic twin,
19:35your your own digital twins and see how
19:38well you can come up with answers that
19:40the humans gave. So this is again an
19:43attempt and I I I really admire what
19:46Colombia is doing in this space um for
19:48them to put this out there that so that
19:50that we can progress the field to try to
19:53figure out when does this stuff work and
19:54when does it not work.
19:58So in the paper we also leave a
20:01framework that that is um for the just
20:04like for qual for quant and here again
20:08I'm not going to go through the the sort
20:10of the the each step of this but for uh
20:14different aspects of the survey you can
20:17use the LLMs to really become a strong
20:21collaborator and one of the ways where
20:24where where I think it it can really
20:26helpful is let's say you design a study
20:29that's going to be fielded to a thousand
20:31people. Before you actually do that, it
20:34is fairly straightforward to now
20:36generate synthetic data for those
20:38thousand people. You you know their
20:40demographic characteristics. You kind of
20:42know who they are going to be uh
20:44approximately. So you can generate
20:45synthetic data and look at what kind of
20:48results you are going to get, right? Um
20:51and those results may suggest uh that
20:54some of the questions are not even very
20:56helpful. It may suggest new questions
20:59and at in the extreme case it may say
21:02this is fantastic. What I'm looking at
21:03is giving me all I need to really look
21:07at and I'm done. I don't even need to
21:09take it to the field. So those are
21:11different sort of ways in which uh you
21:13can leverage the the uh LLM for for for
21:18quant research and uh this is sort of
21:22the notion of backward marketing
21:23research that was that that we teach in
21:25in the classroom where you can actually
21:27anticipate the results much better
21:28before you go out and and and and do the
21:31stuff at minimum you can generate prior
21:33information to guide what do you what do
21:36you want to do with it next gather more
21:38data or or ask the question am I on.
21:42So to to wrap things up in terms of all
21:44the things that I've said, I wanted to
21:46sort of leave you with three sort of
21:47main takeaways. First uh as as I said is
21:50this idea of like faster, cheaper
21:53insights, right? Um that my hope is uh
21:57that the sort of bandwidth that creates
22:01is you get to answer questions more
22:05effectively, the research questions more
22:06effectively, the quality of insights
22:08actually improve. So not only is it
22:10efficient, it is also more effective. So
22:13that's sort of the uh at least my my
22:15hope. Um the other thing uh I mentioned
22:18is the digital twins idea, right? I I
22:20feel like we are first innings
22:25still very exploratory, some really
22:27interesting findings from from variety
22:29of different sources, but far from like
22:32an established sort of a play book yet,
22:35but but I think I think we'll we'll get
22:36there. One really promising area that
22:39that um that that I've seen which I
22:42didn't really talk about is is um what's
22:45been called persona bots or persona chat
22:48bots and and this is sort of the idea of
22:51um uh you have let's say segmentation a
22:54big segmentation study that you did
22:56right and you know bunch of demographic
22:59information psychographic information
23:01behaviors of people for this segment and
23:03historically what we have done is done
23:05really worked hard to create that
23:07segment and come alive and then then
23:09hopefully make decisions based upon
23:11that. What you can do today with with
23:13these persona bots is is you can all
23:17those characteristics for this segment
23:18you can put in a a persona for an LLM
23:23and then interact with it. So, if you're
23:26running a brand uh for a a um a Cheerios
23:31or something like that, you can actually
23:34talk to the segment that that that that
23:37um that you're interested in and and and
23:39it's more interactive for that persona.
23:42You can do concept tests. So, for that
23:44given persona, you can actually do like
23:48uh simple AB tests for example, not
23:51having to involve humans at all at this
23:53stage. So it's a brand new way of sort
23:55of thinking about research and then very
23:58creative use of the data that that you
24:00have and marrying sort of the the human
24:02side of it and then the and the LLM side
24:04of it. And and finally as I as I
24:06mentioned I'm very high on on the value
24:09of of qu how what LLM bring to the table
24:13on qualitative research. Qualitative
24:15historically has been um early stage to
24:18guide quant bigger representative and
24:22all that. Now we now live in a world
24:24where you can do qual quant at scale and
24:28you can analyze those data reasonably
24:30quickly and so I'm a big fan of of qual
24:34research because unlike quan where you
24:37have like um you make up scales right
24:40service performance on a scale of 1 to
24:43five what if we were simply to talk to
24:46people get their qualitative information
24:48and then have AI tools scale that back
24:51to one to five scale if if that's what
24:52we wanted to do. So, it's a complete
24:55reorientation of how how how we should
24:57how we should um do research. So,
25:00hopefully this this was uh helpful. Uh
25:03again, you know, the papers out there. I
25:05welcome you to check that out if if uh
25:07if you have uh an interest. Thank you.
25:10And I I welcome questions at this stage.
25:14>> Thanks very much, Raj. I think that was
25:15a very thorough and and very clear
25:18overview of how we can use uh artificial
25:20intelligence in market research. Uh most
25:23of your comments were addressed to
25:25relatively sophisticated users and I
25:27think you had some good cautionary notes
25:29there. Uh our first audience question is
25:31from John and he is uh saying he can see
25:34how uh LLMs could help uh you know where
25:38uh humans have you know cognitive
25:40limitations and processing and so forth.
25:42But are there some unique insights and
25:44areas that uh humans bring to the
25:47insight uh interpretation of qualitative
25:51data? You made a good point I think at
25:52the end for uh adapting qualitative to a
25:56more quant type perspective but what is
25:58maybe still the role of the uh human
26:01insight in using AI assistant
26:04qualitative research?
26:06>> No, I I think that's that's that's an
26:07important question. Um so if you look at
26:10the whole research pipeline at the
26:12earlier stages in terms of problem
26:14definition and the research the business
26:16questions that that that are being asked
26:18I think humans own that right so while
26:20the LM can generate like some discussion
26:23guides and then then and some surveys um
26:26in the end it's it's up to the human to
26:29actually make sure that that those
26:31questions the research questions get
26:33answered the business questions get get
26:35answered and so the role of humans is is
26:39now more um are are using their judgment
26:45like you as a researcher your judgment
26:48becomes crucial right it's it's it's
26:51it's so it's just like we've had tools
26:54like calculators and stuff like that
26:56they they didn't really take away you
26:58know the the the the need for judgment
27:01in in businesses so I I think uh if
27:04anything from a training so we teach
27:06students right so if if anything the one
27:07thing that we want to emphasize here in
27:10the classroom and then you you all who
27:12are business leaders in your your junior
27:14analysts is like we are less interested
27:17in you designing you know discussion
27:19guides or or or or
27:22even the analysis part of it we are more
27:24interested in you answering the business
27:26question the bar in some ways is is
27:28elevated so yeah so that that that's uh
27:31I hope that makes sense
27:34>> yeah I think that makes a lot of sense I
27:35I had a quick observation about your
27:37point that the qual the insights were
27:39deeper from the LLM uh you know
27:41applications. I wonder if part of that
27:44is uh the the classic challenge that we
27:46face in qualitative research of social
27:48desiraability of answers not just in
27:51qualitative but certainly there people
27:53are reluctant sometimes to disclose or
27:55speak you know maybe their deeper
27:57thoughts about a matter so uh it
27:59probably needs to be used with some
28:01caution but I think there's an issue
28:03there where perhaps the large language
28:05models applied to qualitative research
28:07can help tease out insights that might
28:10not have been stated explicitly but were
28:12implicit maybe in what was being shared.
28:15>> Yeah, I think u as far as when you talk
28:19to u the the the suppliers of these
28:21these u you know AI enabled um depth
28:26interviews, one of the things they say
28:27is what they're finding is that there
28:31are things that humans are more likely
28:34to talk to an AI than than they are to
28:37to a human. I don't have like academic
28:40evidence to support that but that's
28:42that's a hypothesis right I think we
28:44could certainly test that idea of of or
28:47for certain context for sure I can kind
28:49of see where human human interactions
28:52may actually be limiting whereas talking
28:54to an AI enables interviews it's it's
28:57it's a bit bit you you get get richer
29:00insights uh but then there's also this
29:03this sort of idea of if you talk to um
29:07let's say you're doing a survey Okay.
29:09Historically, we do like this, you know,
29:11long 15 minute, 30 minute surveys and
29:14then uh these this instruments and then
29:17these survey questions. I think it may
29:20be easier if if you those questions are
29:22asked as open-ended questions, right?
29:24And and and people
29:27talk their preferences. They don't
29:29indicate on one to five scale what their
29:31preferences are, but you can infer those
29:33on a oneto5 scale. So, right. So I think
29:36I think um there's value there.
29:40>> Sounds great. Our managing director
29:42Keith Smith has some questions for you,
29:43I think, as well.
29:44>> Yeah, absolutely. You know, it was great
29:46presentation. Um really found it
29:48interesting. Um and and follows a lot of
29:50sort of the thread that we're seeing in
29:51a lot of academic research and on
29:52industry about how important this
29:54question of synthetic data and LM for
29:56market research is. Um I'm just curious,
29:58you know, one of the the primary use
30:00cases you see a lot of times for market
30:02research is in the um new product space
30:04or in the um sort of new innovation
30:06space. Um and I'm curious, have you had
30:08any experiences or exposure or thoughts
30:10on how these LLM deal with these sort of
30:12out of sample context right where we're
30:14looking at new products or new features
30:16or new ideas? Um and do you have any
30:18sort of guidance or suggestions on how
30:20to how to uh build these types of models
30:22in a way or use these models in a way
30:24that can account for that out of sample
30:26context? Yeah, I think sort of the the
30:28uh the genesis of the sort of potential
30:31concern is LLMs are is backwardlooking.
30:34they are looking on like stuff that's
30:36already they're trained on what's
30:38already there and um so can they think
30:41out of the box kind of I I think that's
30:43how I translated the the the question
30:46and I think there's um some work that's
30:49come out of Ethan Malik has some so I
30:52think I'm not going to characterize it
30:54perfectly but I think he talked to his
30:57students and they generated ideas and
30:59then um the ideas then he compared human
31:01ideas with the LLM ideas and so while
31:04The LLM had some really really bad
31:06ideas. It also had some really really
31:07good ideas too. So I think it's it's
31:10it's uh I think I would not discount it
31:13to sort of say that it cannot do better
31:16than than humans on on idea generation.
31:19But in terms of you know uh
31:23if if I was to sort of try to do this I
31:25would have LM generate ideas if a peril
31:27experts generate ideas pull them and
31:30then have uh LM assisted um evaluation
31:34of those ideas it's like which are the
31:36top three the question I tend to always
31:38ask is uh can the LLM evaluate so like
31:41out of the 50 good idea 50 ideas the top
31:45three evaluation from LLM versus humans
31:48how close are to each other and does
31:51that sort of uh create a situation we
31:54don't even need to ask humans or or at
31:57least bring LLM into the equation of
31:59search for good ideas
32:02>> in terms of sort of so from the idea
32:04generation perspective I think that's
32:05really insightful even that's in terms
32:07of the sort of market evaluation though
32:10concept in other words can we think
32:12about sort of how elements might be used
32:13to evaluate ideas on whether or not
32:15they're even marketable or how much
32:17interest there might be on the consumer
32:18side regarding
32:20Yeah, I think uh certainly that is
32:23typically what we do in research, right?
32:25So a typical study is going to want to
32:27do uh purchase likelihood intent what
32:30fra you going to be try to predict out
32:32what the shares are. You could to
32:33conjure an analysis kind of stuff then
32:35predict out potential uh shares of
32:38those. So um I let Israeli I know you
32:40know she's she has a paper in fact her
32:44work influenced a fair bit of what what
32:45we did on conjoint analysis. So, so that
32:48is kind of forwardlooking in terms of
32:50new new areas. What the shares are going
32:52to be for something some something um
32:55that's let's call it new new or what the
32:57willingness to pay is going to be for
32:59those new new ideas. Um so whatever
33:03those existing tools can do you could
33:06potentially interrogate the LLM to try
33:08to do the same. And uh look the final
33:11thing you know on this I I would say is
33:14I think this this idea of use of LM
33:16needs to be very evidence-based. I think
33:19what that's what we tried to do in our
33:21paper. That's just two studies though. I
33:24mean there's there's I think we need to
33:25really continue to hammer on this and
33:28try to uh figure out edge conditions
33:31where LLMs completely break down and
33:33situations where they shine and then
33:35then have like an informed opinion. So
33:39all big companies like the Europe,
33:41universities of the world, they have a
33:42repository of research that that they
33:45have done. A very simple thing for them
33:47to do would be to basically think of it
33:50as R&D like whatever they did, they have
33:52the human answers. Could I then have
33:55LLMs replicate those and and then
33:58compare the two and that that gives you
34:00sort of internal insights into yeah this
34:03these you know uh pricing it doesn't do
34:06great but concept testing u you know uh
34:10messaging text messaging all those kinds
34:12of ideas of testing does well I think
34:15that's a kind of an R&D kind of a thing
34:17which is worth investigating it's
34:19>> really interesting we had some of the
34:21came through go ahead
34:24Well, I was going to say the the point
34:25about uh reanalyzing data you already
34:27have. I mean, one of the challenges that
34:29a lot of our corporate members face is,
34:32you know, getting resources to do new
34:33research when maybe in many cases the
34:36answer is already in the knowledge
34:38database in some fashion, but it hasn't
34:40been shared or it hasn't been analyzed
34:42in in the kind of depth that we can now
34:44do as you point out at scale with a lot
34:46of qualitative. Uh I had a a quick
34:49question though I think it's related to
34:51Keith and that is if we continue to sort
34:54of fine-tune these models with domain
34:56specific knowledge you know using the
34:58rag and other approaches that you
35:00described are we sort of converging on
35:03the mean once again just sort of
35:04training them to give us the kind of
35:06answers that we found in the past and
35:08does that tend to undercut the use of
35:11these to explore new areas you know the
35:13analogy would be the so-called model
35:15collapse when you have some of the
35:16earlier LLM s were fed their output as
35:19input, they literally at some point just
35:22collapsed and and you know began
35:24generating gibberish. So is there a
35:26danger of just sort of reproducing the
35:28the past as opposed to exploring the
35:30future?
35:32>> Yeah, Roland Rust has a a very nice
35:34paper in JC along the exactly along
35:36those lines. The model collapse idea. I
35:39think he may may they him and his
35:41co-author uh they they may use exactly
35:43the word what what you use this collapse
35:46um
35:46>> idea and u I think it's it's
35:51certainly a plausible thing u but two
35:55things that help here is one newer data
35:59are coming about that these LLMs are
36:01going to ingest right so there's there's
36:04new information that's that's routinely
36:06coming the other is uh firms uh we
36:10should still continue to sort of
36:12aggressively uh research the white
36:14spaces and do research and have that
36:16guide. So in some ways LLM could become
36:20like the synthesis machine. The quality
36:22of answers in the end is really going to
36:24depend upon quality of input and you
36:27control that and then then you know as
36:29when I say you I mean the businesses
36:30control control that the the better the
36:33quality of sort of data going in the
36:36more insightful sort of novel things
36:39that that would the DLM can can spit out
36:41in a somewhat digestible uh manner.
36:47really really interesting. Again, I want
36:49to just thank you very much for your
36:51presentation and for the insight. Um,
36:53you know, this is a topic area that MSI
36:55is going to continue to follow very
36:56closely um over the next uh many months
36:59here. Um, we do have a synthetic data
37:01initiative actually um that will be
37:03kicking off soon. Um, please look to
37:05that information uh for possible again
37:07new insights. And we do have our MSI
37:09forum event which is coming up in
37:11February. Um, if you're interested in
37:13attending, please check our website.
37:14that will be on many of these same AI
37:16disruption uh questions including again
37:19sort of AI from market research. So
37:20Nash, thank you very much. We really
37:21appreciate it. Um and I appreciate your
37:24your research. Um feel free, we will be
37:27um having a number of of additional MSI
37:30webinars starting in the new year. Um so
37:32keep your eye out for that as well. Um
37:34and otherwise uh we look forward to
37:35seeing you all in the future and have a
37:36good holidays.
37:38Well,
37:38>> thank you Keith. Thank you all. Thank
37:40you all. Bye.

About this transcript

This page contains the full transcript of GMT20251209 170003 Recording gallery 3840x2160 by Advertising Research Foundation, generated from the public captions YouTube serves with the video. The transcript has 6,389 words across 898 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.