GCI World 2026 September Session8 During Lecture — Transcript
Full transcript
- 0:02Hello everyone and welcome to this
- 0:04lecture on feature engineering.
- 0:09In this lecture we will cover five
- 0:12topics.
- 0:13First, an overview of feature
- 0:15engineering. What it is and why we need
- 0:18it.
- 0:19Second, transforming numerical
- 0:21variables.
- 0:23Scaling and log transformation.
- 0:26Third, transforming categorical
- 0:29variables.
- 0:30Label encoding, one-hot encoding, and
- 0:33cross features.
- 0:36Fourth, the use of domain knowledge.
- 0:39Turning raw data into meaningful
- 0:41features using expertise about the
- 0:43problem.
- 0:45And fifth, ensembling.
- 0:47It's about improving your model by
- 0:50merging your models and we cover two
- 0:53major methods. Bagging and boosting.
- 0:58Let's start with an overview of feature
- 1:00engineering.
- 1:04Before we dive in, here is the
- 1:07supervised learning workflow we have
- 1:09been using throughout the course.
- 1:11The flow runs left to right.
- 1:15Data collection.
- 1:16EDA and pre-processing.
- 1:19Splitting into training and validation
- 1:21sets.
- 1:22The train evaluate loop with
- 1:24hyperparameter tuning.
- 1:27And finally, evaluation on a test set.
- 1:30After deployment.
- 1:33Today's focus is on what happens before
- 1:36we train the model.
- 1:38Specifically, the pre-processing step
- 1:41where we shape the input variables
- 1:43themselves.
- 1:45Good features make the rest of the
- 1:47pipeline easier.
- 1:49Poor features bottleneck everything
- 1:51downstream.
- 1:54So, what is feature engineering?
- 1:57Feature engineering is the process of
- 2:00selecting, transforming, or creating
- 2:03input variables to improve the
- 2:05performance of a machine learning model.
- 2:10Recall that a supervised model is just a
- 2:13function y equals to f of x.
- 2:16X is the explanatory variables. Y is the
- 2:20target.
- 2:22Feature engineering is everything we do
- 2:25on the x side of that variable before
- 2:28the models use it.
- 2:31A concrete example,
- 2:33sales prediction.
- 2:35The raw data might give us a date and a
- 2:39product ID. Just two columns.
- 2:43The date and product ID are not directly
- 2:47meaningful to the model.
- 2:49A better feature set derived from the
- 2:51same raw data is
- 2:53the day of week, holiday or not, season,
- 2:58and product category.
- 3:01Each one is a richer, more directly
- 3:04predictive signal than the raw timestamp
- 3:06or ID.
- 3:09Feature engineering is a concept of
- 3:11making a machine learning model with
- 3:14such features that seem more related to
- 3:16the target variable to build a better
- 3:18model.
- 3:22Why is feature engineering so important?
- 3:25Three reasons.
- 3:27To boost model performance,
- 3:29to prevent overfitting,
- 3:32and to make your model more
- 3:33interpretable.
- 3:37One important caveat,
- 3:39feature engineering is fundamentally
- 3:42trial and error.
- 3:44There is no single recipe that works for
- 3:47every data set.
- 3:49In this lecture, we will cover the
- 3:52general techniques you can try across
- 3:54different problems.
- 3:58Feature engineering breaks down into
- 4:00four main types.
- 4:03The first is transforming features,
- 4:06reshaping existing variables so the
- 4:08model can use them better.
- 4:13Transforming includes scaling, log
- 4:16transformation,
- 4:18and encoding categorical variables into
- 4:21numbers.
- 4:23The second activity is creating
- 4:26features,
- 4:27building new variables from existing
- 4:30ones, either by crossing them or by
- 4:33using domain knowledge.
- 4:37There are two more types, extracting
- 4:40features,
- 4:42compressing a large set in a smaller one
- 4:45like PCA,
- 4:47and selecting features, removing columns
- 4:51where the input dimension is too large
- 4:53relative to the data.
- 4:55These are also part of feature
- 4:57engineering.
- 5:00For this lecture,
- 5:01our focus in the highlighted pair,
- 5:04transforming features, and creating
- 5:06features.
- 5:09Next, transforming numerical variables.
- 5:14Before diving into concrete transforming
- 5:17method, let us explain about what types
- 5:20of variables exist in the first place.
- 5:23You need to understand what kind of
- 5:25input variables exist to apply the right
- 5:28transformation that depends on the
- 5:30variable type.
- 5:32There are two broad categories.
- 5:35Quantitative data,
- 5:37also called numerical variables,
- 5:40and qualitative data, also called
- 5:43categorical variables.
- 5:47Within quantitative data, the ratio
- 5:50scale is the most informative.
- 5:53Differences have meaning, and there is a
- 5:55true zero.
- 5:58Body weight, 50 kilos or 70 kilos, and
- 6:02distance, 2 km or 200 km, are ratio
- 6:07scale examples.
- 6:09Saying twice as heavy is meaningful
- 6:12because zero is a fixed reference.
- 6:16The interval scale also has meaningful
- 6:19differences but no true zero.
- 6:24Temperature in Celsius, 50°, 30°,
- 6:28and year,
- 6:292000 or 2025,
- 6:33are interval scale.
- 6:35The gap between values is meaningful,
- 6:38but twice as much is not.
- 6:41You can't say, for example, 30° C
- 6:45is two times hotter than 15° C.
- 6:50Within qualitative data, the ordinal
- 6:53scale has a meaningful order, like
- 6:56academic grades, A, B, C, or survey
- 6:59responses, good or bad.
- 7:03The nominal scale has only distinct
- 7:06categories with no natural order.
- 7:09Nationality, Japanese, Chinese,
- 7:12or product ID.
- 7:16This four-way split, ratio, interval,
- 7:20ordinal, nominal,
- 7:22is what tells us which transformation
- 7:25makes sense
- 7:26for which column.
- 7:30Now, let's look at the transformations
- 7:32themselves, starting with numerical
- 7:34variables.
- 7:37The first transformation is scaling.
- 7:40When features differ greatly in scale,
- 7:43the larger scaled feature can have an
- 7:46undue influence on the model.
- 7:49For example, if one feature is income in
- 7:52dollars, ranging up to millions, and
- 7:55another is age, 0 to 100,
- 7:58a distance-based model will be dominated
- 8:01by income.
- 8:04The first scaling method is
- 8:06standardization.
- 8:08Rescale so the mean is zero and the
- 8:11standard deviation is one.
- 8:13The formula is X minus its mean
- 8:17divided by its standard deviation.
- 8:20The histogram on the right shows that
- 8:22the shape of the distribution is
- 8:24preserved. Only its location and width
- 8:28change.
- 8:30The second method is normalization.
- 8:34Also called min-max scaling.
- 8:37Rescale so the minimum is zero and the
- 8:40maximum is one.
- 8:42The formula
- 8:44is X minus the minimum of X
- 8:46divided by the maximum of X minus the
- 8:49minimum of X.
- 8:51Same idea, different reference points.
- 8:56The next transformation is log
- 8:59transformation.
- 9:01The log compresses large values and
- 9:04stretches small ones
- 9:06which makes it easier for the model to
- 9:08see relationship across orders of
- 9:10magnitude.
- 9:12Log transformation is especially useful
- 9:16when the distribution is heavily skewed.
- 9:19Many small values and a few very large
- 9:22ones like income.
- 9:25One important caveat.
- 9:27Log is not defined for zero or negative
- 9:31values.
- 9:32So you either apply it only to strictly
- 9:34positive quantities
- 9:36or you can use log of 1 plus X for
- 9:39example
- 9:40when the data contains zeros.
- 9:45In Python
- 9:47from sklearn.preprocessing
- 9:50import StandardScaler
- 9:52for standardization
- 9:55and MinMaxScaler
- 9:56for normalization.
- 9:59Both have fit transformed.
- 10:01For log transformation
- 10:03just call np.logx.
- 10:06That's all there is to the
- 10:08implementation.
- 10:12One subtle, but important point,
- 10:15scaling has almost no effect on
- 10:18tree-based models.
- 10:20The reason is that trees split on the
- 10:23rank order of feature values, not the
- 10:26absolute scale.
- 10:28Scaling doesn't change the magnitude
- 10:30relationship between the two values.
- 10:34Tree-based methods, such as decision
- 10:36trees, give us essentially identical
- 10:39results before and after
- 10:41standardization.
- 10:43Scaling still matters, though, for
- 10:45methods including regression models.
- 10:50Now, let's from the slides to the
- 10:52notebook.
- 10:54Please run the cell up to section two,
- 10:56transforming numerical variables.
- 11:06The first code cell in the section 2.1
- 11:09demonstrates standardization on the Iris
- 11:12data set.
- 11:16We import load_iris and StandardScaler
- 11:19from scikit-learn, load the data, and
- 11:22pull out the single column, sepal
- 11:24length.
- 11:25Then we create a StandardScaler and call
- 11:27fit_transform, and add the result back
- 11:30as a new column called sepal length STD.
- 11:40The describe output at the bottom
- 11:42confirms
- 11:43the standardized column has mean
- 11:45approximately zero and standard
- 11:47deviation approximately one, exactly as
- 11:51expected.
- 11:56The next code cell runs the comparison
- 11:58experiment we previewed on the slides.
- 12:02We set up four models in a dictionary,
- 12:04plain logistic regression,
- 12:07logistic regression wrapped in a
- 12:09pipeline with StandardScaler, a plain
- 12:12decision tree,
- 12:13and a decision tree with the same scalar
- 12:15in front.
- 12:17We fit each one and record the train and
- 12:19validation accuracy.
- 12:26The alpha table makes the lesson
- 12:28visible.
- 12:29Logistic regression score changes
- 12:31meaningfully when we add a standard
- 12:33scalar.
- 12:35It is sensible to scale.
- 12:37The decision tree score barely moves
- 12:40whether we standardize or not because
- 12:42the tree splits on rank, not magnitude.
- 12:50Section 2.2 moves to log transformation
- 12:53on the wine data set.
- 12:56We load the wine data and pick two
- 12:58heavily skewed columns.
- 13:00Color intensity and malic acid.
- 13:04The first histograms show the raw
- 13:06distributions.
- 13:10The second histograms show them after
- 13:13applying np.log.
- 13:16The skew is dramatically reduced. Both
- 13:19distributions become much closer to
- 13:22symmetric.
- 13:24Does that actually help downstream
- 13:26classification?
- 13:27The next code cell answers that question
- 13:29with a five-fold cross-validation.
- 13:34We train a logistic regression
- 13:37on the two columns,
- 13:38once with the raw values and once with
- 13:41the log transformed values,
- 13:43scoring with weighted F1.
- 13:48The output principal scores and the log
- 13:50transformed version is noticeably
- 13:52higher,
- 13:54confirming that for linear models,
- 13:56taming skewness has a real impact on
- 13:58performance.
- 14:02Practice question one in section asks
- 14:05you to repeat this kind of comparison on
- 14:06the California housing data with ridge
- 14:09and a decision tree with and without
- 14:11standardization and log.
- 14:37Okay.
- 14:38Then uh we will begin the interactive
- 14:43hands-on exercise. Please open students
- 14:46guide and open lecture materials uh from
- 14:50Google Drive and navigate to today's
- 14:52lecture. And you should uh
- 14:56in
- 14:57in the our lectures, we will use Google
- 15:00Colab and for the hands-on exercise.
- 15:06Uh please open the notebook lecture
- 15:08eight notebook. Let's go through it
- 15:10together.
- 15:28Okay. In the practice uh
- 15:34question one
- 15:37we will uh we will first split the data
- 15:41into a train set and a test set.
- 16:18Sorry, I will Maybe I lost some
- 16:23import things.
- 16:26Let me import everything.
- 16:44Okay. It's quite simple.
- 16:48And then we will train two models on the
- 16:51raw raw features and record their
- 16:54uh test squared R. The first model is
- 16:58ridge.
- 17:31Okay, and then
- 17:33I'll add the print here.
- 17:35So,
- 17:38uh
- 17:39Sorry.
- 17:47Uh so, ridge draws one straight line
- 17:49through the data. The second model is a
- 17:53decision tree.
- 17:55And maybe
- 18:08Okay.
- 18:10Here's the result. Uh a decision tree
- 18:12just asks a few yes or no questions. Uh
- 18:16the squared R is the gap between the
- 18:18algorithm and the random guess.
- 18:21Um so, if we get
- 18:23uh one, then that means our algorithm is
- 18:26perfect. And if we get zero, uh that
- 18:29means we are no better than guessing the
- 18:31average. Here are the results. The ridge
- 18:35gets uh 0. uh 0.
- 18:39594 and the the tree gets 0.618.
- 18:45Now, I'll paste some plotting code the
- 18:48below to uh enhance your intuitive
- 18:52understanding. You don't need to
- 18:54understand the plotting code. It's just
- 18:56here to help you see what the models are
- 18:58doing.
- 18:59>> [snorts]
- 19:00>> Okay.
- 19:09Okay.
- 19:10Okay.
- 19:16Here,
- 19:17uh you can see each dot is means one
- 19:21house. The true price are on the X X
- 19:25axis, predicted price on the Y axis. A
- 19:28perfect model puts every dot on the dash
- 19:31line. Ridge gives a very smooth cloud,
- 19:34but it's even predicts negative prices.
- 19:37The tree looks different. It starts from
- 19:40flat lines. That's because the tree
- 19:43sorts houses into small groups with its
- 19:45yes or no questions, and every house in
- 19:48the group gets the same price. Our tree
- 19:51has uh the the the the depth of our tree
- 19:55is six, so our tree has uh two power six
- 20:00equal to 64 groups, so it can only
- 20:03predict 64 different prices.
- 20:11Um
- 20:13Next, I'll apply standard scaling to the
- 20:16features.
- 20:17For your intuiti- intuiti- uh intuition,
- 20:21standardizing is just changing the units
- 20:23like centimeters versus inches. Now look
- 20:27at Now I will
- 20:30write the code.
- 20:38And the apply the
- 20:41standardize into ridge.
- 20:50Yes, here's the result.
- 20:53Then apply it into tree.
- 21:09Yes, uh so now you can look at the
- 21:13output. Um
- 21:15no
- 21:16no score change. Uh why didn't the
- 21:19tree's result uh result change as we uh
- 21:22taught as we told in the um
- 21:26video, a decision tree only cares about
- 21:28the rank order of values. Uh scaling
- 21:31changed the numbers but never changed
- 21:33the order. So the same houses end up in
- 21:36the same groups with the same prices.
- 21:38The why not ridge? Actually, a linear
- 21:41model just changed its weights to absorb
- 21:45the new units. The only part that cares
- 21:47about units is the penalty on the
- 21:50weights, but our penalty is quite small.
- 21:53You can see
- 21:54in the in the definition of the ridge,
- 21:57we set the alpha into one, so scaling
- 22:00makes no difference here.
- 22:03Uh
- 22:04and I will
- 22:07import a table below so you can get um
- 22:12understanding of why these two models
- 22:16didn't didn't change.
- 22:21Okay.
- 22:25Um
- 22:26you can see this in the table below if
- 22:30you
- 22:31multiply the raw coefficient at the
- 22:34[clears throat] features STD, you get
- 22:37almost exactly the scaled coefficient.
- 22:41This is the kind of like evidence that
- 22:44the weights absorb the effect of
- 22:47standard scaling in the ridge model.
- 22:50Uh in the tree plot uh the dash line is
- 22:54the tree's first split. Scaling didn't
- 22:57change the order of the values, so the
- 22:59houses on the left side and the right
- 23:01side of the line stay exactly the same.
- 23:12Then last questions.
- 23:17Uh let's take a quick look at the log
- 23:20transformation.
- 23:22I'll complete this code part.
- 23:48Okay.
- 23:51You can see the output. The ridge uh
- 23:55goes up, but the tree stays the same.
- 23:58Why does ridge get better?
- 24:01Uh I will draw a plot for you to
- 24:05understand why.
- 24:14Okay.
- 24:15Mm.
- 24:19Uh again, you [clears throat] don't need
- 24:21to understand the plotting code. Uh
- 24:24first in the top row for population and
- 24:27the
- 24:28average occup- oc- occupancy. Most
- 24:31houses
- 24:32Oh, sorry.
- 24:34Most houses are in one
- 24:38bar and a few houses have very, very big
- 24:41values.
- 24:42Uh as I mentioned before, ridge draws
- 24:45one straight line and this few big
- 24:47values pull the line toward them.
- 24:51Uh the log makes big values smaller, so
- 24:54the bottom row looks uh much more even.
- 24:57Now the line fits most of houses better
- 25:00and the line, not the tree, the log,
- 25:02just like scaling, doesn't change the
- 25:04order, so the tree makes the same groups
- 25:07again.
- 25:09Okay, then. Let's go back to the video
- 25:12part.
- 25:27>> Next, transforming categorical
- 25:29variables.
- 25:32The first method is label encoding.
- 25:35Assign each distinct category a unique
- 25:38integer.
- 25:39For example,
- 25:41red for one,
- 25:43green for two,
- 25:45blue for three.
- 25:48So, our color column of red, green,
- 25:51blue, green, green, red
- 25:55becomes 1 2 3 2 2 1.
- 26:03Label encoding is simple and works fine
- 26:06with tree-based models.
- 26:08A tree can split between any pair of
- 26:11integers.
- 26:14The catch is that it introduces a fake
- 26:17ordering.
- 26:18The model might think that blue with
- 26:21value three is somehow greater than red
- 26:24with value one.
- 26:26For linear or distance-based models, the
- 26:29fake order can mislead the model.
- 26:36The second method is one-hot encoding.
- 26:40Create a separate column for each
- 26:42category and assign one to the column
- 26:45that matches.
- 26:48The same color column becomes three new
- 26:50columns.
- 26:52Color red, color green,
- 26:54and color blue.
- 26:57Row by row, the matching column gets a
- 27:00one and the rest gets zero.
- 27:04So, red becomes 1 0 0.
- 27:09Green becomes 0 1 0,
- 27:12and blue becomes 0 0 1,
- 27:14and so on.
- 27:17The full encoded table has one one per
- 27:19row and zeros everywhere else.
- 27:25One-hot encoding's merit is that it
- 27:28removes the fake numerical relationship.
- 27:31Every category is treated
- 27:32asymmetrically.
- 27:35The demerit is high memory consumption.
- 27:39For a high cardinality variable, like
- 27:41product IDs with thousands of distinct
- 27:44values,
- 27:45you get one column per category, which
- 27:48can blow up memory and lead to
- 27:50overfitting.
- 27:54The next idea is cross features, also
- 27:58called interaction features.
- 28:00A new feature built from a product or
- 28:03other combination
- 28:05of two or more existing features.
- 28:09For example,
- 28:11A, B, C
- 28:14can be transformed into
- 28:16A * B,
- 28:18A * C,
- 28:20B * C,
- 28:22and A * B * C,
- 28:25and so on.
- 28:27A classic case is rent prediction.
- 28:30Rent is high when room size is large and
- 28:35distance to station is small.
- 28:38The model needs the product of these
- 28:40two, which a simple model cannot easily
- 28:43learn on its own.
- 28:45On the left, there is a 2D scatter of
- 28:48two normalized leaf features.
- 28:51The two classes occupy opposite corners.
- 28:54Leaf length and width is alone cannot
- 28:57separate them with a single line.
- 29:00But,
- 29:01if we add a new column, length times
- 29:03width,
- 29:04and look at the single column instead,
- 29:07the picture changes.
- 29:10The product column separate the two
- 29:12classes cleanly with a single threshold.
- 29:16The red horizontal line shows that
- 29:18everything above some cutoff is class
- 29:21zero and everything below is class one.
- 29:25A linear model can now solve the problem
- 29:27trivially.
- 29:31Cross features capture interactions that
- 29:33simple models cannot learn on their own,
- 29:36which is their merit.
- 29:38The demerit
- 29:39is that the number of possible cross
- 29:41features grows combinatorially,
- 29:43so they cause memory and can lead to
- 29:45overfitting if you make too many.
- 29:51In Python, from scikit-learn
- 29:53preprocessing,
- 29:55import LabelEncoder
- 29:57and OneHotEncoder.
- 29:59LabelEncoder
- 30:01.fit_transform(df.city)
- 30:05gives you the label encoded column.
- 30:08And pd.get_dummies
- 30:12df.columns.city.device
- 30:14gives you the one hot encoded version.
- 30:19For cross features, you can string
- 30:22concatenate two categorical columns into
- 30:24a new column.
- 30:26df.city
- 30:28+
- 30:29"_"
- 30:31+
- 30:32df.device
- 30:34and then one hot encode that combined
- 30:36column.
- 30:38That is the recipe we will use in the
- 30:40notebook.
- 30:44Let's see the notebook implementations.
- 30:47Please scroll to section three.
- 30:50The notebook uses the Titanic data set
- 30:52where each row is a passenger
- 30:55and the target is whether they survived.
- 30:58The categorical column we will encode is
- 31:00embarked.
- 31:01Port of embarkation.
- 31:04C
- 31:05S
- 31:06and Q.
- 31:12The first code cell of this section
- 31:14loads the Titanic data directly from a
- 31:16public mirror with pd. {dot} read_csv
- 31:20and pulls out the embark column with
- 31:23missing values filled in
- 31:25as S.
- 31:28The next code cell applies label encoder
- 31:31to that column.
- 31:32Each of S C
- 31:35Q
- 31:36gets a unique integer.
- 31:42The side-by-side data frame shows the
- 31:44mapping.
- 31:48Then for one-hot encoding
- 31:51we just call
- 31:53pd.get_dummies
- 31:56on the same column.
- 31:58That produces three boolean columns
- 32:01one per port.
- 32:03We cast them to integers with
- 32:07astype int
- 32:10because many models do not accept true
- 32:13false.
- 32:15The first 10 rows confirm
- 32:18each row has exactly one one.
- 32:26Section 3.3 moves to cross features.
- 32:29We pick five numeric Titanic columns.
- 32:36Age, fare
- 32:38sibsp
- 32:40parch
- 32:41and pclass.
- 32:43And fill ages missing values with a
- 32:45median.
- 32:48The next code cell uses polynomial
- 32:50features
- 32:52to build all pairwise products
- 32:53automatically.
- 32:55We get 15 columns total.
- 33:03The practice question two asks you to
- 33:05combine the sex and embark columns into
- 33:08a single string label like female under
- 33:11bar S or male under bar C. And then look
- 33:14at the survival rate per group.
- 33:17Next, the use of the main knowledge.
- 33:25>> Okay, let's go back to the notebook.
- 33:33Okay, in the practice question two, we
- 33:36will now apply label encoding and
- 33:39one-hot encoding to the
- 33:41Titanic dataset.
- 33:44First, I will um
- 33:47uh
- 33:51First, I will
- 33:53encode both labels.
- 34:41Okay,
- 34:43here we work with the categorical
- 34:46variables. Embarked is the port where
- 34:49each passenger got on the ship. As C or
- 34:52Q, a model can't read letters letters.
- 34:55So, we have to turn them into numbers.
- 34:58This is called encoding.
- 35:00There are two ways. First, label
- 35:02encoding gives each port a number. C is
- 35:05zero, Q is one, S is two. One-hot
- 35:09encoding makes one column per port. One
- 35:12in the column of the passenger's port
- 35:14and zero in the others. We will print uh
- 35:18both
- 35:19both results together after uh part
- 35:22three.
- 35:23Uh which one should we use for logistic
- 35:26uh regression? Use one-hot. With label
- 35:28encoding, the model thinks S is bigger
- 35:31than Q and Q is bigger than C. But,
- 35:34ports have no order. For a decision
- 35:36tree, label encoding is fine. The tree
- 35:39just asked question like, is the number
- 35:42uh less than one? So, it can still
- 35:45separate the ports.
- 35:47And then, I will apply one-hot encoding
- 35:50to the sex column.
- 36:16Uh it has only two values, female and
- 36:19male. So, we get two columns, sex female
- 36:22and sex male. Notice that these two
- 36:24columns always add up to one because one
- 36:28person is either female or male.
- 36:31Uh so, if you know one, you already know
- 36:34the other. So, in practice, you can keep
- 36:36just one of them. For example, with uh
- 36:39drop_first equal to true.
- 36:43In the third part, we will create a
- 36:47cross feature
- 36:49by combining sex and embarked into a
- 36:52single stream label.
- 36:58Sorry.
- 37:16Okay, you can see now we make a cross
- 37:20feature. We simply glue sex and impart
- 37:24in embarked together into one label like
- 37:30you can see like
- 37:33sex
- 37:35You can see like female C or male
- 37:39male S. You can see the sex embarked
- 37:41column.
- 37:44Then we one hot encoded, so we get
- 37:47six columns. You can see them in the
- 37:50printout. Yes, you can see the printout
- 37:54in this line.
- 38:00Why do this?
- 38:01A simple model like logistic regression
- 38:04looks at each column separately. Like
- 38:07being female adds this much, port C adds
- 38:10this much. It can't see it it can't say
- 38:13female and port C is special. The cross
- 38:15feature gives every combination its own
- 38:18column, so the model can learn that
- 38:21directly.
- 38:28Now let's check the survival rate for
- 38:33each group. You can simply run the code
- 38:36in the part four.
- 38:38The highest is female C about
- 38:4188%
- 38:43survived. The lowest is male Q only
- 38:45about 7%. That's gap of almost like
- 38:49uh
- 38:5080%. Compare this with one column along
- 38:54by sex along, the gap is only mm 55% by
- 38:58impact along, it's only 22
- 39:01percent. So, the cross feature shows a
- 39:03much bigger difference than either
- 39:05column by itself.
- 39:07Uh and here's the interesting part. Look
- 39:09at port Q for women. Q is quite good, uh
- 39:1375% for men, Q is the worst,
- 39:17uh 7%. So, the effect of the port
- 39:19depends on sex.
- 39:21Neither column along can show this, only
- 39:24the combination can. So, that's why the
- 39:27combination
- 39:28label is very
- 39:30important.
- 39:31Okay, then let's go back to the video.
- 39:45>> Next, the use of the main knowledge.
- 39:49Domain knowledge means expertise in the
- 39:52specific problem area.
- 39:55It helps us transform raw data into
- 39:58meaningful features that are more
- 39:59directly related to the prediction
- 40:02target.
- 40:03Take rent prediction as an example.
- 40:06The raw data might give us only address,
- 40:09size, and age of the building.
- 40:12Some of these may not be directly
- 40:15meaningful to a model.
- 40:18But if we know real estate,
- 40:21we know what actually drives rent.
- 40:25We can use the knowledge to compute new
- 40:27features from the raw address.
- 40:29Distance to the nearest station and
- 40:32distance to the nearest school.
- 40:34Those are far more predictive than the
- 40:36raw string address.
- 40:39Here's the same idea in the table.
- 40:42For real estate task,
- 40:44predicting house prices or rent,
- 40:47the raw data is address, size, and age.
- 40:52The domain knowledge feature we will
- 40:53build are distance to the nearest
- 40:56station,
- 40:57price per square meter,
- 40:59and school district.
- 41:03For a soccer task, predicting whether a
- 41:05shot becomes a goal,
- 41:07the raw data is ball position and player
- 41:10positions.
- 41:12The domain knowledge tells us the useful
- 41:14features are distance to goal, shooting
- 41:17angle, and the number of defenders
- 41:19between the shooter and the goal.
- 41:23And for e-commerce,
- 41:25predicting whether a user buys a
- 41:27product, the raw data is browsing
- 41:30history and purchase history.
- 41:32Domain knowledge gives us features like
- 41:34number of views in the past week,
- 41:37and time since the last purchase.
- 41:40In every case, the raw data is the same,
- 41:43but the engineering features are
- 41:45tailored to the physics
- 41:47of the problem.
- 41:50Let's see domain knowledge applied to
- 41:52Titanic.
- 41:54Please scroll to section four.
- 41:56We will use three domain knowledge
- 41:58constructions,
- 42:00extracting passenger titles from names,
- 42:03combining sex and class,
- 42:05and collapsing family columns into
- 42:07single family size.
- 42:12Section 4.1 extracts the title hidden
- 42:15inside the name column.
- 42:17The first goal cell extracts the
- 42:19character strings that match Mr., Mrs.,
- 42:23Miss,
- 42:24Master, and so on.
- 42:26We apply it to every name and store the
- 42:29results
- 42:30in a new title column.
- 42:33DF_title.unique
- 42:35show the full set we extracted.
- 42:42Some unfamiliar titles will appear.
- 42:45Doctor,
- 42:47Reverend,
- 42:49Colonel,
- 42:51Mademoiselle,
- 42:54Madame, and others.
- 42:56The value count cell
- 42:58shows how common each one is.
- 43:04Most of them appear fewer than 10 times,
- 43:06so we collapse them in a single rare
- 43:09category and merge a few variants.
- 43:17Mademoiselle and Miss become Miss, and
- 43:20Madame becomes Mrs.
- 43:23The clean up is just a few replace
- 43:25calls.
- 43:30We then one-hot encode the clean title
- 43:32column with pd.get_dummies.
- 43:36Cast to integers
- 43:38and concatenate the new columns back in
- 43:41the data frame.
- 43:44The final df.head confirms that columns
- 43:48like title_ under_Mr, title_under_Miss,
- 43:53title_under_Mister,
- 43:54title_under_Misses,
- 43:56and title_rare are now ready to feed to
- 43:59a classifier.
- 44:04Section 4.2 combines sex_pclass into a
- 44:07single label.
- 44:09The motivation is the women and children
- 44:11first evacuation rule, which was applied
- 44:14most consistently in the first class
- 44:16cabins.
- 44:21The code creates a new column sex_pclass
- 44:26and then groups by that label to show
- 44:28the survival rate per group.
- 44:31The output is striking.
- 44:34Female one, first class women, survive
- 44:38at about 97%.
- 44:40Male under_3, third class men, survive
- 44:45at about 14%.
- 44:47Neither sex nor pclass alone shows
- 44:50anywhere near that spread.
- 44:55Practice question three ask you to build
- 44:58a family size features.
- 45:00SibSp plus Parch plus one.
- 45:04Pause and try it.
- 45:12>> Okay, let's go back to our notebook to
- 45:17complete the
- 45:19uh practice uh practice question three.
- 45:23Mhm.
- 45:27Practice question three is about uh
- 45:29domain knowledge.
- 45:31Uh
- 45:32SibSp counts siblings and Parch counts
- 45:36parents and children.
- 45:39So, the family size equal to
- 45:44Sib
- 45:47SibSp
- 45:49plus Parch.
- 45:54plus one.
- 45:57And is alone
- 46:00equal to
- 46:02when the family size
- 46:11equal to one.
- 46:16Okay.
- 46:20Uh
- 46:24So, in the
- 46:26in
- 46:27we
- 46:28uh when we want to predict the
- 46:31survival rate
- 46:33uh during the evacua- evacuation, what
- 46:36prob- what probably mattered was very
- 46:39simple, like how big is your family and
- 46:42can you stay together? So, we uh add
- 46:45them up plus one for the passenger to
- 46:47get family size. And is alone is just a
- 46:51one if family size is one. It's pretty
- 46:53easy.
- 47:03>> Then we will group by group by family
- 47:06size and compute the mean of survived
- 47:09per group and
- 47:12plot the results.
- 47:18Okay.
- 47:21Just
- 47:24SR equal to
- 47:30group
- 47:32I
- 47:34family size
- 47:39survived
- 47:42mean value.
- 47:44Then I will copy the
- 47:47visualization code.
- 47:58Okay.
- 48:00Uh now here's the survival rate for each
- 48:04family size.
- 48:05Mhm.
- 48:09The you can see in the plot the dash
- 48:11line is the overall survival rate about
- 48:15um
- 48:15uh
- 48:1740%.
- 48:19Mhm.
- 48:21But it's not a simple bigger is better
- 48:23line. There's a sweet spot. People
- 48:27traveling alone only about 30% survived.
- 48:31Families of two to four did much better,
- 48:3455 to 72%. The families of five or more
- 48:40drop sharply and nobody in the families
- 48:42of eight and 11 survived.
- 48:50And uh
- 48:54finally, does it really help a model?
- 48:59Let me complete the code here.
- 49:28Okay.
- 49:30So
- 49:32here we train logistic regression with P
- 49:35class, sex, age and fair and then add
- 49:38family size and is alone.
- 49:41Accuracy goes from
- 49:4378.7%
- 49:45to 79.4%
- 49:48a small gain for from just two extra
- 49:51columns.
- 49:57So here's one
- 50:00there's what here's one note.
- 50:02Actually logistic regression can only
- 50:05learn bigger is better or bigger is
- 50:08worse. It can't learn a sweet spot by
- 50:11itself. That's why is alone helps and a
- 50:14decision tree
- 50:16could catch the sweet spot directly.
- 50:21Okay, let's go back to the video.
- 50:36>> Next
- 50:38assembling.
- 50:41Ensemble learnings is a method to
- 50:43improve your models performance by
- 50:45combining multiple models.
- 50:50The basic idea
- 50:52prepare multiple models then aggregate
- 50:55the predictions.
- 50:56By majority vote for classification or
- 51:00by averaging for regression.
- 51:03The diagram on the right show the
- 51:05pattern.
- 51:06Input goes into N classifiers C1, C2,
- 51:11and CN.
- 51:13Each produces an output O1, O2,
- 51:18ON.
- 51:19We combine them into final output.
- 51:23The majority tends to beat any single
- 51:25model because the individual mistakes
- 51:28cancel out.
- 51:31Two main families dominate practice,
- 51:34bagging
- 51:35and boosting.
- 51:37We will cover both in this section.
- 51:42Bagging stands for
- 51:44bootstrapping aggregating.
- 51:46Step one, take the training data.
- 51:50Step two, generate many bootstrap
- 51:53resamples of it.
- 51:55A bootstrap resample is a draw of N
- 51:58samples from the original training set
- 52:01with replacement.
- 52:03So, duplicates are allowed.
- 52:06Repeat that B times to get B different
- 52:09resample data sets.
- 52:12Each resample data set is slightly
- 52:14different from the others
- 52:16because the random draw picks different
- 52:18rows.
- 52:20Step three,
- 52:22train a one separate model on each
- 52:24resample.
- 52:25We end up with B trained models.
- 52:30Each individual model is a little weaker
- 52:34because it only saw noisy subset of the
- 52:36data.
- 52:37But the combined model aggregated by
- 52:40vote or average is often much stronger.
- 52:46That is bagging in five steps. Resample,
- 52:49train, repeat, aggregate.
- 52:55Why does bagging work? The intuition is
- 52:59the wisdom of the crowd.
- 53:01A graph on the right plots majority vote
- 53:04accuracy
- 53:05against the number of independent voters
- 53:08each with 60% accuracy.
- 53:13A single expert at 60% accuracy gives,
- 53:17unsurprisingly,
- 53:1860% overall accuracy.
- 53:22Five independent voters, each at 60%,
- 53:26push the majority vote accuracy up to
- 53:28about 68%.
- 53:3211 voters reach out 75%.
- 53:36101 voters reach about 98.
- 53:40100 imperfect models can outperform a
- 53:43single strong one
- 53:46if their mistakes are diverse.
- 53:50The big caveat,
- 53:52if every voter makes the same mistake,
- 53:55voting does not help at all.
- 53:58The diversity of mistakes is what makes
- 54:00the math work.
- 54:05A famous example of bagging is a random
- 54:08forest.
- 54:09Bagging is applied to decision trees,
- 54:12but with one extra twist.
- 54:15Without it, every tree would tend to
- 54:17look at the same.
- 54:19A single strong feature would always be
- 54:22picked for the first split.
- 54:27The extra twist is that at each split,
- 54:30only a random subset of M features is
- 54:33considered as candidates.
- 54:36That forces the tree to look different
- 54:38from each other.
- 54:39We train many such trees and take the
- 54:42average.
- 54:43That is a random forest.
- 54:46Let's see bagging in action.
- 54:48Please scroll to section 5.2.
- 54:51The first code cell applies bagging
- 54:57to K nearest neighbors on the breast
- 54:59cancer data set.
- 55:07We set up two models, a plain K neighbor
- 55:10classifier
- 55:12and bagging classifier. That is, 100 K
- 55:15and models trained on 100 different
- 55:18bootstrap resamples
- 55:20with their predictions aggregated.
- 55:22We fit both on the training set and
- 55:24record training and test scores.
- 55:31The output table shows that the training
- 55:33score barely changes, but the test score
- 55:36improves with bagging.
- 55:38The gap improvement on unseen data with
- 55:41no change on training data is exactly
- 55:44the variance reduction we expect from
- 55:46averaging.
- 55:49Practice question four in this section
- 55:51asks you to try the same thing with
- 55:53logistic regression, a decision tree,
- 55:55and KNN as base learners.
- 56:05Section 5.2.2 moves to random forest to
- 56:08the California housing data set.
- 56:11The code set trains random forest
- 56:13regressor and gradient boost regressor
- 56:16side by side and prints R squared for
- 56:19both.
- 56:21Both score well.
- 56:24Gradient boosting tends to edge out
- 56:26random forest on this data set, which is
- 56:28a preview of the boosting section coming
- 56:30up.
- 56:32The next code cell looks at feature
- 56:34importances.
- 56:36Tree-based ensembles expose this
- 56:38attribute, which ranks features by how
- 56:41much they contributed to split across
- 56:44the whole forest.
- 56:46The bar plot
- 56:50shows that on California housing, net
- 56:52inc
- 56:54median income is overwhelmingly the most
- 56:57important feature
- 56:59followed by avg occup latitude and
- 57:02longitude.
- 57:04House age and the other columns matter
- 57:06much less.
- 57:07That kind of ranking is one of the
- 57:09practical pros of using tree-based
- 57:11ensembles.
- 57:13The other ensemble family is boosting.
- 57:17The basic idea, train a weak model
- 57:20first, then check with example it go
- 57:23wrong.
- 57:26Reweight the data, give more weight to
- 57:29the examples that were misclass-
- 57:35>> Sorry, I will stop the video here and go
- 57:38back to the
- 57:40notebook.
- 57:45Okay, here we will
- 57:48apply bagging.
- 57:51Mhm.
- 57:52The idea is quite simple. From our
- 57:54training data, we randomly
- 57:58we randomly draw a new data set of the
- 58:01same size. Some samples get picked twice
- 58:05and some don't get picked at all.
- 58:08We do this 100 so we get 100 slightly
- 58:12different data sets. Okay, let's apply
- 58:15this.
- 58:20Base models. Okay, here's the logistic
- 58:24regression, then we will apply decision
- 58:28tree.
- 58:42And KNN.
- 58:58Here, we will train the model.
- 59:09Get this model score
- 59:17and apply bagging.
- 59:51Okay, let's run code.
- 59:57Ah, okay. I forgot to
- 1:00:00split the data. Sorry.
- 1:00:05It should be okay.
- 1:00:09It's still running.
- 1:00:38Oops, seems
- 1:00:40it takes quite long time. So, then I
- 1:00:44will directly uh tell you the result.
- 1:00:49Uh so,
- 1:00:51actually, the the tree gains the most
- 1:00:54and uh logistic regression doesn't
- 1:00:58change at all because the logistic
- 1:01:01regression is already very stable. And
- 1:01:04why does uh bagging method help the
- 1:01:07model? Like think of asking 100 people
- 1:01:10instead of one. Each person makes one
- 1:01:13mis- makes some mistakes, but different
- 1:01:16mis- uh but uh
- 1:01:18each person make different mis- mistake.
- 1:01:21When they vote, the mistakes cost a lot.
- 1:01:24That's the idea for the bagging.
- 1:01:29Oops, it's still running. Maybe I will
- 1:01:32just
- 1:01:33stop it.
- 1:01:36As you can see, it's
- 1:01:37take quite long time to run this
- 1:01:40bagging method.
- 1:01:43Okay, let's go back to the video.
- 1:01:56>> The other ensemble family is boosting.
- 1:02:00The basic idea, train a weak model
- 1:02:03first, then check with example it go
- 1:02:06wrong.
- 1:02:09Reweight the data, give more weight to
- 1:02:12the examples that were misclassified,
- 1:02:15and train another model on the
- 1:02:16reweighted data.
- 1:02:20Repeat this step many times.
- 1:02:23Each new model focuses on what the
- 1:02:25previous one got wrong.
- 1:02:27At the end, aggregate the predictions.
- 1:02:31Examples of boosting algorithm include
- 1:02:34AdaBoost, gradient boosting,
- 1:02:37XGBoost, and LightGBM.
- 1:02:41The last two are the workhorses of
- 1:02:43practical machine learning competitions.
- 1:02:47Why does boosting work?
- 1:02:49Let's look at an example depicted to the
- 1:02:51right, where we try to fit a sine curve
- 1:02:55with simple box-shaped functions with
- 1:02:57boosting.
- 1:02:59Round one captures the easy patterns.
- 1:03:03Round two fixes what is still wrong.
- 1:03:08Round three fixes what is left after
- 1:03:10that, and so on.
- 1:03:12At each round, the residual,
- 1:03:15the gap between truth and prediction
- 1:03:18gets smaller.
- 1:03:20The figure on the right shows this on a
- 1:03:22noisy sine curve. After one round, RMSE
- 1:03:26is 0.46.
- 1:03:28After five, 0.26.
- 1:03:31After 100,
- 1:03:330.08.
- 1:03:36One caveat, with too many rounds,
- 1:03:39boosting starts fitting the noise in the
- 1:03:41data.
- 1:03:42The residual on the noise side keeps
- 1:03:45shrinking,
- 1:03:46but generalization gets worse.
- 1:03:49So, boosting has its own overfitting
- 1:03:51risk that you control through the number
- 1:03:54of rounds.
- 1:03:57A concrete example is XGBoost.
- 1:04:01Boosting is applied to decision trees,
- 1:04:04but each tree is shallow,
- 1:04:07depth around three to eight,
- 1:04:09and each new tree predicts the residual
- 1:04:12of the running sum so far.
- 1:04:14The final prediction is the sum of all
- 1:04:17tree outputs.
- 1:04:21In Python, bagging with decision trees,
- 1:04:24import BaggingClassifier from
- 1:04:26sklearn.ensemble,
- 1:04:29a DecisionTreeClassifier
- 1:04:31from sklearn.tree.
- 1:04:34Pass the decision tree as the base
- 1:04:36estimator, set n_estimators equals to
- 1:04:39100,
- 1:04:41and call fit and predict as usual.
- 1:04:46A gradient boosting,
- 1:04:48import GradientBoostingRegressor
- 1:04:51from sklearn.ensemble.
- 1:04:55Instantiate it,
- 1:04:57and then again, call fit and predict.
- 1:05:01The defaults give a reasonable starting
- 1:05:04point.
- 1:05:07XGBoost has its own Python package.
- 1:05:10From XGBoost, import XGBRegressor.
- 1:05:14The example here trains for up to 1,000
- 1:05:17rounds, watches RMSE on the validation
- 1:05:21set and stops early.
- 1:05:24With early stopping rounds equals to 20,
- 1:05:28the training iteration stop if
- 1:05:30performance is not improved for 20
- 1:05:33consecutive rounds.
- 1:05:37The early stopping rounds parameter is
- 1:05:40the practical antidote to the
- 1:05:42overfitting caveat we just saw.
- 1:05:46Now, let's see boosting in a notebook.
- 1:05:48Please scroll to section 5.5.
- 1:05:52The first code cell compares
- 1:05:56applying decision tree regressor against
- 1:05:59AdaBoost regressor. With the decision
- 1:06:01tree as the base learner, both on
- 1:06:03California housing.
- 1:06:05The output shows train and test R
- 1:06:07squares side by side.
- 1:06:09AdaBoost typically lifts the test score
- 1:06:12over the single tree because a sequence
- 1:06:14of tree corrects errors that any single
- 1:06:17tree could not.
- 1:06:22Section 5.7 moves to XGBoost.
- 1:06:25The first code cell in the simplest
- 1:06:27possible setup.
- 1:06:29XGB regressor.
- 1:06:32Fit, predict, score.
- 1:06:39The printed test R squared gives the
- 1:06:41baseline.
- 1:06:45The next code cell adds early stopping.
- 1:06:48We split the training data in the
- 1:06:50smaller training piece and a validation
- 1:06:52piece.
- 1:06:54Then train XGB regressors.
- 1:06:58The eval stopping rounds
- 1:07:02argument tells XGBoost to watch RMSE on
- 1:07:06the validation set after every round
- 1:07:09and stop as soon as it has not improved
- 1:07:12for 10 consecutive rounds.
- 1:07:15Running this cell
- 1:07:16produces a long stream of log lines.
- 1:07:20One per round.
- 1:07:25Which it scroll past.
- 1:07:29The number that matters is the final
- 1:07:30test R squared printed at the bottom.
- 1:07:33It beats the simple version because
- 1:07:36early stopping picks the right number of
- 1:07:38rounds automatically.
- 1:07:41Here is the practical summary table for
- 1:07:43assembling.
- 1:07:46Use bagging, specifically random forest,
- 1:07:49when you want to avoid overlearning or
- 1:07:52as a baseline early in a project when
- 1:07:55you have not yet identified useful
- 1:07:57features.
- 1:07:59Use the more general bagging classifier,
- 1:08:02a bagging regressor, if you want to
- 1:08:05apply bagging to non-tree based learner
- 1:08:08like K nearest neighbors, a logistic
- 1:08:11regression.
- 1:08:13Use boosting, specifically XGBoost or
- 1:08:17gradient boosting,
- 1:08:19when you want to push accuracy as far as
- 1:08:21it will go, especially you have already
- 1:08:24found useful features through EDA.
- 1:08:27AdaBoost is a classic variant.
- 1:08:31And use no ensemble,
- 1:08:34just a standalone model,
- 1:08:36when interpretability of the result
- 1:08:38matters more than raw accuracy.
- 1:08:42A single decision tree or linear model
- 1:08:44is much easier to explain to a
- 1:08:46stakeholder.
- 1:08:50Let's wrap up.
- 1:08:51First, feature engineering can hugely
- 1:08:54impact on the model's performance.
- 1:08:57Feature transformation includes
- 1:08:59standardization,
- 1:09:01normalization, log, and then encoding.
- 1:09:05Feature creation includes combining
- 1:09:07features and using domain knowledge.
- 1:09:11Second, feature engineering is
- 1:09:14iterative.
- 1:09:16Expect several trial and error loops
- 1:09:18between feature design and model
- 1:09:20evaluation.
- 1:09:22There is no single recipe that works for
- 1:09:24every data set.
- 1:09:26And third,
- 1:09:28ensembling is very useful in real-world
- 1:09:31application.
- 1:09:32Bagging methods, such as random forest,
- 1:09:36and boosting methods, such as AdaBoost,
- 1:09:39XGBoost, and LightGBM, are the first
- 1:09:42choice.
- 1:09:44That concludes this lecture on feature
- 1:09:46engineering.
- 1:09:47Thank you very much for listening, and I
- 1:09:50hope you enjoyed it.
About this transcript
This page contains the full transcript of GCI World 2026 September Session8 During Lecture by GCI World (Matsuo-Iwasawa Lab), generated from the public captions YouTube serves with the video. The transcript has 6,259 words across 1,298 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.