YouTube2Text

GCI World 2026 September Session8 During Lecture — Transcript

by GCI World (Matsuo-Iwasawa Lab) · 6,259 words · 1,298 segments · language en · Watch on YouTube

Full transcript

  1. 0:02Hello everyone and welcome to this
  2. 0:04lecture on feature engineering.
  3. 0:09In this lecture we will cover five
  4. 0:12topics.
  5. 0:13First, an overview of feature
  6. 0:15engineering. What it is and why we need
  7. 0:18it.
  8. 0:19Second, transforming numerical
  9. 0:21variables.
  10. 0:23Scaling and log transformation.
  11. 0:26Third, transforming categorical
  12. 0:29variables.
  13. 0:30Label encoding, one-hot encoding, and
  14. 0:33cross features.
  15. 0:36Fourth, the use of domain knowledge.
  16. 0:39Turning raw data into meaningful
  17. 0:41features using expertise about the
  18. 0:43problem.
  19. 0:45And fifth, ensembling.
  20. 0:47It's about improving your model by
  21. 0:50merging your models and we cover two
  22. 0:53major methods. Bagging and boosting.
  23. 0:58Let's start with an overview of feature
  24. 1:00engineering.
  25. 1:04Before we dive in, here is the
  26. 1:07supervised learning workflow we have
  27. 1:09been using throughout the course.
  28. 1:11The flow runs left to right.
  29. 1:15Data collection.
  30. 1:16EDA and pre-processing.
  31. 1:19Splitting into training and validation
  32. 1:21sets.
  33. 1:22The train evaluate loop with
  34. 1:24hyperparameter tuning.
  35. 1:27And finally, evaluation on a test set.
  36. 1:30After deployment.
  37. 1:33Today's focus is on what happens before
  38. 1:36we train the model.
  39. 1:38Specifically, the pre-processing step
  40. 1:41where we shape the input variables
  41. 1:43themselves.
  42. 1:45Good features make the rest of the
  43. 1:47pipeline easier.
  44. 1:49Poor features bottleneck everything
  45. 1:51downstream.
  46. 1:54So, what is feature engineering?
  47. 1:57Feature engineering is the process of
  48. 2:00selecting, transforming, or creating
  49. 2:03input variables to improve the
  50. 2:05performance of a machine learning model.
  51. 2:10Recall that a supervised model is just a
  52. 2:13function y equals to f of x.
  53. 2:16X is the explanatory variables. Y is the
  54. 2:20target.
  55. 2:22Feature engineering is everything we do
  56. 2:25on the x side of that variable before
  57. 2:28the models use it.
  58. 2:31A concrete example,
  59. 2:33sales prediction.
  60. 2:35The raw data might give us a date and a
  61. 2:39product ID. Just two columns.
  62. 2:43The date and product ID are not directly
  63. 2:47meaningful to the model.
  64. 2:49A better feature set derived from the
  65. 2:51same raw data is
  66. 2:53the day of week, holiday or not, season,
  67. 2:58and product category.
  68. 3:01Each one is a richer, more directly
  69. 3:04predictive signal than the raw timestamp
  70. 3:06or ID.
  71. 3:09Feature engineering is a concept of
  72. 3:11making a machine learning model with
  73. 3:14such features that seem more related to
  74. 3:16the target variable to build a better
  75. 3:18model.
  76. 3:22Why is feature engineering so important?
  77. 3:25Three reasons.
  78. 3:27To boost model performance,
  79. 3:29to prevent overfitting,
  80. 3:32and to make your model more
  81. 3:33interpretable.
  82. 3:37One important caveat,
  83. 3:39feature engineering is fundamentally
  84. 3:42trial and error.
  85. 3:44There is no single recipe that works for
  86. 3:47every data set.
  87. 3:49In this lecture, we will cover the
  88. 3:52general techniques you can try across
  89. 3:54different problems.
  90. 3:58Feature engineering breaks down into
  91. 4:00four main types.
  92. 4:03The first is transforming features,
  93. 4:06reshaping existing variables so the
  94. 4:08model can use them better.
  95. 4:13Transforming includes scaling, log
  96. 4:16transformation,
  97. 4:18and encoding categorical variables into
  98. 4:21numbers.
  99. 4:23The second activity is creating
  100. 4:26features,
  101. 4:27building new variables from existing
  102. 4:30ones, either by crossing them or by
  103. 4:33using domain knowledge.
  104. 4:37There are two more types, extracting
  105. 4:40features,
  106. 4:42compressing a large set in a smaller one
  107. 4:45like PCA,
  108. 4:47and selecting features, removing columns
  109. 4:51where the input dimension is too large
  110. 4:53relative to the data.
  111. 4:55These are also part of feature
  112. 4:57engineering.
  113. 5:00For this lecture,
  114. 5:01our focus in the highlighted pair,
  115. 5:04transforming features, and creating
  116. 5:06features.
  117. 5:09Next, transforming numerical variables.
  118. 5:14Before diving into concrete transforming
  119. 5:17method, let us explain about what types
  120. 5:20of variables exist in the first place.
  121. 5:23You need to understand what kind of
  122. 5:25input variables exist to apply the right
  123. 5:28transformation that depends on the
  124. 5:30variable type.
  125. 5:32There are two broad categories.
  126. 5:35Quantitative data,
  127. 5:37also called numerical variables,
  128. 5:40and qualitative data, also called
  129. 5:43categorical variables.
  130. 5:47Within quantitative data, the ratio
  131. 5:50scale is the most informative.
  132. 5:53Differences have meaning, and there is a
  133. 5:55true zero.
  134. 5:58Body weight, 50 kilos or 70 kilos, and
  135. 6:02distance, 2 km or 200 km, are ratio
  136. 6:07scale examples.
  137. 6:09Saying twice as heavy is meaningful
  138. 6:12because zero is a fixed reference.
  139. 6:16The interval scale also has meaningful
  140. 6:19differences but no true zero.
  141. 6:24Temperature in Celsius, 50°, 30°,
  142. 6:28and year,
  143. 6:292000 or 2025,
  144. 6:33are interval scale.
  145. 6:35The gap between values is meaningful,
  146. 6:38but twice as much is not.
  147. 6:41You can't say, for example, 30° C
  148. 6:45is two times hotter than 15° C.
  149. 6:50Within qualitative data, the ordinal
  150. 6:53scale has a meaningful order, like
  151. 6:56academic grades, A, B, C, or survey
  152. 6:59responses, good or bad.
  153. 7:03The nominal scale has only distinct
  154. 7:06categories with no natural order.
  155. 7:09Nationality, Japanese, Chinese,
  156. 7:12or product ID.
  157. 7:16This four-way split, ratio, interval,
  158. 7:20ordinal, nominal,
  159. 7:22is what tells us which transformation
  160. 7:25makes sense
  161. 7:26for which column.
  162. 7:30Now, let's look at the transformations
  163. 7:32themselves, starting with numerical
  164. 7:34variables.
  165. 7:37The first transformation is scaling.
  166. 7:40When features differ greatly in scale,
  167. 7:43the larger scaled feature can have an
  168. 7:46undue influence on the model.
  169. 7:49For example, if one feature is income in
  170. 7:52dollars, ranging up to millions, and
  171. 7:55another is age, 0 to 100,
  172. 7:58a distance-based model will be dominated
  173. 8:01by income.
  174. 8:04The first scaling method is
  175. 8:06standardization.
  176. 8:08Rescale so the mean is zero and the
  177. 8:11standard deviation is one.
  178. 8:13The formula is X minus its mean
  179. 8:17divided by its standard deviation.
  180. 8:20The histogram on the right shows that
  181. 8:22the shape of the distribution is
  182. 8:24preserved. Only its location and width
  183. 8:28change.
  184. 8:30The second method is normalization.
  185. 8:34Also called min-max scaling.
  186. 8:37Rescale so the minimum is zero and the
  187. 8:40maximum is one.
  188. 8:42The formula
  189. 8:44is X minus the minimum of X
  190. 8:46divided by the maximum of X minus the
  191. 8:49minimum of X.
  192. 8:51Same idea, different reference points.
  193. 8:56The next transformation is log
  194. 8:59transformation.
  195. 9:01The log compresses large values and
  196. 9:04stretches small ones
  197. 9:06which makes it easier for the model to
  198. 9:08see relationship across orders of
  199. 9:10magnitude.
  200. 9:12Log transformation is especially useful
  201. 9:16when the distribution is heavily skewed.
  202. 9:19Many small values and a few very large
  203. 9:22ones like income.
  204. 9:25One important caveat.
  205. 9:27Log is not defined for zero or negative
  206. 9:31values.
  207. 9:32So you either apply it only to strictly
  208. 9:34positive quantities
  209. 9:36or you can use log of 1 plus X for
  210. 9:39example
  211. 9:40when the data contains zeros.
  212. 9:45In Python
  213. 9:47from sklearn.preprocessing
  214. 9:50import StandardScaler
  215. 9:52for standardization
  216. 9:55and MinMaxScaler
  217. 9:56for normalization.
  218. 9:59Both have fit transformed.
  219. 10:01For log transformation
  220. 10:03just call np.logx.
  221. 10:06That's all there is to the
  222. 10:08implementation.
  223. 10:12One subtle, but important point,
  224. 10:15scaling has almost no effect on
  225. 10:18tree-based models.
  226. 10:20The reason is that trees split on the
  227. 10:23rank order of feature values, not the
  228. 10:26absolute scale.
  229. 10:28Scaling doesn't change the magnitude
  230. 10:30relationship between the two values.
  231. 10:34Tree-based methods, such as decision
  232. 10:36trees, give us essentially identical
  233. 10:39results before and after
  234. 10:41standardization.
  235. 10:43Scaling still matters, though, for
  236. 10:45methods including regression models.
  237. 10:50Now, let's from the slides to the
  238. 10:52notebook.
  239. 10:54Please run the cell up to section two,
  240. 10:56transforming numerical variables.
  241. 11:06The first code cell in the section 2.1
  242. 11:09demonstrates standardization on the Iris
  243. 11:12data set.
  244. 11:16We import load_iris and StandardScaler
  245. 11:19from scikit-learn, load the data, and
  246. 11:22pull out the single column, sepal
  247. 11:24length.
  248. 11:25Then we create a StandardScaler and call
  249. 11:27fit_transform, and add the result back
  250. 11:30as a new column called sepal length STD.
  251. 11:40The describe output at the bottom
  252. 11:42confirms
  253. 11:43the standardized column has mean
  254. 11:45approximately zero and standard
  255. 11:47deviation approximately one, exactly as
  256. 11:51expected.
  257. 11:56The next code cell runs the comparison
  258. 11:58experiment we previewed on the slides.
  259. 12:02We set up four models in a dictionary,
  260. 12:04plain logistic regression,
  261. 12:07logistic regression wrapped in a
  262. 12:09pipeline with StandardScaler, a plain
  263. 12:12decision tree,
  264. 12:13and a decision tree with the same scalar
  265. 12:15in front.
  266. 12:17We fit each one and record the train and
  267. 12:19validation accuracy.
  268. 12:26The alpha table makes the lesson
  269. 12:28visible.
  270. 12:29Logistic regression score changes
  271. 12:31meaningfully when we add a standard
  272. 12:33scalar.
  273. 12:35It is sensible to scale.
  274. 12:37The decision tree score barely moves
  275. 12:40whether we standardize or not because
  276. 12:42the tree splits on rank, not magnitude.
  277. 12:50Section 2.2 moves to log transformation
  278. 12:53on the wine data set.
  279. 12:56We load the wine data and pick two
  280. 12:58heavily skewed columns.
  281. 13:00Color intensity and malic acid.
  282. 13:04The first histograms show the raw
  283. 13:06distributions.
  284. 13:10The second histograms show them after
  285. 13:13applying np.log.
  286. 13:16The skew is dramatically reduced. Both
  287. 13:19distributions become much closer to
  288. 13:22symmetric.
  289. 13:24Does that actually help downstream
  290. 13:26classification?
  291. 13:27The next code cell answers that question
  292. 13:29with a five-fold cross-validation.
  293. 13:34We train a logistic regression
  294. 13:37on the two columns,
  295. 13:38once with the raw values and once with
  296. 13:41the log transformed values,
  297. 13:43scoring with weighted F1.
  298. 13:48The output principal scores and the log
  299. 13:50transformed version is noticeably
  300. 13:52higher,
  301. 13:54confirming that for linear models,
  302. 13:56taming skewness has a real impact on
  303. 13:58performance.
  304. 14:02Practice question one in section asks
  305. 14:05you to repeat this kind of comparison on
  306. 14:06the California housing data with ridge
  307. 14:09and a decision tree with and without
  308. 14:11standardization and log.
  309. 14:37Okay.
  310. 14:38Then uh we will begin the interactive
  311. 14:43hands-on exercise. Please open students
  312. 14:46guide and open lecture materials uh from
  313. 14:50Google Drive and navigate to today's
  314. 14:52lecture. And you should uh
  315. 14:56in
  316. 14:57in the our lectures, we will use Google
  317. 15:00Colab and for the hands-on exercise.
  318. 15:06Uh please open the notebook lecture
  319. 15:08eight notebook. Let's go through it
  320. 15:10together.
  321. 15:28Okay. In the practice uh
  322. 15:34question one
  323. 15:37we will uh we will first split the data
  324. 15:41into a train set and a test set.
  325. 16:18Sorry, I will Maybe I lost some
  326. 16:23import things.
  327. 16:26Let me import everything.
  328. 16:44Okay. It's quite simple.
  329. 16:48And then we will train two models on the
  330. 16:51raw raw features and record their
  331. 16:54uh test squared R. The first model is
  332. 16:58ridge.
  333. 17:31Okay, and then
  334. 17:33I'll add the print here.
  335. 17:35So,
  336. 17:38uh
  337. 17:39Sorry.
  338. 17:47Uh so, ridge draws one straight line
  339. 17:49through the data. The second model is a
  340. 17:53decision tree.
  341. 17:55And maybe
  342. 18:08Okay.
  343. 18:10Here's the result. Uh a decision tree
  344. 18:12just asks a few yes or no questions. Uh
  345. 18:16the squared R is the gap between the
  346. 18:18algorithm and the random guess.
  347. 18:21Um so, if we get
  348. 18:23uh one, then that means our algorithm is
  349. 18:26perfect. And if we get zero, uh that
  350. 18:29means we are no better than guessing the
  351. 18:31average. Here are the results. The ridge
  352. 18:35gets uh 0. uh 0.
  353. 18:39594 and the the tree gets 0.618.
  354. 18:45Now, I'll paste some plotting code the
  355. 18:48below to uh enhance your intuitive
  356. 18:52understanding. You don't need to
  357. 18:54understand the plotting code. It's just
  358. 18:56here to help you see what the models are
  359. 18:58doing.
  360. 18:59>> [snorts]
  361. 19:00>> Okay.
  362. 19:09Okay.
  363. 19:10Okay.
  364. 19:16Here,
  365. 19:17uh you can see each dot is means one
  366. 19:21house. The true price are on the X X
  367. 19:25axis, predicted price on the Y axis. A
  368. 19:28perfect model puts every dot on the dash
  369. 19:31line. Ridge gives a very smooth cloud,
  370. 19:34but it's even predicts negative prices.
  371. 19:37The tree looks different. It starts from
  372. 19:40flat lines. That's because the tree
  373. 19:43sorts houses into small groups with its
  374. 19:45yes or no questions, and every house in
  375. 19:48the group gets the same price. Our tree
  376. 19:51has uh the the the the depth of our tree
  377. 19:55is six, so our tree has uh two power six
  378. 20:00equal to 64 groups, so it can only
  379. 20:03predict 64 different prices.
  380. 20:11Um
  381. 20:13Next, I'll apply standard scaling to the
  382. 20:16features.
  383. 20:17For your intuiti- intuiti- uh intuition,
  384. 20:21standardizing is just changing the units
  385. 20:23like centimeters versus inches. Now look
  386. 20:27at Now I will
  387. 20:30write the code.
  388. 20:38And the apply the
  389. 20:41standardize into ridge.
  390. 20:50Yes, here's the result.
  391. 20:53Then apply it into tree.
  392. 21:09Yes, uh so now you can look at the
  393. 21:13output. Um
  394. 21:15no
  395. 21:16no score change. Uh why didn't the
  396. 21:19tree's result uh result change as we uh
  397. 21:22taught as we told in the um
  398. 21:26video, a decision tree only cares about
  399. 21:28the rank order of values. Uh scaling
  400. 21:31changed the numbers but never changed
  401. 21:33the order. So the same houses end up in
  402. 21:36the same groups with the same prices.
  403. 21:38The why not ridge? Actually, a linear
  404. 21:41model just changed its weights to absorb
  405. 21:45the new units. The only part that cares
  406. 21:47about units is the penalty on the
  407. 21:50weights, but our penalty is quite small.
  408. 21:53You can see
  409. 21:54in the in the definition of the ridge,
  410. 21:57we set the alpha into one, so scaling
  411. 22:00makes no difference here.
  412. 22:03Uh
  413. 22:04and I will
  414. 22:07import a table below so you can get um
  415. 22:12understanding of why these two models
  416. 22:16didn't didn't change.
  417. 22:21Okay.
  418. 22:25Um
  419. 22:26you can see this in the table below if
  420. 22:30you
  421. 22:31multiply the raw coefficient at the
  422. 22:34[clears throat] features STD, you get
  423. 22:37almost exactly the scaled coefficient.
  424. 22:41This is the kind of like evidence that
  425. 22:44the weights absorb the effect of
  426. 22:47standard scaling in the ridge model.
  427. 22:50Uh in the tree plot uh the dash line is
  428. 22:54the tree's first split. Scaling didn't
  429. 22:57change the order of the values, so the
  430. 22:59houses on the left side and the right
  431. 23:01side of the line stay exactly the same.
  432. 23:12Then last questions.
  433. 23:17Uh let's take a quick look at the log
  434. 23:20transformation.
  435. 23:22I'll complete this code part.
  436. 23:48Okay.
  437. 23:51You can see the output. The ridge uh
  438. 23:55goes up, but the tree stays the same.
  439. 23:58Why does ridge get better?
  440. 24:01Uh I will draw a plot for you to
  441. 24:05understand why.
  442. 24:14Okay.
  443. 24:15Mm.
  444. 24:19Uh again, you [clears throat] don't need
  445. 24:21to understand the plotting code. Uh
  446. 24:24first in the top row for population and
  447. 24:27the
  448. 24:28average occup- oc- occupancy. Most
  449. 24:31houses
  450. 24:32Oh, sorry.
  451. 24:34Most houses are in one
  452. 24:38bar and a few houses have very, very big
  453. 24:41values.
  454. 24:42Uh as I mentioned before, ridge draws
  455. 24:45one straight line and this few big
  456. 24:47values pull the line toward them.
  457. 24:51Uh the log makes big values smaller, so
  458. 24:54the bottom row looks uh much more even.
  459. 24:57Now the line fits most of houses better
  460. 25:00and the line, not the tree, the log,
  461. 25:02just like scaling, doesn't change the
  462. 25:04order, so the tree makes the same groups
  463. 25:07again.
  464. 25:09Okay, then. Let's go back to the video
  465. 25:12part.
  466. 25:27>> Next, transforming categorical
  467. 25:29variables.
  468. 25:32The first method is label encoding.
  469. 25:35Assign each distinct category a unique
  470. 25:38integer.
  471. 25:39For example,
  472. 25:41red for one,
  473. 25:43green for two,
  474. 25:45blue for three.
  475. 25:48So, our color column of red, green,
  476. 25:51blue, green, green, red
  477. 25:55becomes 1 2 3 2 2 1.
  478. 26:03Label encoding is simple and works fine
  479. 26:06with tree-based models.
  480. 26:08A tree can split between any pair of
  481. 26:11integers.
  482. 26:14The catch is that it introduces a fake
  483. 26:17ordering.
  484. 26:18The model might think that blue with
  485. 26:21value three is somehow greater than red
  486. 26:24with value one.
  487. 26:26For linear or distance-based models, the
  488. 26:29fake order can mislead the model.
  489. 26:36The second method is one-hot encoding.
  490. 26:40Create a separate column for each
  491. 26:42category and assign one to the column
  492. 26:45that matches.
  493. 26:48The same color column becomes three new
  494. 26:50columns.
  495. 26:52Color red, color green,
  496. 26:54and color blue.
  497. 26:57Row by row, the matching column gets a
  498. 27:00one and the rest gets zero.
  499. 27:04So, red becomes 1 0 0.
  500. 27:09Green becomes 0 1 0,
  501. 27:12and blue becomes 0 0 1,
  502. 27:14and so on.
  503. 27:17The full encoded table has one one per
  504. 27:19row and zeros everywhere else.
  505. 27:25One-hot encoding's merit is that it
  506. 27:28removes the fake numerical relationship.
  507. 27:31Every category is treated
  508. 27:32asymmetrically.
  509. 27:35The demerit is high memory consumption.
  510. 27:39For a high cardinality variable, like
  511. 27:41product IDs with thousands of distinct
  512. 27:44values,
  513. 27:45you get one column per category, which
  514. 27:48can blow up memory and lead to
  515. 27:50overfitting.
  516. 27:54The next idea is cross features, also
  517. 27:58called interaction features.
  518. 28:00A new feature built from a product or
  519. 28:03other combination
  520. 28:05of two or more existing features.
  521. 28:09For example,
  522. 28:11A, B, C
  523. 28:14can be transformed into
  524. 28:16A * B,
  525. 28:18A * C,
  526. 28:20B * C,
  527. 28:22and A * B * C,
  528. 28:25and so on.
  529. 28:27A classic case is rent prediction.
  530. 28:30Rent is high when room size is large and
  531. 28:35distance to station is small.
  532. 28:38The model needs the product of these
  533. 28:40two, which a simple model cannot easily
  534. 28:43learn on its own.
  535. 28:45On the left, there is a 2D scatter of
  536. 28:48two normalized leaf features.
  537. 28:51The two classes occupy opposite corners.
  538. 28:54Leaf length and width is alone cannot
  539. 28:57separate them with a single line.
  540. 29:00But,
  541. 29:01if we add a new column, length times
  542. 29:03width,
  543. 29:04and look at the single column instead,
  544. 29:07the picture changes.
  545. 29:10The product column separate the two
  546. 29:12classes cleanly with a single threshold.
  547. 29:16The red horizontal line shows that
  548. 29:18everything above some cutoff is class
  549. 29:21zero and everything below is class one.
  550. 29:25A linear model can now solve the problem
  551. 29:27trivially.
  552. 29:31Cross features capture interactions that
  553. 29:33simple models cannot learn on their own,
  554. 29:36which is their merit.
  555. 29:38The demerit
  556. 29:39is that the number of possible cross
  557. 29:41features grows combinatorially,
  558. 29:43so they cause memory and can lead to
  559. 29:45overfitting if you make too many.
  560. 29:51In Python, from scikit-learn
  561. 29:53preprocessing,
  562. 29:55import LabelEncoder
  563. 29:57and OneHotEncoder.
  564. 29:59LabelEncoder
  565. 30:01.fit_transform(df.city)
  566. 30:05gives you the label encoded column.
  567. 30:08And pd.get_dummies
  568. 30:12df.columns.city.device
  569. 30:14gives you the one hot encoded version.
  570. 30:19For cross features, you can string
  571. 30:22concatenate two categorical columns into
  572. 30:24a new column.
  573. 30:26df.city
  574. 30:28+
  575. 30:29"_"
  576. 30:31+
  577. 30:32df.device
  578. 30:34and then one hot encode that combined
  579. 30:36column.
  580. 30:38That is the recipe we will use in the
  581. 30:40notebook.
  582. 30:44Let's see the notebook implementations.
  583. 30:47Please scroll to section three.
  584. 30:50The notebook uses the Titanic data set
  585. 30:52where each row is a passenger
  586. 30:55and the target is whether they survived.
  587. 30:58The categorical column we will encode is
  588. 31:00embarked.
  589. 31:01Port of embarkation.
  590. 31:04C
  591. 31:05S
  592. 31:06and Q.
  593. 31:12The first code cell of this section
  594. 31:14loads the Titanic data directly from a
  595. 31:16public mirror with pd. {dot} read_csv
  596. 31:20and pulls out the embark column with
  597. 31:23missing values filled in
  598. 31:25as S.
  599. 31:28The next code cell applies label encoder
  600. 31:31to that column.
  601. 31:32Each of S C
  602. 31:35Q
  603. 31:36gets a unique integer.
  604. 31:42The side-by-side data frame shows the
  605. 31:44mapping.
  606. 31:48Then for one-hot encoding
  607. 31:51we just call
  608. 31:53pd.get_dummies
  609. 31:56on the same column.
  610. 31:58That produces three boolean columns
  611. 32:01one per port.
  612. 32:03We cast them to integers with
  613. 32:07astype int
  614. 32:10because many models do not accept true
  615. 32:13false.
  616. 32:15The first 10 rows confirm
  617. 32:18each row has exactly one one.
  618. 32:26Section 3.3 moves to cross features.
  619. 32:29We pick five numeric Titanic columns.
  620. 32:36Age, fare
  621. 32:38sibsp
  622. 32:40parch
  623. 32:41and pclass.
  624. 32:43And fill ages missing values with a
  625. 32:45median.
  626. 32:48The next code cell uses polynomial
  627. 32:50features
  628. 32:52to build all pairwise products
  629. 32:53automatically.
  630. 32:55We get 15 columns total.
  631. 33:03The practice question two asks you to
  632. 33:05combine the sex and embark columns into
  633. 33:08a single string label like female under
  634. 33:11bar S or male under bar C. And then look
  635. 33:14at the survival rate per group.
  636. 33:17Next, the use of the main knowledge.
  637. 33:25>> Okay, let's go back to the notebook.
  638. 33:33Okay, in the practice question two, we
  639. 33:36will now apply label encoding and
  640. 33:39one-hot encoding to the
  641. 33:41Titanic dataset.
  642. 33:44First, I will um
  643. 33:47uh
  644. 33:51First, I will
  645. 33:53encode both labels.
  646. 34:41Okay,
  647. 34:43here we work with the categorical
  648. 34:46variables. Embarked is the port where
  649. 34:49each passenger got on the ship. As C or
  650. 34:52Q, a model can't read letters letters.
  651. 34:55So, we have to turn them into numbers.
  652. 34:58This is called encoding.
  653. 35:00There are two ways. First, label
  654. 35:02encoding gives each port a number. C is
  655. 35:05zero, Q is one, S is two. One-hot
  656. 35:09encoding makes one column per port. One
  657. 35:12in the column of the passenger's port
  658. 35:14and zero in the others. We will print uh
  659. 35:18both
  660. 35:19both results together after uh part
  661. 35:22three.
  662. 35:23Uh which one should we use for logistic
  663. 35:26uh regression? Use one-hot. With label
  664. 35:28encoding, the model thinks S is bigger
  665. 35:31than Q and Q is bigger than C. But,
  666. 35:34ports have no order. For a decision
  667. 35:36tree, label encoding is fine. The tree
  668. 35:39just asked question like, is the number
  669. 35:42uh less than one? So, it can still
  670. 35:45separate the ports.
  671. 35:47And then, I will apply one-hot encoding
  672. 35:50to the sex column.
  673. 36:16Uh it has only two values, female and
  674. 36:19male. So, we get two columns, sex female
  675. 36:22and sex male. Notice that these two
  676. 36:24columns always add up to one because one
  677. 36:28person is either female or male.
  678. 36:31Uh so, if you know one, you already know
  679. 36:34the other. So, in practice, you can keep
  680. 36:36just one of them. For example, with uh
  681. 36:39drop_first equal to true.
  682. 36:43In the third part, we will create a
  683. 36:47cross feature
  684. 36:49by combining sex and embarked into a
  685. 36:52single stream label.
  686. 36:58Sorry.
  687. 37:16Okay, you can see now we make a cross
  688. 37:20feature. We simply glue sex and impart
  689. 37:24in embarked together into one label like
  690. 37:30you can see like
  691. 37:33sex
  692. 37:35You can see like female C or male
  693. 37:39male S. You can see the sex embarked
  694. 37:41column.
  695. 37:44Then we one hot encoded, so we get
  696. 37:47six columns. You can see them in the
  697. 37:50printout. Yes, you can see the printout
  698. 37:54in this line.
  699. 38:00Why do this?
  700. 38:01A simple model like logistic regression
  701. 38:04looks at each column separately. Like
  702. 38:07being female adds this much, port C adds
  703. 38:10this much. It can't see it it can't say
  704. 38:13female and port C is special. The cross
  705. 38:15feature gives every combination its own
  706. 38:18column, so the model can learn that
  707. 38:21directly.
  708. 38:28Now let's check the survival rate for
  709. 38:33each group. You can simply run the code
  710. 38:36in the part four.
  711. 38:38The highest is female C about
  712. 38:4188%
  713. 38:43survived. The lowest is male Q only
  714. 38:45about 7%. That's gap of almost like
  715. 38:49uh
  716. 38:5080%. Compare this with one column along
  717. 38:54by sex along, the gap is only mm 55% by
  718. 38:58impact along, it's only 22
  719. 39:01percent. So, the cross feature shows a
  720. 39:03much bigger difference than either
  721. 39:05column by itself.
  722. 39:07Uh and here's the interesting part. Look
  723. 39:09at port Q for women. Q is quite good, uh
  724. 39:1375% for men, Q is the worst,
  725. 39:17uh 7%. So, the effect of the port
  726. 39:19depends on sex.
  727. 39:21Neither column along can show this, only
  728. 39:24the combination can. So, that's why the
  729. 39:27combination
  730. 39:28label is very
  731. 39:30important.
  732. 39:31Okay, then let's go back to the video.
  733. 39:45>> Next, the use of the main knowledge.
  734. 39:49Domain knowledge means expertise in the
  735. 39:52specific problem area.
  736. 39:55It helps us transform raw data into
  737. 39:58meaningful features that are more
  738. 39:59directly related to the prediction
  739. 40:02target.
  740. 40:03Take rent prediction as an example.
  741. 40:06The raw data might give us only address,
  742. 40:09size, and age of the building.
  743. 40:12Some of these may not be directly
  744. 40:15meaningful to a model.
  745. 40:18But if we know real estate,
  746. 40:21we know what actually drives rent.
  747. 40:25We can use the knowledge to compute new
  748. 40:27features from the raw address.
  749. 40:29Distance to the nearest station and
  750. 40:32distance to the nearest school.
  751. 40:34Those are far more predictive than the
  752. 40:36raw string address.
  753. 40:39Here's the same idea in the table.
  754. 40:42For real estate task,
  755. 40:44predicting house prices or rent,
  756. 40:47the raw data is address, size, and age.
  757. 40:52The domain knowledge feature we will
  758. 40:53build are distance to the nearest
  759. 40:56station,
  760. 40:57price per square meter,
  761. 40:59and school district.
  762. 41:03For a soccer task, predicting whether a
  763. 41:05shot becomes a goal,
  764. 41:07the raw data is ball position and player
  765. 41:10positions.
  766. 41:12The domain knowledge tells us the useful
  767. 41:14features are distance to goal, shooting
  768. 41:17angle, and the number of defenders
  769. 41:19between the shooter and the goal.
  770. 41:23And for e-commerce,
  771. 41:25predicting whether a user buys a
  772. 41:27product, the raw data is browsing
  773. 41:30history and purchase history.
  774. 41:32Domain knowledge gives us features like
  775. 41:34number of views in the past week,
  776. 41:37and time since the last purchase.
  777. 41:40In every case, the raw data is the same,
  778. 41:43but the engineering features are
  779. 41:45tailored to the physics
  780. 41:47of the problem.
  781. 41:50Let's see domain knowledge applied to
  782. 41:52Titanic.
  783. 41:54Please scroll to section four.
  784. 41:56We will use three domain knowledge
  785. 41:58constructions,
  786. 42:00extracting passenger titles from names,
  787. 42:03combining sex and class,
  788. 42:05and collapsing family columns into
  789. 42:07single family size.
  790. 42:12Section 4.1 extracts the title hidden
  791. 42:15inside the name column.
  792. 42:17The first goal cell extracts the
  793. 42:19character strings that match Mr., Mrs.,
  794. 42:23Miss,
  795. 42:24Master, and so on.
  796. 42:26We apply it to every name and store the
  797. 42:29results
  798. 42:30in a new title column.
  799. 42:33DF_title.unique
  800. 42:35show the full set we extracted.
  801. 42:42Some unfamiliar titles will appear.
  802. 42:45Doctor,
  803. 42:47Reverend,
  804. 42:49Colonel,
  805. 42:51Mademoiselle,
  806. 42:54Madame, and others.
  807. 42:56The value count cell
  808. 42:58shows how common each one is.
  809. 43:04Most of them appear fewer than 10 times,
  810. 43:06so we collapse them in a single rare
  811. 43:09category and merge a few variants.
  812. 43:17Mademoiselle and Miss become Miss, and
  813. 43:20Madame becomes Mrs.
  814. 43:23The clean up is just a few replace
  815. 43:25calls.
  816. 43:30We then one-hot encode the clean title
  817. 43:32column with pd.get_dummies.
  818. 43:36Cast to integers
  819. 43:38and concatenate the new columns back in
  820. 43:41the data frame.
  821. 43:44The final df.head confirms that columns
  822. 43:48like title_ under_Mr, title_under_Miss,
  823. 43:53title_under_Mister,
  824. 43:54title_under_Misses,
  825. 43:56and title_rare are now ready to feed to
  826. 43:59a classifier.
  827. 44:04Section 4.2 combines sex_pclass into a
  828. 44:07single label.
  829. 44:09The motivation is the women and children
  830. 44:11first evacuation rule, which was applied
  831. 44:14most consistently in the first class
  832. 44:16cabins.
  833. 44:21The code creates a new column sex_pclass
  834. 44:26and then groups by that label to show
  835. 44:28the survival rate per group.
  836. 44:31The output is striking.
  837. 44:34Female one, first class women, survive
  838. 44:38at about 97%.
  839. 44:40Male under_3, third class men, survive
  840. 44:45at about 14%.
  841. 44:47Neither sex nor pclass alone shows
  842. 44:50anywhere near that spread.
  843. 44:55Practice question three ask you to build
  844. 44:58a family size features.
  845. 45:00SibSp plus Parch plus one.
  846. 45:04Pause and try it.
  847. 45:12>> Okay, let's go back to our notebook to
  848. 45:17complete the
  849. 45:19uh practice uh practice question three.
  850. 45:23Mhm.
  851. 45:27Practice question three is about uh
  852. 45:29domain knowledge.
  853. 45:31Uh
  854. 45:32SibSp counts siblings and Parch counts
  855. 45:36parents and children.
  856. 45:39So, the family size equal to
  857. 45:44Sib
  858. 45:47SibSp
  859. 45:49plus Parch.
  860. 45:54plus one.
  861. 45:57And is alone
  862. 46:00equal to
  863. 46:02when the family size
  864. 46:11equal to one.
  865. 46:16Okay.
  866. 46:20Uh
  867. 46:24So, in the
  868. 46:26in
  869. 46:27we
  870. 46:28uh when we want to predict the
  871. 46:31survival rate
  872. 46:33uh during the evacua- evacuation, what
  873. 46:36prob- what probably mattered was very
  874. 46:39simple, like how big is your family and
  875. 46:42can you stay together? So, we uh add
  876. 46:45them up plus one for the passenger to
  877. 46:47get family size. And is alone is just a
  878. 46:51one if family size is one. It's pretty
  879. 46:53easy.
  880. 47:03>> Then we will group by group by family
  881. 47:06size and compute the mean of survived
  882. 47:09per group and
  883. 47:12plot the results.
  884. 47:18Okay.
  885. 47:21Just
  886. 47:24SR equal to
  887. 47:30group
  888. 47:32I
  889. 47:34family size
  890. 47:39survived
  891. 47:42mean value.
  892. 47:44Then I will copy the
  893. 47:47visualization code.
  894. 47:58Okay.
  895. 48:00Uh now here's the survival rate for each
  896. 48:04family size.
  897. 48:05Mhm.
  898. 48:09The you can see in the plot the dash
  899. 48:11line is the overall survival rate about
  900. 48:15um
  901. 48:15uh
  902. 48:1740%.
  903. 48:19Mhm.
  904. 48:21But it's not a simple bigger is better
  905. 48:23line. There's a sweet spot. People
  906. 48:27traveling alone only about 30% survived.
  907. 48:31Families of two to four did much better,
  908. 48:3455 to 72%. The families of five or more
  909. 48:40drop sharply and nobody in the families
  910. 48:42of eight and 11 survived.
  911. 48:50And uh
  912. 48:54finally, does it really help a model?
  913. 48:59Let me complete the code here.
  914. 49:28Okay.
  915. 49:30So
  916. 49:32here we train logistic regression with P
  917. 49:35class, sex, age and fair and then add
  918. 49:38family size and is alone.
  919. 49:41Accuracy goes from
  920. 49:4378.7%
  921. 49:45to 79.4%
  922. 49:48a small gain for from just two extra
  923. 49:51columns.
  924. 49:57So here's one
  925. 50:00there's what here's one note.
  926. 50:02Actually logistic regression can only
  927. 50:05learn bigger is better or bigger is
  928. 50:08worse. It can't learn a sweet spot by
  929. 50:11itself. That's why is alone helps and a
  930. 50:14decision tree
  931. 50:16could catch the sweet spot directly.
  932. 50:21Okay, let's go back to the video.
  933. 50:36>> Next
  934. 50:38assembling.
  935. 50:41Ensemble learnings is a method to
  936. 50:43improve your models performance by
  937. 50:45combining multiple models.
  938. 50:50The basic idea
  939. 50:52prepare multiple models then aggregate
  940. 50:55the predictions.
  941. 50:56By majority vote for classification or
  942. 51:00by averaging for regression.
  943. 51:03The diagram on the right show the
  944. 51:05pattern.
  945. 51:06Input goes into N classifiers C1, C2,
  946. 51:11and CN.
  947. 51:13Each produces an output O1, O2,
  948. 51:18ON.
  949. 51:19We combine them into final output.
  950. 51:23The majority tends to beat any single
  951. 51:25model because the individual mistakes
  952. 51:28cancel out.
  953. 51:31Two main families dominate practice,
  954. 51:34bagging
  955. 51:35and boosting.
  956. 51:37We will cover both in this section.
  957. 51:42Bagging stands for
  958. 51:44bootstrapping aggregating.
  959. 51:46Step one, take the training data.
  960. 51:50Step two, generate many bootstrap
  961. 51:53resamples of it.
  962. 51:55A bootstrap resample is a draw of N
  963. 51:58samples from the original training set
  964. 52:01with replacement.
  965. 52:03So, duplicates are allowed.
  966. 52:06Repeat that B times to get B different
  967. 52:09resample data sets.
  968. 52:12Each resample data set is slightly
  969. 52:14different from the others
  970. 52:16because the random draw picks different
  971. 52:18rows.
  972. 52:20Step three,
  973. 52:22train a one separate model on each
  974. 52:24resample.
  975. 52:25We end up with B trained models.
  976. 52:30Each individual model is a little weaker
  977. 52:34because it only saw noisy subset of the
  978. 52:36data.
  979. 52:37But the combined model aggregated by
  980. 52:40vote or average is often much stronger.
  981. 52:46That is bagging in five steps. Resample,
  982. 52:49train, repeat, aggregate.
  983. 52:55Why does bagging work? The intuition is
  984. 52:59the wisdom of the crowd.
  985. 53:01A graph on the right plots majority vote
  986. 53:04accuracy
  987. 53:05against the number of independent voters
  988. 53:08each with 60% accuracy.
  989. 53:13A single expert at 60% accuracy gives,
  990. 53:17unsurprisingly,
  991. 53:1860% overall accuracy.
  992. 53:22Five independent voters, each at 60%,
  993. 53:26push the majority vote accuracy up to
  994. 53:28about 68%.
  995. 53:3211 voters reach out 75%.
  996. 53:36101 voters reach about 98.
  997. 53:40100 imperfect models can outperform a
  998. 53:43single strong one
  999. 53:46if their mistakes are diverse.
  1000. 53:50The big caveat,
  1001. 53:52if every voter makes the same mistake,
  1002. 53:55voting does not help at all.
  1003. 53:58The diversity of mistakes is what makes
  1004. 54:00the math work.
  1005. 54:05A famous example of bagging is a random
  1006. 54:08forest.
  1007. 54:09Bagging is applied to decision trees,
  1008. 54:12but with one extra twist.
  1009. 54:15Without it, every tree would tend to
  1010. 54:17look at the same.
  1011. 54:19A single strong feature would always be
  1012. 54:22picked for the first split.
  1013. 54:27The extra twist is that at each split,
  1014. 54:30only a random subset of M features is
  1015. 54:33considered as candidates.
  1016. 54:36That forces the tree to look different
  1017. 54:38from each other.
  1018. 54:39We train many such trees and take the
  1019. 54:42average.
  1020. 54:43That is a random forest.
  1021. 54:46Let's see bagging in action.
  1022. 54:48Please scroll to section 5.2.
  1023. 54:51The first code cell applies bagging
  1024. 54:57to K nearest neighbors on the breast
  1025. 54:59cancer data set.
  1026. 55:07We set up two models, a plain K neighbor
  1027. 55:10classifier
  1028. 55:12and bagging classifier. That is, 100 K
  1029. 55:15and models trained on 100 different
  1030. 55:18bootstrap resamples
  1031. 55:20with their predictions aggregated.
  1032. 55:22We fit both on the training set and
  1033. 55:24record training and test scores.
  1034. 55:31The output table shows that the training
  1035. 55:33score barely changes, but the test score
  1036. 55:36improves with bagging.
  1037. 55:38The gap improvement on unseen data with
  1038. 55:41no change on training data is exactly
  1039. 55:44the variance reduction we expect from
  1040. 55:46averaging.
  1041. 55:49Practice question four in this section
  1042. 55:51asks you to try the same thing with
  1043. 55:53logistic regression, a decision tree,
  1044. 55:55and KNN as base learners.
  1045. 56:05Section 5.2.2 moves to random forest to
  1046. 56:08the California housing data set.
  1047. 56:11The code set trains random forest
  1048. 56:13regressor and gradient boost regressor
  1049. 56:16side by side and prints R squared for
  1050. 56:19both.
  1051. 56:21Both score well.
  1052. 56:24Gradient boosting tends to edge out
  1053. 56:26random forest on this data set, which is
  1054. 56:28a preview of the boosting section coming
  1055. 56:30up.
  1056. 56:32The next code cell looks at feature
  1057. 56:34importances.
  1058. 56:36Tree-based ensembles expose this
  1059. 56:38attribute, which ranks features by how
  1060. 56:41much they contributed to split across
  1061. 56:44the whole forest.
  1062. 56:46The bar plot
  1063. 56:50shows that on California housing, net
  1064. 56:52inc
  1065. 56:54median income is overwhelmingly the most
  1066. 56:57important feature
  1067. 56:59followed by avg occup latitude and
  1068. 57:02longitude.
  1069. 57:04House age and the other columns matter
  1070. 57:06much less.
  1071. 57:07That kind of ranking is one of the
  1072. 57:09practical pros of using tree-based
  1073. 57:11ensembles.
  1074. 57:13The other ensemble family is boosting.
  1075. 57:17The basic idea, train a weak model
  1076. 57:20first, then check with example it go
  1077. 57:23wrong.
  1078. 57:26Reweight the data, give more weight to
  1079. 57:29the examples that were misclass-
  1080. 57:35>> Sorry, I will stop the video here and go
  1081. 57:38back to the
  1082. 57:40notebook.
  1083. 57:45Okay, here we will
  1084. 57:48apply bagging.
  1085. 57:51Mhm.
  1086. 57:52The idea is quite simple. From our
  1087. 57:54training data, we randomly
  1088. 57:58we randomly draw a new data set of the
  1089. 58:01same size. Some samples get picked twice
  1090. 58:05and some don't get picked at all.
  1091. 58:08We do this 100 so we get 100 slightly
  1092. 58:12different data sets. Okay, let's apply
  1093. 58:15this.
  1094. 58:20Base models. Okay, here's the logistic
  1095. 58:24regression, then we will apply decision
  1096. 58:28tree.
  1097. 58:42And KNN.
  1098. 58:58Here, we will train the model.
  1099. 59:09Get this model score
  1100. 59:17and apply bagging.
  1101. 59:51Okay, let's run code.
  1102. 59:57Ah, okay. I forgot to
  1103. 1:00:00split the data. Sorry.
  1104. 1:00:05It should be okay.
  1105. 1:00:09It's still running.
  1106. 1:00:38Oops, seems
  1107. 1:00:40it takes quite long time. So, then I
  1108. 1:00:44will directly uh tell you the result.
  1109. 1:00:49Uh so,
  1110. 1:00:51actually, the the tree gains the most
  1111. 1:00:54and uh logistic regression doesn't
  1112. 1:00:58change at all because the logistic
  1113. 1:01:01regression is already very stable. And
  1114. 1:01:04why does uh bagging method help the
  1115. 1:01:07model? Like think of asking 100 people
  1116. 1:01:10instead of one. Each person makes one
  1117. 1:01:13mis- makes some mistakes, but different
  1118. 1:01:16mis- uh but uh
  1119. 1:01:18each person make different mis- mistake.
  1120. 1:01:21When they vote, the mistakes cost a lot.
  1121. 1:01:24That's the idea for the bagging.
  1122. 1:01:29Oops, it's still running. Maybe I will
  1123. 1:01:32just
  1124. 1:01:33stop it.
  1125. 1:01:36As you can see, it's
  1126. 1:01:37take quite long time to run this
  1127. 1:01:40bagging method.
  1128. 1:01:43Okay, let's go back to the video.
  1129. 1:01:56>> The other ensemble family is boosting.
  1130. 1:02:00The basic idea, train a weak model
  1131. 1:02:03first, then check with example it go
  1132. 1:02:06wrong.
  1133. 1:02:09Reweight the data, give more weight to
  1134. 1:02:12the examples that were misclassified,
  1135. 1:02:15and train another model on the
  1136. 1:02:16reweighted data.
  1137. 1:02:20Repeat this step many times.
  1138. 1:02:23Each new model focuses on what the
  1139. 1:02:25previous one got wrong.
  1140. 1:02:27At the end, aggregate the predictions.
  1141. 1:02:31Examples of boosting algorithm include
  1142. 1:02:34AdaBoost, gradient boosting,
  1143. 1:02:37XGBoost, and LightGBM.
  1144. 1:02:41The last two are the workhorses of
  1145. 1:02:43practical machine learning competitions.
  1146. 1:02:47Why does boosting work?
  1147. 1:02:49Let's look at an example depicted to the
  1148. 1:02:51right, where we try to fit a sine curve
  1149. 1:02:55with simple box-shaped functions with
  1150. 1:02:57boosting.
  1151. 1:02:59Round one captures the easy patterns.
  1152. 1:03:03Round two fixes what is still wrong.
  1153. 1:03:08Round three fixes what is left after
  1154. 1:03:10that, and so on.
  1155. 1:03:12At each round, the residual,
  1156. 1:03:15the gap between truth and prediction
  1157. 1:03:18gets smaller.
  1158. 1:03:20The figure on the right shows this on a
  1159. 1:03:22noisy sine curve. After one round, RMSE
  1160. 1:03:26is 0.46.
  1161. 1:03:28After five, 0.26.
  1162. 1:03:31After 100,
  1163. 1:03:330.08.
  1164. 1:03:36One caveat, with too many rounds,
  1165. 1:03:39boosting starts fitting the noise in the
  1166. 1:03:41data.
  1167. 1:03:42The residual on the noise side keeps
  1168. 1:03:45shrinking,
  1169. 1:03:46but generalization gets worse.
  1170. 1:03:49So, boosting has its own overfitting
  1171. 1:03:51risk that you control through the number
  1172. 1:03:54of rounds.
  1173. 1:03:57A concrete example is XGBoost.
  1174. 1:04:01Boosting is applied to decision trees,
  1175. 1:04:04but each tree is shallow,
  1176. 1:04:07depth around three to eight,
  1177. 1:04:09and each new tree predicts the residual
  1178. 1:04:12of the running sum so far.
  1179. 1:04:14The final prediction is the sum of all
  1180. 1:04:17tree outputs.
  1181. 1:04:21In Python, bagging with decision trees,
  1182. 1:04:24import BaggingClassifier from
  1183. 1:04:26sklearn.ensemble,
  1184. 1:04:29a DecisionTreeClassifier
  1185. 1:04:31from sklearn.tree.
  1186. 1:04:34Pass the decision tree as the base
  1187. 1:04:36estimator, set n_estimators equals to
  1188. 1:04:39100,
  1189. 1:04:41and call fit and predict as usual.
  1190. 1:04:46A gradient boosting,
  1191. 1:04:48import GradientBoostingRegressor
  1192. 1:04:51from sklearn.ensemble.
  1193. 1:04:55Instantiate it,
  1194. 1:04:57and then again, call fit and predict.
  1195. 1:05:01The defaults give a reasonable starting
  1196. 1:05:04point.
  1197. 1:05:07XGBoost has its own Python package.
  1198. 1:05:10From XGBoost, import XGBRegressor.
  1199. 1:05:14The example here trains for up to 1,000
  1200. 1:05:17rounds, watches RMSE on the validation
  1201. 1:05:21set and stops early.
  1202. 1:05:24With early stopping rounds equals to 20,
  1203. 1:05:28the training iteration stop if
  1204. 1:05:30performance is not improved for 20
  1205. 1:05:33consecutive rounds.
  1206. 1:05:37The early stopping rounds parameter is
  1207. 1:05:40the practical antidote to the
  1208. 1:05:42overfitting caveat we just saw.
  1209. 1:05:46Now, let's see boosting in a notebook.
  1210. 1:05:48Please scroll to section 5.5.
  1211. 1:05:52The first code cell compares
  1212. 1:05:56applying decision tree regressor against
  1213. 1:05:59AdaBoost regressor. With the decision
  1214. 1:06:01tree as the base learner, both on
  1215. 1:06:03California housing.
  1216. 1:06:05The output shows train and test R
  1217. 1:06:07squares side by side.
  1218. 1:06:09AdaBoost typically lifts the test score
  1219. 1:06:12over the single tree because a sequence
  1220. 1:06:14of tree corrects errors that any single
  1221. 1:06:17tree could not.
  1222. 1:06:22Section 5.7 moves to XGBoost.
  1223. 1:06:25The first code cell in the simplest
  1224. 1:06:27possible setup.
  1225. 1:06:29XGB regressor.
  1226. 1:06:32Fit, predict, score.
  1227. 1:06:39The printed test R squared gives the
  1228. 1:06:41baseline.
  1229. 1:06:45The next code cell adds early stopping.
  1230. 1:06:48We split the training data in the
  1231. 1:06:50smaller training piece and a validation
  1232. 1:06:52piece.
  1233. 1:06:54Then train XGB regressors.
  1234. 1:06:58The eval stopping rounds
  1235. 1:07:02argument tells XGBoost to watch RMSE on
  1236. 1:07:06the validation set after every round
  1237. 1:07:09and stop as soon as it has not improved
  1238. 1:07:12for 10 consecutive rounds.
  1239. 1:07:15Running this cell
  1240. 1:07:16produces a long stream of log lines.
  1241. 1:07:20One per round.
  1242. 1:07:25Which it scroll past.
  1243. 1:07:29The number that matters is the final
  1244. 1:07:30test R squared printed at the bottom.
  1245. 1:07:33It beats the simple version because
  1246. 1:07:36early stopping picks the right number of
  1247. 1:07:38rounds automatically.
  1248. 1:07:41Here is the practical summary table for
  1249. 1:07:43assembling.
  1250. 1:07:46Use bagging, specifically random forest,
  1251. 1:07:49when you want to avoid overlearning or
  1252. 1:07:52as a baseline early in a project when
  1253. 1:07:55you have not yet identified useful
  1254. 1:07:57features.
  1255. 1:07:59Use the more general bagging classifier,
  1256. 1:08:02a bagging regressor, if you want to
  1257. 1:08:05apply bagging to non-tree based learner
  1258. 1:08:08like K nearest neighbors, a logistic
  1259. 1:08:11regression.
  1260. 1:08:13Use boosting, specifically XGBoost or
  1261. 1:08:17gradient boosting,
  1262. 1:08:19when you want to push accuracy as far as
  1263. 1:08:21it will go, especially you have already
  1264. 1:08:24found useful features through EDA.
  1265. 1:08:27AdaBoost is a classic variant.
  1266. 1:08:31And use no ensemble,
  1267. 1:08:34just a standalone model,
  1268. 1:08:36when interpretability of the result
  1269. 1:08:38matters more than raw accuracy.
  1270. 1:08:42A single decision tree or linear model
  1271. 1:08:44is much easier to explain to a
  1272. 1:08:46stakeholder.
  1273. 1:08:50Let's wrap up.
  1274. 1:08:51First, feature engineering can hugely
  1275. 1:08:54impact on the model's performance.
  1276. 1:08:57Feature transformation includes
  1277. 1:08:59standardization,
  1278. 1:09:01normalization, log, and then encoding.
  1279. 1:09:05Feature creation includes combining
  1280. 1:09:07features and using domain knowledge.
  1281. 1:09:11Second, feature engineering is
  1282. 1:09:14iterative.
  1283. 1:09:16Expect several trial and error loops
  1284. 1:09:18between feature design and model
  1285. 1:09:20evaluation.
  1286. 1:09:22There is no single recipe that works for
  1287. 1:09:24every data set.
  1288. 1:09:26And third,
  1289. 1:09:28ensembling is very useful in real-world
  1290. 1:09:31application.
  1291. 1:09:32Bagging methods, such as random forest,
  1292. 1:09:36and boosting methods, such as AdaBoost,
  1293. 1:09:39XGBoost, and LightGBM, are the first
  1294. 1:09:42choice.
  1295. 1:09:44That concludes this lecture on feature
  1296. 1:09:46engineering.
  1297. 1:09:47Thank you very much for listening, and I
  1298. 1:09:50hope you enjoyed it.

About this transcript

This page contains the full transcript of GCI World 2026 September Session8 During Lecture by GCI World (Matsuo-Iwasawa Lab), generated from the public captions YouTube serves with the video. The transcript has 6,259 words across 1,298 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.