Amazon SageMaker: Data Quality in Visual ETL | Amazon Web Services — Transcript

Full transcript

1. 0:00Hi everyone. I'm Armatus
2. 0:02senior solution architect specialist for
3. 0:04analytics at AWS. Today, I'll show you
4. 0:07how to validate data quality right
5. 0:10inside a visual ETL pipeline on Amazon
6. 0:12Sagemaker Studio.
7. 0:14I start in visual ETL with a source, a
8. 0:17table from the glue data catalog.
9. 0:20Here, a Titanic data set in CSV.
10. 0:24The data preview lets me see the records
11. 0:26I'll be working with before I build
12. 0:28anything.
13. 0:29Next, I add evaluate data quality
14. 0:31transform and connect it to my source.
15. 0:34This is where I define the data quality
16. 0:36rules the data has to meet.
17. 0:41I'll check that our required column
18. 0:43exists.
19. 0:49The node also lets me enable anomaly
20. 0:52detection, which learns my data's normal
21. 0:56shape over time and flags a statistical
22. 0:58drift. And I can put these results, rule
23. 1:01level, pass fail metric to cloud watch
24. 1:03and our result summary to S3 as an audit
25. 1:06trail. I pointed the S3 output at a
26. 1:08location in my project bucket.
27. 1:14Let me run it.
28. 1:17The job starts.
29. 1:22And when it finished, the data quality
30. 1:24tab shows the outcome. Overall pass,
31. 1:27one of one rule pass.
32. 1:30100% quality score.
33. 1:32And because this is Sagemaker Studio,
34. 1:35the score is tracked over time natively.
35. 1:38No separate BI tool.
36. 1:41Now, I'll add a second rule.
37. 1:43The age field must not be empty.
38. 1:47Real data set have gaps.
39. 1:49And this is exactly the kind of checks
40. 1:52that catches them.
41. 2:02I run again
42. 2:03and this time the job fails.
43. 2:13The data quality tab shows 50% one of
44. 2:17two rows passed.
45. 2:18The age rule found blank values.
46. 2:21With a stop job on failure on the job
47. 2:25stops before loading. So, bad data never
48. 2:29reaches the target.
49. 2:30But stopping the whole job isn't always
50. 2:32what I want. Instead, the data quality
51. 2:35node can emit asset rows
52. 2:37and failed rows as separate outputs.
53. 2:40Already split for me, no filter needed.
54. 2:43Good reports go one way, the records
55. 2:45that failed the check go another.
56. 2:47I'll send each branch to its own
57. 2:50destination. Here, two S3 targets
58. 2:54writing parquet.
59. 2:55The failed rows land in a quarantine
60. 2:58location for review.
61. 3:00The passed rows continue down the normal
62. 3:02path.
63. 3:03The pipeline
64. 3:05keeps flowing instead of blocking.
65. 3:12Across my runs, I get aggregated metrics
66. 3:15and a data quality history. The quality
67. 3:17score and score by rule set over time.
68. 3:28And I'm not limited to the visual
69. 3:30editor. So, when from my notebook, I
70. 3:32read the outputs directly. Here, pandas
71. 3:35reading the passed and failed parquet
72. 3:37files is straight from S3.
73. 3:40Now, I can act on them in code.
74. 3:43Inspect all the records that failed,
75. 3:45reprocess them, or trigger a downstream
76. 3:49step.
77. 3:49>> [clears throat]
78. 3:50>> The same data quality signal drives both
79. 3:53the no-code pipeline and my custom
80. 3:56logic.
81. 3:57Thank you for watching.