Overview
-
Learn how to use Python tools and techniques to solve one of the main challenges data scientists face: getting good data to train their algorithms.
Syllabus
-
Introduction
- Why is data inegstion important?
- What you should know
- Using the exercise files
- Overview of data scientists work
- Where does data come from?
- Different types of data
- The data pipeline (ETL)
- Final destination (data lake)
- Working in CSV
- Working in XML
- Working in Parquet, Avro, and ORC
- Unstructured text
- JSON
- Challenge: CSV to JSON
- Solution: CSV to JSON
- Working with JSON
- Making HTTP calls
- Processing event-based data
- Challenge: Location from IP
- Solution: Location from IP
- Try to find an API
- Working with Beautiful Soup
- Working with Scrapy
- Working with Selenium
- Other considerations
- Challenge: GitHub API
- Solution: GitHub API
- What are schemas?
- Working with ontologies
- What should be in schema
- Schema changes
- Schema validations
- Types of databases
- Hosted and cost of ops
- Working with relational databases
- Working with key or value databases
- Working with document databases
- Working with graph databases
- Challenge: ETL
- Solution: ETL
- Data is never 100% okay
- Causes of errors
- Filling missing values
- Finding outliers (manual)
- Finding outliers (ML)
- Challenge: Clean rides according to ride duration
- Solution: Clean rides according to ride duration
- Design your data
- KPIs
- What to monitor?
- Next steps
