Course Overview
In the modern world, every organization relies on data more every day. Businesses tend to deal with large amount of data which could be spread across various sources and to effectively analyze it – it is crucial to consolidate it into one key location. The modern era posed us with many challenges – especially in the Data sector as it is getting harder to connect all the dots together. Microsoft’s Azure Cloud is a great solution that makes it easier to manage and analyze the data that we deal with across various sources.
All exercises are based on Azure Cloud. with additional Linux and virtual network appliances that covers common scenarios. After the workshop, you will receive PowerPoint slides, tools and lab instructions.
Who should attend
Data engineers, developers, IT professionals, database administrators, security consultants and other people responsible for creating, implementing and maintaining cloud data platform solutions.
Course Objectives
This intense hands-on training if perfect for anyone that has experience with Windows Servers, SQL Servers and deals with or is interested in data management and analysis. During the training, you will gain hands-on experience on how to choose and work with the compute and storage options for data engineering workloads, deep-dive into Azure Data Lake and Databricks. Moreover, you will also go through the key aspects of Azure Synapse Analytics and you will have the chance to become the master of Hybrid Transactional Analytical Processing. By the end of the training you will also gain practical knowledge on data transformation and data movement orchestration using Azure Data Factory and Realtime stream processing!
The training is though and authored by one of the best Experts in the data field – Microsoft’s Data Platform MVP Damian Widera.
Course Content
Module 1: How to choose and work with the compute and storage options for data engineering workloads
- Microsoft Azure services for data engineering
- Reference architecture for the data engineering workloads
- Azure Storage account configuration
- Secure data in the Azure Storage
Module 2: Explore and design Data Lake
- Why do we need a Data Lake
- Security design of a data lake
- Data Lake performance
Module 3: Benefits and challenges of the Delta Lake
- Why do we need to create a Delta Lake
- Usage patterns of the Delta Lake in data engineering workloads
- Challenges of the Delta Lake – transaction handling
Module 4: Azure Databricks
- Create and configure the cluster
- Perform data engineering tasks – work with data frames, perform ETL operations in Scala and Python
- Integrate Databricks with CosmosDB, Azure SQL Database and Azure Storage
- Working with the Delta Lake
Module 5: Azure Synapse Analytics – serverless pool
- Prepare for the explorative analytics
- Working with the metadata
- Secure data and manage users
Module 6: Azure Synapse Analytics – Apache Spark pool
- Ingest data with Apache Spark
- Data transformation
- Connection to SQL pool
Module 7: Azure Synapse Analytics – dedicated pool
- Configure Azure Synapse Analytics with the Dedicated Pool
- Security considerations during the data ingestion
- Security considerations during data processing and retrieval
Module 8: Understand the Hybrid Transactional Analytical Processing
- Configure Azure Synapse Link with Azure CosmosDB
- Query Azure CosmosDB with Apache Spark and serverless pool
Module 9: Data transformation and data movement orchestration using Azure Data Factory
- Create connections to data services
- Create data flows and pipelines
- Perform data transformation and enrichment
- Secure the solution
Module 10: Realtime stream processing
- With Azure Stream Analytics
- With Event Hubs and Azure Databrick