Session: Incremental data processing: A path to streamlined data engineering pipelines | Open Source Data Summit

Session: Incremental data processing: A path to streamlined data engineering pipelines

Prasad Pathak, Data Engineer @ Tesla

In this session, we invite you to explore the paradigm shift brought about by Incremental data processing. While the allure of 'Complete data processing' remains, Incremental processing offers a compelling alternative. We will delve into the core principles of Incremental data processing, illuminating its benefits, including accelerated data processing, reduced operational costs, and enhanced data quality.

Along this journey, we'll navigate the challenges inherent to Incremental processing and unveil best practices to mitigate them. Through a detailed example design pipeline, we will illustrate how Incremental data processing can be seamlessly integrated into your existing data engineering architecture, empowering organizations to stay competitive in an era driven by data. Join Prasad to unlock the transformative potential of incremental data processing and revolutionize your data engineering pipelines.

Trying to dive deeper into incremental data processing?

With incremental processing, Uber also recently improved processing speed and cut compute costs by 80%. This blog covers how Uber set their transactional data lake in motion with incremental ETL: https://www.uber.com/blog/ubers-lakehouse-architecture/
This whitepaper from Onehouse covers how a universal data lakehouse allows for a move from traditional batch processing to a managed cloud service for incremental data processing: https://www.onehouse.ai/whitepaper/onehouse-universal-data-lakehouse-whitepaper

Where & when?

Open Source Data Summit 2024 was held on October 2nd, 2024.

What is the cost of access to the live virtual sessions?

OSDS is always free and open for all to attend.

What is Open Source Data Summit?

OSDS is a peer-to-peer gathering of data industry professionals, experts, and enthusiasts to explore the dynamic landscape of open source data tools and storage.

The central theme of OSDS revolves around the advantages of open source data products and their pivotal role in modern data ecosystems.

OSDS is the annual peer hub for knowledge exchange that fosters a deeper understanding of open source options and their role in shaping the data-driven future.

Who attends OSDS?

OSDS is attended by data engineers, data architects, developers, DevOps practitioners and managers, and data leadership.

Anyone who is looking for enriched perspectives on open source data tools and practical insights to navigate the evolving data landscape should attend this event.

On October 2nd, 2024 we convened for discussions about:

Benefits of open source data tools
Cost/performance trade-offs
Building data storage solutions
Challenges surrounding open source data tool integration
Solutions for the cost of storing, accessing, and managing data
Data streams and ingestion
Hub-and-spoke data integration models
Choosing the right engine for your workload

Are you interested in speaking or sponsoring the next Open Source Data Summit?

Submit a talk proposal here or reach out to astronaut@solutionmonday.com.

Register for on-demand access to the OSDS 2024 sessions and announcements about OSDS 2025

"*" indicates required fields