Session: Choosing an open table format for your transactional data lake

Shana Schipers, Principal Specialist SA, Analytics @ AWS

Join Shana for an explorative talk on the burgeoning popularity of transactional data lakes within modern data platforms. These innovative solutions, such as those supported by AWS, have unlocked a realm of possibilities, making previously challenging tasks like Change Data Capture (CDC) and GDPR compliance seamlessly achievable. By embracing open-table formats like Apache Hudi, Apache Iceberg, and Delta Lake, organizations can combine the power of transactional capabilities with the flexibility of Amazon S3 data lakes, all at scale. However, with numerous options available, there's no "one size fits all" solution.

This session aims to be your compass in this journey, offering guidance on selecting the right table format for your specific use case. We'll explore essential criteria and attention points to accelerate your decision-making process and ensure the success of your data lake endeavors.

Interested in learning more about open table formats?

Check out the benchmarks for Delta, Hudi, and Iceberg: https://medium.com/@kywe665/delta-hudi-iceberg-a-benchmark-compilation-a5630c69cffc
Read about feature comparisons between Hudi, Delta Lake, and Iceberg: https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison
Read about project popularity between Delta, Hudi, and Iceberg: https://medium.com/@kywe665/delta-hudi-iceberg-which-is-most-popular-29ca56767199
Learn more about a new open source project OneTable which prices omnidirectional interoperability between all of the open table formats: https://onetable.dev/

Where & when?

Open Source Data Summit 2024 was held on October 2nd, 2024.

What is the cost of access to the live virtual sessions?

OSDS is always free and open for all to attend.

What is Open Source Data Summit?

OSDS is a peer-to-peer gathering of data industry professionals, experts, and enthusiasts to explore the dynamic landscape of open source data tools and storage.

The central theme of OSDS revolves around the advantages of open source data products and their pivotal role in modern data ecosystems.

OSDS is the annual peer hub for knowledge exchange that fosters a deeper understanding of open source options and their role in shaping the data-driven future.

Who attends OSDS?

OSDS is attended by data engineers, data architects, developers, DevOps practitioners and managers, and data leadership.

Anyone who is looking for enriched perspectives on open source data tools and practical insights to navigate the evolving data landscape should attend this event.

On October 2nd, 2024 we convened for discussions about:

Benefits of open source data tools
Cost/performance trade-offs
Building data storage solutions
Challenges surrounding open source data tool integration
Solutions for the cost of storing, accessing, and managing data
Data streams and ingestion
Hub-and-spoke data integration models
Choosing the right engine for your workload

Are you interested in speaking or sponsoring the next Open Source Data Summit?

Submit a talk proposal here or reach out to astronaut@solutionmonday.com.

Register for on-demand access to the OSDS 2024 sessions and announcements about OSDS 2025

"*" indicates required fields