devxlogo

Databricks introduces LakeFlow at Data + AI Summit

Databricks introduces LakeFlow at Data + AI Summit

LakeFlow Summit

Databricks, a prominent data and AI software company, has introduced LakeFlow, a comprehensive data engineering solution designed to improve data pipeline construction. The announcement was made at the annual Data + AI Summit. LakeFlow marks a shift from Databricks’ previous reliance on partners like Fivetran, Rudderstack, and dbt for data preparation and loading tools.

It offers a native solution within the Databricks platform that manages data ingestion, transformation, and orchestration. The move came in response to feedback from the Databricks CIO Forum. Two years ago, the advisory board expressed a desire for more robust data ingestion capabilities from various SaaS applications and databases.

This revealed that many enterprises were building custom solutions for data ingestion due to specific security needs and edge cases. LakeFlow consists of three core components:

LakeFlow Connect provides connectors between various data sources and the Databricks platform. It integrates with Databricks’ Unity Data Catalog for data governance and currently supports databases like SQL Server and MySQL, and enterprise applications such as Salesforce, Workday, and Google Analytics.

Databricks unveils LakeFlow solution

LakeFlow Pipelines, built on Databricks’ Delta Live Tables framework, facilitates data transformation and ETL processes in SQL or Python. It offers a low-latency mode and incremental data processing for efficient data synchronization.

LakeFlow Jobs handles automated orchestration and ensures data health and delivery. It supports various tasks, such as updating dashboards and training machine learning models. The introduction of LakeFlow reflects Databricks’ commitment to reducing operational complexity for its customers by offering an integrated data ingestion and transformation solution.

This aligns with the broader industry trend of cost reduction and service consolidation among enterprises. Databricks is rolling out LakeFlow in phases, with LakeFlow Connect available for preview soon. Interested users can sign up for the waitlist on the Databricks website.

See also  Nvidia's growth prospects questioned amid surge

The company’s focus remains on its partner ecosystem, but the integration of LakeFlow aims to address specific customer demands for a seamless, all-encompassing data engineering solution.

devxblackblue

About Our Editorial Process

At DevX, we’re dedicated to tech entrepreneurship. Our team closely follows industry shifts, new products, AI breakthroughs, technology trends, and funding announcements. Articles undergo thorough editing to ensure accuracy and clarity, reflecting DevX’s style and supporting entrepreneurs in the tech sphere.

See our full editorial policy.

About Our Journalist