#DeltaTable

Mimoune Djouallah

@mimdj.bsky.social

2 months ago

a sql script that export #deltatable metadata from a #ducklkake db

never thought that #sql can do that !!!!

github.com/djouallah/Fa...

Mimoune Djouallah

@mimdj.bsky.social

4 months ago

Explaining how Python engines read and write #DeltaTable is not for the faint of heart.
The theory is everything will depends on the delta kernet rust for read and write, but we are not there yet
github.com/djouallah/Fa...
#duckdb #delta_rs #datafusion #chdb #daft #polars #rust #lakesail

Mimoune Djouallah

@mimdj.bsky.social

8 months ago

Fabric_Notebooks_Demo/ducklake/Export_delta_metadata.ipynb at main · djouallah/Fabric_Notebooks_Demo Fabric Python Notebooks examples. Contribute to djouallah/Fabric_Notebooks_Demo development by creating an account on GitHub.

POC, export #ducklake metadata to #deltatable format, #apacheSpark and Fabric DWH, #PowerBI can read it
github.com/djouallah/Fa...

Mimoune Djouallah

@mimdj.bsky.social

11 months ago

How to Read a Delta Table with Deletion Vectors and Column Mapping in Python When attempting to read a Delta table using Python with the deltalake library (Delta_rs, not Spark), you may encounter the following error: import deltalake DeltaTable(‘/lakehouse/default/Tab…

How to Read a #DeltaTable with Deletion Vectors and Column Mapping in #Python

datamonkeysite.com/2025/03/19/h...
#Microsoftfabric #onelake #duckdb #deltatlake

Mimoune Djouallah

@mimdj.bsky.social

1 year ago

I don't think it was publicized a lot, but #Snowflake can read #DeltaTable as an #Iceberg table too, basically in this example, I am writing to #onelake and reading existing data too, generated by #Spark

Mimoune Djouallah

@mimdj.bsky.social

1 year ago

#duckdb #pyiceberg #apachepolaris #MicrosoftFabric #Onelake #deltatable #PowerBI

I can do anything I want 😁

Mimoune Djouallah

@mimdj.bsky.social

1 year ago

Smart Data Pipeline Design: Check for Delta Table Changes with Minimal Overhead Scenario I have a notebook that processes hot data every 5 minutes. Meanwhile, another pipeline processes historical data, and I want to create a summary table that uses the hot data incrementally …

using #deltatable Python packages to detect changes downstream, and process data either incrementally or full refresh
#Python #MicrosoftFabric #onelake
datamonkeysite.com/2024/11/10/s...

Mimoune Djouallah

@mimdj.bsky.social

1 year ago

Reading #deltatable from #Unitycatalog using #daft, it is read only for now.

running in #MicrosoftFabric Notebook
it is already supported using Mirrored Unity catalog in #onelake but it is nice to have more options, specially thinking of write support.

github.com/djouallah/Fa...