🔥 Woah, just learned about Taleshape for the first time!
I'm working on a greenfield product and am _very_ curious about using it. We have Metabase right now for internal BI but this looks great for at least the application feature side of things for our customers.
It blows my mind every day that the software I use the most day to day is almost 50 years old. There's also not an editor out there that has the higher potential for LLM integration, given every buffer is programmable by default. #emacs
I got a line on some primo pure uncut acetominaphen
I was very much hoping this wasn't going down a dark path.
I really wish it didn’t feel like people are hoping it was completely malicious. 😞
Freedom, another member of the Ruby Central Board, wrote this post, but it closely aligns with my feelings and experiences as well.
apiguy.substack.com/p/a-board-me...
It really is a problem that people that are so intelligent and innovative in one domain think their domain knowledge transfers to make them an authority on an unrelated topic. 😔
I’m not sure how much I like the new Liquid Glass bit from apple. The screen kinda feels more cluttered but maybe that’s a boomer take.
That being said, the animations are snappier and things feel more responsive. That is what I love about iOS 26 so far.
I’m definitely onboard with this take.
It feels like they’re not “design patterns” as much as “collections of behavior” helping you discover (not build into!) a specific role/set of responsibilities that behavior defines. They’re heuristics for naming behavior not implementation guides
Not really. I looked into it but I just had a hard time shifting away from the asset focused model Dagster has to a pipeline focused one like Prefect. I'd like to experiment with it one day though cause it looks cool!
It helps our company is a year old and the data platform is of course very green. There maybe a world where this stack has unforeseen limitations that’ll impact us, but that’s hard to imagine with what we know now.
To be clear, DuckDB is our metadata database as well as our primary engine.
Postgres is one possibility for that but we’re just using a DuckDB since it’s simpler and cheaper.
We felt that Dagsters focus on materialized assets instead of the DAG itself fit our model better. The baked in metadata management, lineage, and scheduling capabilities were also a big plus. Airflow is incredible software but the overhead for managing these things is too high for our small team.
MotherDuck is providing the storage for DuckLake parquet files (in their s3 buckets currently although you can BYO) and providing compute instances. They have a feature called read scaling that allows you to federate read only duckdb compute instances to specific db share which is useful for us.
Dagster is orchestrating all our pipelines end to end. We're pulling data from APIs, working on app db ingestion, (soon) orchestrating LLM training and (soon) isolated federated db shares. DBT is just the modeling part of our pipeline atm.
For large datasets the metadata is so useful because you get things like schema evolution, time travel (querying the db as it was in the past), and a few other features. DuckLakes format is based on a database storing the metadata and iceberg is based on parsing actual files at query time.
DuckLake is an open table format similar to Apache Iceberg. So your data is stored as files and both Iceberg and DuckLake store additional data file metadata which act as a mapping to the to the actual data files. The difference is, iceberg is based on files and DuckLake is based on a DB.
There’s a dagster-dbt integration library that exposes the DBT models and metadata to be translated into Dagster assets + metadata. It’s a pretty seamless integration. You even get the DBT tests as asset checks 😎. Dagster then just runs the DBT binary during materialization so it just works.
There's a bit of a learning curve with this stack, especially if you come from tools Airflow, Spark, or Trino. But once you're able to understand it, it's surprisingly simple. This is one of the things I'm most proud to have been working on in my career.
Dagsters "software defined assets" model is such a good heuristic for managing data orchestration pipelines. It's architecture is also incredibly simple and easy to maintain. Controlling dependencies, visualizing lineage, and integrations with tons of other tech make it a joy to work with.
It's a very simple stack whose core philosophy is _just write SQL_. DuckDB makes managing the data _easy_ and allows us to ingest sources natively from just about anywhere we'd need to. Using DuckLake also means we have almost 1:1 parity with an Iceberg stack w/o the overhead of metadata files.
I've been leading the data infrastructure efforts at my job (I used to work as a Data Engineer in big tech) and the stack we've landed on is so enjoyable to work with.
@dagster.io - Orchestration
@duckdb.org - Database
@motherduck.com - Data Warehouse/storage
DBT - Data modeling
I’ve been seeing Helix pop up more and more in readme.mds for various local dev tooling… looks cool!
One piece of tech I’ve been nerding out on this year has been DuckDB. This is one of the reasons why:
> …you could enrich a local customers table with user behavior data from a 10-billion-row Parquet file stored on S3–all in one SQL query..
motherduck.com/blog/pg-duck...
Oof been off this app way to long. X is such a shit show, but tech Twitter yields enough interesting stuff I haven't been able to pry myself away....
Ahh incredible. Thank you. I guess I _did_ mis-understand after all lol.
Another question for developers who’ve started an app:
- How did you handle payments? Specifically for subscriptions.
Reading through Stripe pricing info it seems like pricing for “recurring billing” capability is ~$600 USD per month.
Any chance I misunderstood this?
Ohhh love the lifetime access. I’ve seen a few that are $10-15 a month for like 10,000 unique visits which is annoying. Thanks for the suggestion!
To all my friends that have built and launched an app I need some suggestions:
What did you use for:
- Creating your Privacy Policy?
- Creating your ToS Agreement?
- Creating that annoying cookie opt-in-out thing?
I'm building a rails app and need to implement these things and would love thoughts
