Javi Santana

Why we maintain a ClickHouse fork at Tinybird (and how it's different) We've long contributed to the open source ClickHouse community, and we didn't make this decision lightly. Here's why we maintain our own private ClickHouse fork.

Why we maintain a ClickHouse fork at Tinybird (and how it's different)

www.tinybird.co/blog-posts/w...

I wanted to play a little bit and tried to do it without sharding, I’ll publish the blog post in the following weeks. This one is playing “easy mode”

t.co/DDckrn8cw5

I wrote about how to ingest 1b rows per second with ClickHouse.

The point of the post is not the number, pushing 1b rows is “easy” the actual challenge comes when you need to do that in a reliable way, that’s what I try to explain

python and c++

We use go based tools but not as a language in our product

Blog · Tinybird Musings on transformations, tables and everything in between.

- We are using Tinybird to handle autoscaling (instead of Prometheus)
- Scaling up ingestion in an OLAP workload
- Our opinionated way for compute-compute separation
- How we optimized ingestion with C++ (not Rust, sorry)
- Cutting costs with Karpenter

www.tinybird.co/blog

I'm fucking tired of clickbaity content, people writing bad content just for SEO or not going into details because "people have a short attention span". Maybe that's true but some people appreciate long-form, detailed and super technical content.

Today we posted 5 high quality engineer posts:

Lessons learned from 5 years operating huge ClickHouse® clusters: Part II This is the second part of the series. Here's more of what I've learned from operating petabyte-scale ClickHouse clusters for the last 5+ years.

The second part of "handling clickhouse clusters at petabyte scale" www.tinybird.co/blog-posts/w...

I've operated petabyte-scale ClickHouse® clusters for 5 years I've been operating large ClickHouse clusters for years. Here's what I've learned about architecture, storage, upgrades, config, testing, costs, and ingestion.

We've been operating petabyte-scale cluster for 5 years now, especially dealing with low latency use cases with large amounts of data, some of them under 10ms (for reference, most big data systems latency is 1 second at best).

www.tinybird.co/blog-posts/w...

Thanks Gunnar

Thanks Nico

We need a new git A few years ago continuous integration systems were not a thing.

We need faster feedback loops

failingwithdata.substack.com/p/we-need-a-...

Tinybird Forward » - The fastest way to ship software with big data requirements | Product Hunt Tinybird Forward is a new evolution of the Tinybird user experience. Build software with big data requirements faster and with fewer dependencies. Test locally, bootstrap with a prompt, deploy with on...

I'd appreciate if you could upvote us in ProductHunt. It's about our latest product iteration

www.producthunt.com/posts/tinybi...

Introducing Tinybird Forward: Ship software with big data requirements. YouTube video by Tinybird

🌅 Tinybird Forward is here »

Forward is a major evolution of Tinybird, designed to make shipping software with big data requirements faster and more intuitive.

No complex infra project. No context switching. No esoteric architectures. Just code.

(🔊 sound on! 👇)

youtu.be/vaSjWu3XFdY

You need developers that care. I think they call it "being accountable".

Today we are releasing a new Tinybird, thanks to every single one spending your time on keeping this running.

curl tinybird.co | sh

Maintenance is not fun. You need people that make it fun, developers that write reports you actually want to read about the most boring fix.

Alerting is tricky and you need to fiddle for weeks until you nail it.

Every new customer makes the development team slower. Maybe it's just 1 hour a month, that's 0.01% of a developer's time in a 40 developer team. Seems low but it adds up.

That small issue you don't have time to fix is blocking someone.

Every single day there is a moment you remember there are thousands of requests coming into your system every second.

You spend 1 month chasing that bug impossible to reproduce.

You fail and you need to explain it to your customers that lose money because of you.

When you're in production, things fail, users complain, oncall alerts wake you up in the middle of the night, incidents happen. Those endless video conferences nobody leaves until they are 100% sure not a single client is still affected.

Production hurts.

If it doesn't, it means you didn't try hard enough.

A new LLM version release with a new super powerful option: --schema

An mini BI example, in this case I'm querying a parquet file with all the events coming from gitlab tickets webhook

I've helped huge companies scale logs analysis. Here’s how. I've spent years optimizing logs explorers across multiple domains with trillions of logs to process. Here's what I've learned about building a performant logs analytics system.

And the post www.tinybird.co/blog-posts/r...

Data intensive SaaS products will love to leave your data in icerbeg/s3

Guess who is going to pay all the storage and operation costs? 🤣

You can subscribe and get updates in our blog www.tinybird.co/blog

That's the benefit of having a data engineering team focused 100% on complex data-intensive applications.

And posts like this will keep coming, I'm a little bit tired of "developer marketing" with things that explain the surface and don't get into deep technical details

We’ve got a deep technical post almost ready on real-time log analytics, based on our experience designing these systems with companies handling trillions of rows.

Nop, general available models

You'll need to iterate a few times until you have the data you need but it's better than spending a few hours working on a script.

So it generates data that fits the table schema. But usually, you want the data to have a particular shape, so you can give some instructions to the command:

𝚝𝚋 𝚖𝚘𝚌𝚔 𝚝𝚊𝚋𝚕𝚎 --𝚙𝚛𝚘𝚖𝚙𝚝 "𝚍𝚊𝚝𝚎𝚜 𝚜𝚑𝚘𝚞𝚕𝚍 𝚋𝚎 𝚒𝚗 𝚂𝚎𝚙𝚝𝚎𝚖𝚋𝚎𝚛 𝟸𝟶𝟷𝟾"

We are using LLMs in the next Tinybird iterations but we are trying to be subtle, not going with the conversational interface. So for example, when you want to create synthetic data to test your logic you can do

𝚝𝚋 𝚖𝚘𝚌𝚔 𝚝𝚊𝚋𝚕𝚎