Blue Bots, Done Quick!

Hmm, no, those wouldn't be too hard to add but I had no idea they existed - I don't think those were in the tracery parser I converted from CBDQ. Supported modifiers are replace, capitalize, capitalizeAll, a, s, firstS, and ed.

Everything's still one database and one table, but instead of selecting every bit that needs posting I use modulo and only select rows WHERE id % (x) = (y) where x and y are shell arguments, so that I can set up x cron jobs and pass different y arguments to them to target different sections of db.

Ah, I saw this commit github.com/v21/traceryh..., but looking closer at it I see that it was because of the Twitter API changes, and only two months before CBDQ shut down

Hmm, looking at your repos, am I correct that you originally had a ten minute interval but you changed the minimum to thirty later? Any reason you did that?

(If I did the same, it would reduce the number of total posts by a third...)

Ugh, thanks, fixed it now. Had some changes on a git branch I assumed was on a different branch, so when I tried fixing a bunch of stuff a few days, that overwrote the last handful of changes to that file.

Update: I think I found a way to solve this. Basically I'm now splitting the database in five parts and running five smaller cron jobs instead of one big one. I didn't expect that change to have a huge effect, but it works surprisingly well so far.

Okay, I think all bots are running as normal again now. And I *hope* the server won't crash as a result of that. Let's keep it running all night and find out!

(I sincerely hope that when you say that you supported via Patreon, you don't mean that you were actually charged money for that, because my records show zero paying patrons and I've been assuming that Patreon has been truthful about that.)

... yeah, okay, that's not really an acceptable fix. A quick calculation of my own hourly bot's performance for the past 19 hours reveals that it has posted on average once every 137 minutes. Which is... very much more than sixty.

Thanks for your patience while I ponder on the best way to fix this.

I guess one way to do it could be allowing replies, but only checking for replies once or twice an hour instead of every five minutes.

Sure, this is still a hypothetical, and the best case scenario would still be to not have to add these restrictions at all.

I'm on PHP, no Node installed. I've had multiple cron jobs running at the same time until now, and most of the time that works okay, but yeah, it takes tons of memory and has caused quite a few server crashes by now, so I would prefer keeping the concurrency as low as possible.

I'm experimenting with small changes to the script that publishes your posts to Bluesky to reduce future crashes. At the moment, a side effect of said experimenting is that quite a few bots will post less frequently than usual. Hopefully, this is only temporary until I come up with something better.

(Until now I have been running multiple cron jobs at the same time, always starting a new job every five minutes regardless of whether the last job was finished or not. I changed that now so that all jobs terminate after five minutes, which helps server load, but that way not all bots will post...)

That's the bottleneck, yeah. Have spent the evening testing, tweaking and measuring, and it looks like the tracery and database updates take virtually no time at all, but on average a bot takes >4 seconds to authenticate and 6.5 seconds to post, basically setting a cap at ~360 posts an hour...

Oooh, I definitely wanna chat performance stuff, I seriously don't know what I'm doing here, so any advice you can give on improvements is strongly appreciated.

Okay so far the response to this post has been overwhelmingly positive, and I refuse to believe that is actually representative, please someone tell me you hate it too before I spend way too much time implementing it just to watch it crash and burn on launch 🙈

It feels like a nuclear option because it, by design, would kill off or neuter a ton of great bots, but it would definitely make the remaining bots run smoother.

BBDQ bots are now posting to bluesky ~1400-1500 times/hour, not including replies. Reducing that number by 80-90 % would help me A LOT.

Also kinda wondering and a little afraid to ask, but another hypothetical...

If BBDQ bots by default were restricted to only post 4 times a day and don't read replies, and anything above that was a paid option (say, $1/month with quantum discounts), would ANYONE be okay with that?

There are literally a hundred times more bots on there than I originally expected, and I love each and every one of you, but I kinda want to work on other projects too, and that's considerably harder when one project keeps crashing the server because it is posting to bluesky on behalf of 1000+ bots

A kinda hypothetical question, asked out of curiosity:

If I were unable to keep hosting BBDQ, would anyone be willing to take it over and keep it running in my place, assuming I handed you all the code and told you exactly how to use it?

It's kinda gotten way too big for my server to handle 😩

I appreciate the thought, but honestly I'll probably remove the Patreon link the next time I get around to updating the code. Patreon works well for some people, but this not the type of projects that normally make money on there, at least not enough money to actually make a real difference.

(Of course, one QUICK solution to this issue would probably be to drastically reduce the number of active bots by adding a paywall... But I can't imagine that would be a very popular solution. I probably wouldn't be too comfortable charging for this anyway, since the whole idea is stolen from CBDQ.)

There may be some bad code involved here, I am an amateur hobbyist after all, but I have no idea where...

Increasing server memory could work and I'm considering it, but it would cost me money - and I don't know *how long* that would be an effective solution if the number of bots keep growing.

The reason BBDQ (and everything else on my server) occasionally goes down, is that the site was never scaled for running 1100+ bots, so occasionally it runs out of memory and stops working until I restart the server.

For now, I don't really have a clever solution to this problem, sorry.

Ugh, thanks, restarting now

Ugh, thanks, restarting now

Ugh, thanks, restarting now

Yeah, that sounds right.

Yes, Tracery has a modifier for this (and since it's in Tracery, it should work on toot sweet too).

Instead of "I ate a #nut#", write "I ate #nut.a#". That's all. Depending on the value of nut, that will automatically return "I ate a walnut" or "I ate an almond".