TL;DR possible thumbnail and local image outage tonight
Tonight I plan to update the image engine behind lemmy, called pictrs, to the latest stable version and to also perform a migration to a different database (sled to postgres).
In my testing there was minimal outage, but it did take quite some hours to do the migration (12+, though my test server is a lot slower than the production one).
Once the migration is complete, I intend to turn on a thumbnail cache cleaner that will significantly reduce the size of the thumbnail cache on the server. Lemmy currently saves many thumbnails locally, and never deletes them. This means we have over 260GB of images that get backed up each night for no reason. Lemmy makes it hard to tell the difference between cached images and user uploaded ones, but the cleaning tool will go through each post in the database, find the thumbnail associated with it, and delete it if it's more than a month old.
Any questions, let me know!
Also, we are still having issues with lemmy.world. They are testing out new tools to block kbin traffic if they start to see huge amounts of activity coming from them, which will hopefully mitigate this to some extent. There is a tool running to help get posts and comments to us in real time, but it won't work for votes which means you'll see lots of lemmy.world posts and comments with zero votes, and the votes will be delayed in arriving by up to 1.5 days.
Feel free to ask questions about that too!
Night update - this is definitely taking longer than I expected...
Next morning update - still going ๐
Still going ๐คฏ - but have sought help and have made an adjustment. Hopefully that solves it!
It's done!
I'll do the cache cleaner probably tomorrow, but this shouldn't affect uptime so will do it whenever I get a chance.