I’m not sure if that’s only on Reddthat and 0.18 servers (I’ll go check out my account on beehaw now), but I’m getting shown posts very near the top of the list from, like, 20-30 days ago, sometimes even going as far as a year or two ago. They haven’t that many upvotes or comments either. Anyone seeing that?
Wow. And I expect that’ll only go up from here! Is the infra handling it well? Are you all setup with autoscaling or stuff like that?
We are only using 40% of our cpu, and 50% of our memory.
I can easily add extra CPUs and Memory if needed. We recently added extra memory to deal with the instabilities. (Check the latest announcement on the memory, and why your display picture is throwing a 404 😥). I’m terribly sorry about that!
I did a “how I’m hosting reddthat here”: https://reddthat.com/post/19103
Honestly I never expected it to blow up as it did 😅. Especially with people actually funding our little adventure. I originally bought a 12 month server which cost just under $120 which I had thought would be enough for a slow and gradual increase of users.
The next plan is to scale out all the services to their own server. So we have Lemmy & lemmy-ui, Pictrs (pictures), Postgres (database). Biggest memory hog is lemmy, due to the spikes and coming in 0.18.1 will be an in memory cache. Biggest cpu hog is pictrs, as we have to convert the images -> webp, or gifs -> webm & soon to be videos. I’d like to scale out pictrs to have multiple instances but it is currently impossible as it uses an internal kv-store which is unable to be accessed by multiple instances. That will allow us to scale out each of the server to allocate more memory per item that is needed.
Ideally i’d like to run a docker-swarm, have each service scaled. But the only ones that could be possibly done like that (at this time) is lemmy and lemmy-ui. Which will be the first ones we do, but having the docker-swarm with only 2 of the services would just add complexity when it isn’t needed. Plus, they have some internal configuration files, so i’d need to build the deployment pipleline to build the config into our containers. Not hard, but not a huge priority.
😊 I got plans, and now we got funding. Worst comes to worse, I’ll throw extra cpu & memory at this box until it’s stupidly huge as a temporary measure (which is what other lemmy instances have done), and then migrate the services to other servers at a later date.
Tiff
nice!
Is lemmy tightly coupled to pictrs? Looks like it’s the biggest roadblock in scaling flexibility. Would be useful if it could be replaced at some point.
Not that I intend to fire up my own instance or anything, but as an amateur sysadmin, it’s nice to have a peek, at this early stage of the community, at how it’s all put together! It’ll be even more interesting to see how it’ll grow 😊
I hadn’t noticed until you mentioned, I guess I had it cached until now, it was only when I went to my profile page that the picture started breaking. No worries though, I don’t care about that!
Not tightly, but there are no (short-term) plans to migrate away from pictrs.