Here’s a few things that are happening that I feel should be addressed:
-196 is reopening in reddit
-our community is causing excessive load on the blahaj.zone servers
-i finally figured out how to add newlines in Lemmy markdown

Two of these things are more important than the third but I figured they should all be addressed.
First things first, the subreddit. Even though the og subreddit is reopening now, I do not plan to close the community. At this point 196 on Lemmy and 196 on Reddit are both separate yet similar communities and this lemmy community has a sizable enough user base and is active enough I feel closing it would be stupid.

Speaking of a sizable user base, the blahaj.zone servers are under excessive load which has been causing the admins to incur fees upwards of $3,000 a month. This is in no small part due to 196 being the fourth largest community on Lemmy.
I’m not sure what the future solution of this will be. It seems like they’re going to switch to a cheaper service provider which will also cause the entire website to be slower, at a lower cost.
For this reason, I think that a future migration to a different Lemmy server may be in order. I will be reaching out to major admins of other communities as well as our own ada to see if the pre-existing community can be transferred, or if we’ll have to start from scratch if we choose to move. This is very much a developing situation, and I will keep you all posted.

Finally, as you can see. I finally figured out how to add newlines to plaintext meaning that all of my future postings will be much more professional looking. Check out our new and improved rules and regulations pinned post to see just how much of a difference it makes.

You guys are the community, so I’d like to see what you think. Please discuss in the comments section of this post, and I will try to base my actions off of whatever you all conclude on.

Also please point out any typos I may have made I voice typed this on my phone

  • Ada@lemmy.blahaj.zone
    link
    fedilink
    English
    arrow-up
    139
    ·
    edit-2
    11 months ago

    Just be to clear, the state of the union post was an explanation of coming downtime, not a call for panic :)

    We’ve got the solution and way forward for these issues.

    You don’t have to move if your main concerns are impact on our resources.

  • Yote.zip@pawb.social
    link
    fedilink
    arrow-up
    74
    ·
    11 months ago

    There’s no way $3k/month is an accurate number to run a single community. Lemmy.world (and mastodon.world etc) combined takes ~1k/month to run, and they’re far larger and more active.

    Also, anyone who goes back to Reddit doesn’t belong with the 196 community. No bootlickers allowed.

    • Kaity A@lemmy.blahaj.zone
      link
      fedilink
      English
      arrow-up
      82
      ·
      11 months ago

      To be clear, $3k is an accurate, but unacceptable amount.

      As in that’s what it’s actually costing us, but it’s not what it should be costing. I’d imagine more like $250 is what we should be paying if I wasn’t using AWS in the silly way I am.

      I’m admitting up front that I’ve been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.

      Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.

      And so I didn’t notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that’s what got me.

      About half that $3k is just data transfer costs.

      Anyhow the notice was just to let our users know what is going on and that there’ll be some maintenance windows in their future so it doesn’t surprise anyone.

      We have a plan and it will all work out.

      Don’t panic or have any kneejerk reactions, it’s just an FYI.

    • Moss@lemmy.blahaj.zoneOPM
      link
      fedilink
      arrow-up
      25
      ·
      edit-2
      11 months ago

      They’re paying for Amazon web services instead of using more professional hosting alternatives, and things are nowhere near as optimized as lemmy.world. also they are hosting a Mastodon instance in addition to the Lemmy instance. also I’m pretty sure that ruud has dedicated hardware for lemmy.world, whereas this is being done entirely off the cloud.

      • Yote.zip@pawb.social
        link
        fedilink
        arrow-up
        32
        ·
        11 months ago

        It really seems like that should be fixed before blaming 196 for their infrastructure. If they don’t know how to fix it then yeah it’s probably best to move 196 away so they don’t bleed out.

        • Kaity A@lemmy.blahaj.zone
          link
          fedilink
          English
          arrow-up
          50
          ·
          11 months ago

          Just want to say, I don’t blame anyone else but myself.

          I certainly don’t blame anyone at 196.

          I hope I’m really clear about that. It’s one of the reasons I specifically didn’t name 196 in my announcement.

          We’ve got a solution planned, we’ve already started to implement it and have the image transfer issue solved already.

          We can afford to cover this ridiculous AWS bill, I just need to do some maintenance work so this doesn’t continue because I can’t continue to line Jeff Bezos’ pockets like this indefinitely.

    • Moss@lemmy.blahaj.zoneOPM
      link
      fedilink
      arrow-up
      9
      ·
      edit-2
      11 months ago

      They’re paying for Amazon web services instead of using more professional hosting alternatives, and things are nowhere near as optimized as lemmy.world. also I’m pretty sure that ruud has dedicated hardware for lemmy.world, whereas this is being done entirely off the cloud. in addition to all of that, they’re hosting a mastodon instance on the same website and the costs are for both

      • Ada@lemmy.blahaj.zone
        link
        fedilink
        English
        arrow-up
        46
        ·
        11 months ago

        They’re paying for Amazon web services instead of using more professional hosting alternatives

        This was a throw away instance of two or three people before the Reddit wave. We were not planning on growing from three users to six thousand :)

        • Kes@lemmy.blahaj.zone
          link
          fedilink
          English
          arrow-up
          34
          ·
          11 months ago

          Y’all are champs for suddenly taking on thousands of users at your own expense on what was meant to be a throwaway instance. It went from a small private server to hosting one of the most active communities on Lemmy in a short time, and you and your team of volunteer tech sorceresses have been doing the best you can to keep lemmy.blahaj.zone alive as it grows

        • Toribor@corndog.social
          link
          fedilink
          English
          arrow-up
          4
          ·
          11 months ago

          I’m running my Lemmy instance in AWS as well for just a few people and trying to keep costs low. I can definitely imagine it’s rough trying to scale up and control costs.

          • Kaity A@lemmy.blahaj.zone
            link
            fedilink
            English
            arrow-up
            11
            ·
            11 months ago

            We have already begun moving, and this announcement was to warn our users about future downtime as we migrate more critical services like databases etc.

      • ProfessionalHandJob@lemmy.beyondcombustion.net
        link
        fedilink
        English
        arrow-up
        5
        ·
        11 months ago

        ya know, it’s not hard to setup a totally independent instance on the same hypervisor as another.

        I have servers, you have people, we could make independent lemmy instances on my two Dell R720xd and one Dell R730xd servers. They’re not overloaded with people like lemmy.world yet, that would put way way more stress on that server.

  • Sanctus@lemmy.world
    link
    fedilink
    arrow-up
    49
    ·
    11 months ago

    No matter what I am not heading back to Reddit. Just FYI I think there is a 196 on lemmy.world already

  • Tavi@lemmy.blahaj.zone
    cake
    link
    fedilink
    English
    arrow-up
    40
    ·
    11 months ago

    I mean, I like it here.

    • old 196 is reopening -> most likely less traffic
    • AWS is really expensive apparently, mostly image and transfer

    Text is cheap to serve, but images (and video) are a PAIN. If a significant chunk of expense can be mitigated if I use Imgur or some other service and link my silly memes rather than direct upload then yeah sure, but moving the community around tends to lose people.

  • nowitsabby@lemmy.blahaj.zone
    link
    fedilink
    arrow-up
    33
    ·
    11 months ago

    I don’t know if moving is necessary. I like this community here and the admins have a path forward. If anything maybe we should be campaigning for more people to donate toward costs

  • Paige (she/her)@lemmy.blahaj.zone
    link
    fedilink
    arrow-up
    29
    ·
    11 months ago

    It’s a bit annoying that the Reddit mods are reopening the community after moving a bunch of people elsewhere. Now we have a Raddle, Lemmy, Discord, and Reddit 196 each with different moderators.

    Considering the Reddit has almost 50 times the number of subscribers as the Lemmy, I would assume that will be the most popular one, though we will probably have a higher percent of active users.

    I would hope that eventually the different mods of the different 196s could do some form of cross-site collaboration to have unity between the 196s but I don’t know what that could entail. Maybe the next 196 r/place could shout out the Lemmy community for example.

    • HardlightCereal@lemmy.world
      link
      fedilink
      arrow-up
      19
      ·
      11 months ago

      Just FYI the Raddle admins are enbyphobic and there are serious issues with power moderators abusing positions of authority all over that site. I don’t think we should be including Raddle in the conversation

        • BartsBigBugBag@lemmy.tf
          link
          fedilink
          arrow-up
          6
          ·
          11 months ago

          As someone who has been in the anarcho-scene on and off for a couple decades, it’s actually rather new that the mass of anarchists have become more accepting. One of my favorite bands growing up a was Against Me! and when Laura Jane Grace came out as a trans woman, there was sooo much vitriol from the scene it drove me away entirely for years. I’m so glad for the spread of intersectionality and more nuanced takes in anarchism at large, it’s refreshing.

        • ActuallyASeal@lemmy.world
          link
          fedilink
          arrow-up
          5
          ·
          11 months ago

          HardlightCereal got banned from Raddle for doxing someone. The site’s primary admin quite literally uses they pronouns. That doesn’t preclude enbypobia in itself but should at least make you question what HardlightCereal’s actual goal is without more evidence.

          • HardlightCereal@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            11 months ago

            I didn’t doxx anybody. Another user his their Raddle username from me in order to avoid being reported to the admins for rulebreaking behaviour, so I used their displayname on a discord server we had shared and on which they had violated Raddle’s terms of service. They later came to me under an anonymous identity and asked me not to use their name from the discord server, so I immediately agreed that I would change to their Raddle name if it was provided. They refused to tell me their Raddle username, and so I could not change the references to their discord displayname on the post.

            So in short: They hid their identity to avoid getting in trouble for breaking the rules, made requests of me and made sure I couldn’t fulfil them, and then complained to the admins I didn’t do the thing they prevented me from doing.

          • HardlightCereal@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            11 months ago

            I didn’t doxx anybody. Another user his their Raddle username from me in order to avoid being reported to the admins for rulebreaking behaviour, so I used their displayname on a discord server we had shared and on which they had violated Raddle’s terms of service. They later came to me under an anonymous identity and asked me not to use their name from the discord server, so I immediately agreed that I would change to their Raddle name if it was provided. They refused to tell me their Raddle username, and so I could not change the references to their discord displayname on the post.

            So in short: They hid their identity to avoid getting in trouble for breaking the rules, made requests of me and made sure I couldn’t fulfil them, and then complained to the admins I didn’t do the thing they prevented me from doing.

    • WizBiz [He/Him]@lemmy.blahaj.zone
      link
      fedilink
      arrow-up
      6
      ·
      edit-2
      11 months ago

      I feel like there should just be a new seperate discord for r/196. As it stands, the current options either are too unofficial or are like 196cord, where it’s a mish mash of two completely different communities with oposing personalities. While 196 seems pretty calm, friendly and is in general pretty down to earth, the other one has a lot of LARPers and pretty hostile to people and even besides that, the mods either don’t care or are clearly not in the right mental headspace to moderate.

      • Paige (she/her)@lemmy.blahaj.zone
        link
        fedilink
        arrow-up
        2
        ·
        11 months ago

        I joined the discord at some point and it was too fast to pay attention so I left. It definitely seems like a different platform for a community like this.

  • BartsBigBugBag@lemmy.tf
    link
    fedilink
    arrow-up
    20
    ·
    edit-2
    11 months ago

    Aww. I really loved being on blahaj, but it’s not worth causing that kinda stress on our sweet admins. I support wherever you decide to move us, except Reddit, but I think you’re already on the same page.

    Is there anything we can do in the meantime to help contribute to the admins or to help otherwise?

    Thanks for running this space, Its my favorite in the fediverse.

  • Wow, I had no clue blahaj.zone was running on AWS that’s definitely a way to run a cloud server if money is no issue. AWS is okay for small projects but as you scale up the prices start to hurt.

    However maybe a traditional dedicated server from a service like dedicated.com would work just as well or better, I’ve had some great luck with affordable dedicated servers from them in the past.

    • Ada@lemmy.blahaj.zone
      link
      fedilink
      English
      arrow-up
      22
      ·
      11 months ago

      It’s running on AWS because it was a throw away server for a few people before it exploded to over 6000 people. We did not design our initial setup with overnight expansion in mind :)

      • 𝓒𝔂𝓫𝓮𝓻𝓑𝓸𝔂@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        13
        ·
        11 months ago

        That’s definitely understandable. I just hope you fine folks don’t have to keep footing an enormous cloud provider fee for much longer.

        That really is a lot of $$$, just think of how many Blahaj you could have bought with that instead.

        But seriously though, even though I’m not a direct user of your instance I greatly appreciate the work and money everyone is putting into the blahaj.zone instance, I love 196.

        • Ada@lemmy.blahaj.zone
          link
          fedilink
          English
          arrow-up
          13
          ·
          11 months ago

          That’s definitely understandable. I just hope you fine folks don’t have to keep footing an enormous cloud provider fee for much longer.

          Nope, it won’t be much longer. We’re getting away from AWS very soon, and the downtime that will cause is what our state of the instance post was about :)

  • sorrybookbroke@sh.itjust.works
    link
    fedilink
    arrow-up
    14
    ·
    11 months ago

    Yeah moving it off of blahaj seems like a good idea. It seems like 196 is one of the largest mid sized communities here, I can understand that creates alot of issues for the server owners.

    Also, how do you the new line thing? It looks so professional and clean

  • ezri@lemmy.blahaj.zone
    link
    fedilink
    arrow-up
    8
    ·
    11 months ago

    I’m confused why you’d post about moving instances without talking to the admins of this instance about it first tbh

  • Teon@kbin.social
    link
    fedilink
    arrow-up
    7
    ·
    11 months ago

    Maybe they need a better hosting company.
    And maybe someone needs to make 196 it’s own instance and collect donations. That way people can post there and possibly a lemmy / kbin instance.
    lemmy196.com or 196kbin.com or any other TLD.

  • ranmagender@lemmy.blahaj.zone
    link
    fedilink
    arrow-up
    6
    ·
    11 months ago

    going to a different server seems like a smart idea. we left reddit for a reason so seems counterintuitive to go back there. But yeah ive noticed the servers being all haywire and if theres a way to alleviate that cost and stress on the admins and go somewhere thats better equipped to handle the load then thats all fine and dandy