I’ve been seeing the exact same posts, in the same order, with almost no change in up/downvotes. It’s been like this for at least 6 hours.
Did everyone agree to take an internet break and not tell me or is something not working correctly?
my apologies everyone - it was working rock solid for a few days and I got lulled into a false sense of security and took my eye off the site for a few hours :/ I am still working on some alerting to tell me when rabbitmq dies, but it’s obviously got some room for improvement.
In any event, I am aware, I’ve fixed the current problem and the queues are draining now. I’ll work on a better monitoring and response automation system tomorrow.
Thanks for your help.
Tho I noticed a post made 8 hrs ago isn’t federating to original server.See: https://lemmy.world/c/bayarea And https://fedia.io/m/bayarea@lemmy.world/t/1272729
that’s not so good. this is a different problem - there isn’t a backup like we’ve been seeing. I’ll see if I can tell what is going on
I’m working on it - not exactly sure what is wrong now - there aren’t any logs to indicate what is going on yet.
I just tried this again. When I just posted, and the last time I did it, I got an http error message. However, when I press back, the post is there. Not sure if this is a fedia thing or an mbin thing or a Lemmy thing. But that’s likely the problem.
I am seeing the same thing, and I’m not sure. I am trying to narrow down what is going on though
Jerry, we’ve got what you termed the rabbitmq issue happening again. Posts on new aren’t updating.
yeah, apologies. it seems to know when I am out at dinner. it’s catching up now
Yeah, as @hello_there@fedia.io pionted out, it looks like parts aren’t updating again. I’ve had the same posts on ‘hot’ for a while now.
Thanks for the ping - working on it
Much thanks!
Literally just sorted by newest to save my sanity and found this.
Is it just fedia.io or is this happening to other mbin instances too? seems like this happens every now and then, and i also find posts from fedia aren’t appearing on lemmy instances until there is server admin intervention of some sort, then it all updates at the same time. maybe the server admin isn’t available due to the jewish new year holiday?
Per @jerry there’s some bug in federation where an incoming message is malformed or something and it causes the message queue to silently freeze. He just has to restart it and things start moving again. Unfortunately it sounds like the mbin devs haven’t got it figured out yet so we deal with the problem now and then.
My apologies - I thought I had the problem fixed, but clearly not. It’s working again, and I will get back to fixing the alerting and auto-error correcting tomorrow.
Thank you for your work!
No worries man. I appreciate you taking the time and expense running the instance.
My apologies - I was off doing other things today as I thought this problem had been fixed. Things are almost complete with catching up now and I’ll be working on the problem more tomorrow.
Yea, federation keeps breaking unfortunately. Hopefully an automated solution can be implemented.
Btw, can someone tell me how to ping a user? Admin could probably use a heads up.
I’ve tried to look up how to DM a user and I haven’t found a way yet, so I’d like to know as well.
To DM someone open their user profile by clicking their name then click the little envelope icon. To ping someone just mention their username preceeded by an
ampersand@ like so: @dhhyfddehhfyy4673Cool, thanks! And just a heads up, but ampersand is &, don’t think the at symbol has an equivalent one word term.
Ah, right. I wrote that lying in bed around 1 AM, surprised I got any of it correct.
Just checking in after not having been on here for a few hours, and yes, completely.
