This Mastodon account will be used for all maintenance, upgrade and downtime information about the Organise.Earth and Rebellion.Global family of servers.
Please ensure that your XR Branch designates a tech coordinator to monitor this account.
Downtime is anticipated.
Love and Code,
Further update from the datacenter:
"Short update: ceph cluster is almost fully recovered, all VMs we checked are reachable. There is however a network problem persisting that causes packets from the server network to the outside world (including DNS requests to our own dns servers) to randomly drop. We are currently determining which of the switches is the main cause for this."
An issue was corrected at https://forms.organise.earth where corrupted tables were resulting in 501 errors on attempt to load survey results and/or surveys. The database has been repaired and service can now resume.
From the datacenter:
"So from what we can see at the moment what happened is: the server was upgraded, the default ip6tables was replaced by an nftables wrapper, the migration was triggered and *before* the ip6tables alternative was reset to the legacy/old ip6tables setting an error was thrown as the nftables based variant complains about old rules being present."
Global Base at https://base.organise.earth is down for maintenance
GitLab at https://code.organise.earth is presently down for maintenance.
The https://pad.organise.earth is down due to a corrupted table `etherpad.store`, seemingly caused by the shared disk read IO issues previously (now remedied at the datacenter). A database repair operation is underway.
Disruptions at the Swiss datacenter as they urgently upgrade the CEPH cluster to remedy a serious issue with shared disk read performance discovered by XR Global Tech, affecting all servers in the datacenter. We were given no forewarning of this event, and so could not alert teams in time. Apologies for the downtime.
Mastodon instance for Extinction Rebellion