Throughout this evening and tomorrow we will be deploying the major update that we have been completing for hosting our own files as a backup to Rackspace (some of this work has partially been deployed already).
The Rackspace issue continues to be on-going, however this will negate our reliance on Rackspace for uploaded documents/photos etc. Please see here for their latest update:
When this patch is active, all uploaded files will be retrievable from Pabau servers if Rackspace fails.
Depending when the release hits your account, all issues relating to uploads/retrieval should be resolved fully by tomorrow night.
Reports suggest that we are seeing very few upload issues, so this approach will be in addition to the fixes rackspace are completing.
PABAU STEPS TAKEN:
Our team has been working through the night and we have introduced a new cloud hosting service which we will now be using for uploads, this includes quotes/PDFs/photos/consents etc, eliminating our Rackspace dependency.
Files will still be uploaded to Rackspace as the service comes back online, but we will also have another copy in our new hosted service.
Although you may still see some slowness, and failed messages, uploads will now be saved on our end, and our team will now work on the next stage of this project, which is for the files to be shown from our new hosting if Rackspace fails.
We are pushing the fix for uploaded documents in the next hour, please bear with us and let us know of any other anomalies you see in relation file uploads or retrieval, so that my team can jump straight on these.
Our internal tests show that Rackspace are continuing with fixes and the service is improving – the latest update from him can be found here:-
RACKSPACE STEPS TAKEN:
As of 30 November 2023, to increase storage capacity within the region and mitigate the impact being experienced, engineers continue to replace drives, pulling in inventory from Rackspace data centers globally.
As part of this process, engineers are utilizing drives with higher capacity as possible.
Given the type of storage configuration in this environment, the drive replacement process necessitates that drives are replaced as a group, rather than individually.
As of 21:41 UTC, engineers confirmed they successfully completed a group replacement and are monitoring the environment.
Additionally, engineers have identified some drives, once installed, are not being recognized by the configuration, and must either be re-introduced or replaced once again, adding to the time of recovery.
Once properly installed (a process that must be carefully managed by engineers so as not to introduce additional impact) and incorporated into the logical volume, this additional capacity enables installation, deletion, replication and synchronization processes operate in a healthy environment.
While affected customers may continue to experience impact, engineers are observing incremental improvement in the environment (and a reduction in the rate of errors experienced by customers).
They continue work to fully remediate the issue in the best possible time.
We are currently experiencing a global issue due to an outage with our upstream hosting provider, Rackspace.
This is effecting uploading of consent forms, saving photos, creating quotes, and is effecting all devices via web and our app.
Our engineers are working on a solution to allow us to resume uploads, however for now, we are reliant on Rackspace resolving this issue.
Please follow https://status.pabau.com/subscribe for more updates, or check directly with Rackspace using the link below:
Support
Resources
Resources
Company
