You may have noticed some intermittent email access for the last few hours. This is due to a mail server at Demon Internet which is attempting to deliver the same message over and over again, tying up all of the server resources.
I have contacted both Demon and the originator of the email message (which is not spam) and neither of them have been helpful in resolving the issue as yet, Demon saying it's the customers problem - and the customer must log a problem with Demon - and their customer has not yet responded to my email, and I've been unable to track them down on the phone.
So, in the meantime, I've had to block all mail from Demon Internet, until such time as I can be sure the problem is cleared. This may be for up to 48 hours. Allowing all this mail from Demon will mean the server is likely to fall over, or other mail will failed to be delivered, and so this is the best course of action.
I apologise for any inconvenience this may cause, but unfortunately it's outside of my control. I will continue to try and resolve the issue with the other parties.
Availability of the server was a bit poor yesterday due to a network failure at our ISP's datacenter, BlueSquare... details from my ISP below:
Sorry for any inconvenience this caused. I'm speaking to my ISP how they can improve their redundancy.
The connectivity to this server was only lost to the UK for the majority of the time, with network peering from the US (and thus Asia and Australia) and Europe remaining OK.
--
Dear customer,
Today we experienced an issue with BlueSquare's primary fibre link - a
break in the fibre was detected at the Telehouse (East) datacentre in
London.
Traffic is currently being routed through BlueSquare's secondary fibre
link, whilst engineers work on repairing the primary fibre.
During this time, however, there was significant packet-loss due to the
third fibre connection between Telehouse (East) and Telecity (HEX). This
is what caused the majority of the outage that customers, PoundHost, and
other providers in BlueSquare experienced this afternoon.
At this time, traffic is still being routed through the secondary
connection, although extensive work is being spent to bring the primary
connection back in service.
Let me reiterate that this issue is not directly related to PoundHost -
we are one of a handful of providers at BlueSquare affected by this outage.
We will send out a more detailed announcement once we are confident that
service will no longer be affected.
Many thanks for your support and patience during this time.
--
Update: 03/02/2007 10:51 and this just in... a further explanation:
Further to the two e-mails sent yesterday, I can now confirm further details of yesterdays issues.
The main cause of the outage for PoundHost, and the outage suffered by other ISPs located in BlueSquare, was due to their fibre provider, TeliaSonera, *disconnecting* the wrong fibre when testing another customer's link, due to them mis-labeling the fibre in their London datacentre suite (Telehouse East).
As per yesterdays email, our second diverse fibre, which was working fine, was down for general maintenance due to router and switch software upgrades which we were carrying out. When the primary fibre went down, we bought our second fibre up as quickly as possible, however due to the maintenance we were doing the connection caused packetloss and routing from some ISP's but not others.
We have now fully completed our maintenance on this fibre overnight, so in the event of any further issues on the primary fibre we can easily route over the second diverse fibre without the issues experienced yesterday afternoon.
Some customers have mentioned that contacting us via phone was difficult, this is due to our phone system being VoIP (voice over IP) based, so the bad connection meant we had troubles hearing customers, and in some cases no phones at all. I have this morning ordered 4 BT lines to replace the VoIP system to be installed within the next 14 days direct into our support office using the existing numbers, to avoid this happening in the future.
Following customer feedback, we have now setup a new NOC (Network Operations Centre) status page, which we will be finishing and launching on Tuesday to keep customers updated should there ever be any network issues in the future. This will allow you to get a quick glance status, and updates such as these on the website. We will email you on Tuesday once this site is completed with more details.
Again, we apologise for any inconvenience caused yesterday. It's an unfortunate chain of events that we already have systems and links in place to deal with (this kind of fibre break), but as with all things in life when you need it, it's the one time it isn't available, in our case the second fibre being down for our own internal maintenance. For reference, our two diverse fibres are called 'THE' and 'TCX' which you will see in any 'trace routes' you complete, depending on the routing destination.
After a night of hacking around and some hair-raising and hair-pulling moments I've done a full upgrade of the operating system on the server, to try and resolved some dependency issues in some of the Libraries. This has left a few other things out of shape, which I'm in the process of trying to sort out. Hopefully this will all be back up and running in a few hours (and then I can get some sleep!)
Sorry for the inconvenience caused and thank you for your patience.
We're having some problems with mail running through mail.hosteverything.net at the moment due to a flakey SSL library causing problems with POP3 mail checking. I'm doing some heavy emergency upgrades to try and (hopefuly) get everything back into sync asap.
Earlier today there was an outage when one of the disks in suith.hosteverything.net developed a fault, and had to be rebuilt. No data was lost in the process, but the server had to remain off-line during the rebuild (which takes around 3.5 hours to run).
The server is now back up. All undelivered mail has been queued on our backup mail servers and will start to find it's way into your account in the next hour.
Sorry for any inconveience caused in this outage.
Just a quick note to say that this Thursday, 17th August 2006, the hosteverything.net primary server (suith) that your email and website is hosted on is going to be moved to a new hosting facility. The server needs to be physically moved from one side of London to the other in the process so your email and website won't be available for a period of approximately 3 hours, from 20:50 GMT onwards. I know this is a bit of a pain but the result will be faster, more reliable connectivity. And I won't have to pay an 80% increase in hosting fees.
Any email that is attempted to be delivered during the down-time should be delivered once the service is restored, so the effects should be negligible to those on UK time. Sorry if it's a bit more inconvenient to those in the USA and elsewhere.
this might affect your access to hosteverything.net services:
Dear Customer,
We need to perform emergency maintenance on our core routers in our Maidenhead Data centre at 7.30am tomorrow morning. [28/07/2006 - BST]
This will unfortunately cause approx 20 minutes of network downtime, which we hope can be reduced if all goes well to just a few minutes downtime, however we are allowing a 20 minute window for worst case.
No action is needed on your part and this will only affect Maidenhead hosted customers, London is un-affected.
We apologise for the short notice on this maintenance, however it is critical to continued smooth running of the network.
Regards,
Matthew
Apparently there is a problem with the Fibre Optic connection where my server is hosted. hmmmmmm.
In the last few days there have been a couple of outages which have affected the main HostEverything.net web and mail server. The first was on Friday the 31st of March and lasted for about an hour between 17:30 and 18:30, the second on Monday the 3rd of April between 13:30 and 16:30 . These outages were due to network problems at our UK ISP, PoundHost and they are working to make sure it doesn't happen again.
Sorry for any inconvenience caused.