RSS Feed
Latest Updates
Jul
19
[COMPLETED] Network Maintenance - Denver Data center - 20th July 2017
Posted by Ruchir Shastri on 19 July 2017 12:36 AM
Update: 20 July 2017 22:10 PM

Everything has been performing smoothly and a test to migrate routing to backup engine succeeded as well. This maintenance is now complete. We may need to schedule another maintenance to optimize configuration for better efficiency at a later time.

===================================


Update: 20 July 2017 22:00 PM


Services are back up and both nodes have been upgraded to latest JunOS. We will be performing further troubleshooting for some more time and monitoring as well.

===================================


Update: 20 July 2017 21:45 PM


Backup engine has been upgraded and currently Master node is getting upgraded. Network will not be reachable now and we will update again once this is complete.

===================================

Update: 20 July 2017 21:20 PM


We faced some issue with NSSU and hence doing individual member upgrades - started with backup node.

===================================

Update: 20 July 2017 20:55 PM


We will be starting this in 5 minutes from now. We will keep you updated about the progress.

===================================
Hello Customers,

Please take a note of below maintenance window which affects servers located in only Denver data center.

Maintenance Window: 20th July 2017 21:00 to 22:30 Mountain Time

Task: Upgrade JunOS to latest version on our Layer 3 Juniper Switches. We have redundant pair of switches in virtual chassis whereby we will update one switch at a time.

Impact:
There may be periods of packet loss or loss of network connectivity while the upgrade is in progress.

This maintenance is required since we encountered an issue with these switches on Monday which caused traffic to stop flowing completely. This is apparently a bug in current version of JunOS that is installed and hence we will be upgrading it to the latest.

We will be having senior network engineers to complete this task onsite and we will keep this announcement updated as the task progresses.

Thanks in advance for your cooperation. You can contact our tech support department with any queries that you may have by logging into support portal at https://support.softsyshosting.com 

Best regards,

SoftSys Support
Read more »



Jul
17
[RESOLVED] Network Outage - Denver Data-center
Posted by Brad Hack on 17 July 2017 08:01 AM
Update: 11:11AM

Services have been operational since about 45 minutes now. We do not have complete information available on what caused this but it is apparently related to a bug in JunOS (operating system of Juniper switches). We run a redundant pair of these in Virtual Chassis environment but there was apparently something triggered internally in these switches which did not allow traffic to flow from any of these switches. We have core dump available which will be provided to Juniper support team and we will follow up with updates.

We may need to schedule a maintenance window to perform a (minor) version upgrade of JunOS. We will open a fresh announcement for the same.

Apologies for the inconvenience caused and we will be working towards avoiding similar issues in future.

======================


Update: 10:30AM


The services came back up for some time now. We are observing and monitoring everything and will confirm back shortly.

======================

Update: 10:00AM


The services came back up briefly but went down again. We will be upgrading JunOS to 15.x on this stack to eliminate any bug in JunOS that may have caused this.

======================

Update: 10:00AM


We have tried a number of recommended options including removal of one member of chassis and the reverse; however, unfortunately it has not helped. We are booting back both the switches again and will keep you posted on how it goes.

Apologies again and thank you for your patience while we work diligently on resolving this issue.

======================

Update: 9:20AM


We are continuing to troubleshoot and make progress. The ports on specific Juniper switch are all showing blocked state with no ARP / MAC entries. We are attempting a reboot of switch stack and will keep you posted on progress.

======================

Update: 8:43AM


Our network engineers continue to work on this. Our core Juniper switches are not responding and we are trying to fix this. We are dedicatedly working on this to bring the affected services back up as soon as possible. At this time, we have not been provided with an ETA but we will keep you updated on the progress.

======================
We are currently facing network outage in part of our Denver data center. Our Techs are working on this and we will continue to update the status at regular intervals.

Apologies for the inconvenience caused.

Softsys Support
Read more »



May
12
Critical Microsoft Vulnerability
Posted by Ruchir Shastri on 12 May 2017 08:00 PM
Hello All,

In view of recent events that have highlighted the persistent risk of the threat posed by cyber-attacks
( http://www.bbc.co.uk/news/health-39899646 ) we strongly advise our customers to apply the following security update, released by Microsoft on March 14th this year: https://technet.microsoft.com/en-us/library/security/ms17-010.aspx?f=255&MSPPError=-2147217396 as bare minimum and latest Windows updates which include May security roll up (which includes all of the above), if not having done so already.

Microsoft have announced a vulnerability in Microsoft Server Message Block 1.0 (SMBv1) server today. This service (SMB) is utilised to present shares, printers and more on a Microsoft Domain network.

This vulnerability exposes core Active directory components to Remote Code Execution from unauthenticated attackers. They would be able to execute any code they wished to potentially gain access to the entire network. The patches Microsoft have provided should be tested installed as a matter of urgency.

Specific Security reports of the Common Vulnerabilities and Exposures (CVEs) are below -

Windows SMB Remote Code Execution Vulnerability : CVE-2017-0143
Windows SMB Remote Code Execution Vulnerability : CVE-2017-0144
Windows SMB Remote Code Execution Vulnerability : CVE-2017-0145
Windows SMB Remote Code Execution Vulnerability : CVE-2017-0146
Windows SMB Remote Code Execution Vulnerability : CVE-2017-0148

The released patches target the SMBv1 service and the way it handles the particular requests that can be used to exploit it.

We will proceed with applying this on all fully managed Windows servers on our network along with host machines which will in turn be rebooted over this weekend.

Thanks for your understanding and co-operation.
Read more »



Feb
13
[FIXED] - Vulnerability Note VU#867968
Posted by Ruchir Shastri on 13 February 2017 12:09 AM
Hi Customers,

With regards to vulnerability note http://www.kb.cert.org/vuls/id/867968 - we have updated our firewall filters to block outbound SMB connections (TCP ports 139 and 445 along with UDP ports 137 and 138). If you are utilizing any of these ports for outbound SMB connections and want to have an exception created, please open a ticket with our technical support team with IP address that you'll like to exclude from above rule (adding an exception will mean that your server is open to above vulnerability until Microsoft rolls out a patch for this).

Thanks,
Softsys Support
www.softsyshosting.com
Read more »



Jan
13
[COMPLETED] - Chicago - Upgrade To 10Gbps Uplinks
Posted by Ruchir Shastri on 13 January 2017 07:51 PM

Update: 15 Jan 2016 10:10AM CST

This is now completed with almost no downtime..

---------------------

Update: 15 Jan 2016 10:00AM CST

We will be starting this shortly.

---------------------

Date: 15th January, 2017

Time:  Starting 10AM Chicago time (CST) for about 30 minutes

Expected Impact:  Brief periods of latency and small outage during spanning tree reconvergence.

Purpose of Work:  We will be upgrading our uplinks in Chicago data center with redundant 10Gbps uplinks from our provider (currently setup as 4 * 1Gbps LACP). The optics are laid out on both ends and our data center will turning up these links and disable the old links. This upgrade provide increased throughput and better network performance for our entire Chicago setup.


Read more »