RSS Feed
Latest Updates
Jan
13
[COMPLETED] - Chicago - Upgrade To 10Gbps Uplinks
Posted by Ruchir Shastri on 13 January 2017 07:51 PM

Update: 15 Jan 2016 10:10AM CST

This is now completed with almost no downtime..

---------------------

Update: 15 Jan 2016 10:00AM CST

We will be starting this shortly.

---------------------

Date: 15th January, 2017

Time:  Starting 10AM Chicago time (CST) for about 30 minutes

Expected Impact:  Brief periods of latency and small outage during spanning tree reconvergence.

Purpose of Work:  We will be upgrading our uplinks in Chicago data center with redundant 10Gbps uplinks from our provider (currently setup as 4 * 1Gbps LACP). The optics are laid out on both ends and our data center will turning up these links and disable the old links. This upgrade provide increased throughput and better network performance for our entire Chicago setup.


Read more »



Jan
4

Dates: January 6, 7, 8 & January 10, 12, 14

Time:  Maintenance window will be open from 8:00pm mountain - 2:00am mountain time

Expected Impact:  Brief periods of latency and packet loss during the maintenance windows.

Purpose of Work:  We will be performing various network upgrades, including:

    Bringing optical amplifiers online to increase the capacity on our dark fiber ring
    Adding new line cards to our MX240 routers and grooming existing circuits over to them
    Replacing our aging MX80s and EX4500 switches at 1801 California St with Juniper QFX5100 switches

During the maintenance window, we will be providing updates regarding specific activities that will be taking place during these windows.  Unfortunately, due to the scope of the work we need to perform, and the interrelatedness and dependencies of all of this work, we are bulk scheduling maintenance windows.

UPDATES:

2017-01-06 21:10 MST: We are beginning the portion of our maintenance with the greatest risk for impact. If you experience any packet loss or delay it is very likely to be related.

2017-01-07 01:15 MST: We have concluded our maintenance activities for the evening. All systems are operational.  

2017-01-07 20:34 MST: We are beginning our maintenance for the evening. Tasks tonight include upgrading border1 to the latest JTAC recommended JunOS and then working to bring the replacements for our EX4500 switches online.

2017-01-07 23:50 MST: Our maintenance is concluded for the evening.  We encountered some odd issues with our border routers during the code upgrades, which took substantial amounts of time to investigate.  As a result, we will be working on the EX4500 replacement tomorrow evening.

2017-01-08 20:36PM:  We will not be conducting any network maintenance this evening, and we will also be rescheduling the timing of the remaining tasks.  The schedule was not maintainable, and at this point most of our staff working on this project have gotten only 4 hours sleep over the past two nights.  A more official notice will follow, but the next maintenance events will be scheduled for Tuesday, Thursday, and Saturday this coming week, with slightly shorter windows (9PM - 1AM).  

Reminder - maintenance will be connected this evening, January 10 between 9pm and 1am. 

2017-01-10 23:30 MST: The data center experienced an unexpected issue which caused traffic drop during tonight's maintenance. This has been resolved now.

2017-01-10 23:36 MST:  During our network maintenance for the evening, we were operating under reduced redundancy.  Unfortunately, as our luck would have it, one of our border routers decided to coredump, causing a network wide outage for approximately 25 minutes.  During these 25 minutes, we were rolling back our previous changes made to effect the maintenance window, which involved reverting both physical and logical networking configurations.  Connectivity was disrupted at 22:50 and restored at 23:15.  We are still actively working to determine more about the outage.  The maintenance window may be extended to address any issues that remain related to this unexpected problem.

2017-01-11 00:43 MST:  All systems are working as expected at this time.  All further maintenance is canceled, until we can work with our vendor to get to the bottom of the issues that occured this evening.


Read more »



Dec
27
[Completed] -Network Maintenance: Chicago IL, Juniper Switch Software Upgrade
Posted by Ruchir Shastri on 27 December 2016 10:56 PM

Update 30 Dec 2016 10PM CST

The upgrade has been completed to an intermediate release since the upgrade to latest one failed. We will attempt to upgrade to latest release some time later and update this news schedule. 

-------------------------

Update 30 Dec 2016 9:30PM CST

The upgrade actually failed and we had to revert back to older version. We are now re-attempting to upgrade to another (not the latest) version before upgrading to the latest. 

-------------------------

Update 30 Dec 2016 8:50PM CST

This didn't work as expected and took our network offline for a few minutes. It is back up and we will keep you updated.

-------------------------

Update 30 Dec 2016 8:15PM CST

This upgrade is currently in progress and we will keep you posted.

-------------------------

We have scheduled a software upgrade on our Layer 3 Juniper Switches in Chicago, IL to bring them up to the current recommended software version which fixes several bugs from previous version.

Date: Friday, December 30th, 2016
Start Time: 8:00 PM CST (GMT -6)
End Time: 10:00 PM CST (GMT -6)
Maintenance Scope: Layer 3 Juniper Switches

Customer Impact:

During the maintenance, we will be updating the software on our Layer 3 Juniper Switches in Chicago, IL. These switches are setup in Virtual Chassis configuration for redundancy and we will be performing NSSU (Non Stop Software Upgrade) to avoid downtime. We expect the customer impact of this maintenance to be minimal, though some brief periods of localized inaccessibility and suboptimal routing may occur. The upgrade process is expected to take approximately one hour. Additional time is being allowed in case of issues with the process and for testing following the upgrade.


Read more »



Oct
26
Security Advisory: Important Linux Kernel Vulnerability
Posted by Mark Brown on 26 October 2016 10:24 AM

A vulnerability has recently been disclosed in the Linux kernel which affects all supported Linux systems running any distribution.  This vulnerability is serious and may allow a remote exploit or local user to cause privilege escalation, resulting in root access to your server. A working example of the exploit has already been publicly disclosed, thus no advanced knowledge of the Linux kernel is required to gain root access once shell access has been obtained on the target system.

The CVE entry for this vulnerability may be found here:

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-5195
[http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-5195]

Windows users are unaffected by this vulnerability.  We have not evaluated the status of Linux distributions or versions not mentioned here.  You should consult your distribution's security documentation to determine if your distribution is vulnerable and how to apply the update.

It is especially important to upgrade your kernel if your system provides multi-user services via shell accounts, jail or container-based virtualization, or dynamic web sites. Updates which fix this vulnerability and other security issues and bugs are available immediately for most systems.

CENTOS

CentOS has patched this vulnerability as of:

     * CentOS 7: October 24th, 2016
     * CentOS 6: October 26th, 2016
     * CentOS 5: Updated packages are not yet available.  This post will be updated when they are published.

To verify your system is running the correct kernel, run the following
command:

uname -r

The version should be greater than or equal to the following, depending on your distribution:

     * CentOS 7: 3.10.0-327.36.3
     * CentOS 6: 2.6.32-642.6.2

If your version does not match, please run the following command and ensure an update to the kernel package is included:

yum -y update kernel

If no update is available, please try the following commands, then repeat the command above:

yum clean metadata
curl -s mirror.steadfast.net/mirrorize | bash

These commands will remove cached update information and force your server to use our mirror server, which is known to already contain the updated kernel version.

After the upgrade processes, you should REBOOT your system _as soon as possible_.

Red Hat published the following advisories regarding this
vulnerability:

     * https://access.redhat.com/security/cve/CVE-2016-5195
[https://access.redhat.com/security/cve/CVE-2016-5195]
     * https://access.redhat.com/security/vulnerabilities/2706661
[https://access.redhat.com/security/vulnerabilities/2706661]

DEBIAN

Debian has patched this vulnerability as of October 20th, 2016.

To verify your system is running the correct kernel, run the following
command:

uname -v

The version (appearing after "#1 SMP Debian") should be greater than or equal to the following, depending on your distribution:

     * Debian 8 (Jessie): 3.16.36-1+deb8u2
     * Debian 7 (Wheezy): 3.2.84-1
     * Debian 6 (Squeeze): You must upgrade to Debian 7 or later.  This version is no longer supported.

If your version does not match, please run the following command and ensure an update to the kernel package is included:

apt-get update
apt-get install linux-image-`uname -r`

If no update is available, please try the following commands, then repeat the commands above:

curl -s mirror.steadfast.net/mirrorize | bash

This command will force your server to use our mirror server, which is known to already contain the updated kernel version.

After the upgrade processes, you should REBOOT your system _as soon as possible_.

Debian published the following advisories regarding this
vulnerability:

     * https://security-tracker.debian.org/tracker/CVE-2016-5195
[https://security-tracker.debian.org/tracker/CVE-2016-5195]
     * Jessie: https://www.debian.org/security/2016/dsa-3696
[https://www.debian.org/security/2016/dsa-3696]
     *
Wheezy: https://lists.debian.org/debian-lts-announce/2016/10/msg00026.html
[https://lists.debian.org/debian-lts-announce/2016/10/msg00026.html]


Read more »



Nov
7
Enhanced PayPal Billing Gateway
Posted by Ruchir Shastri on 07 November 2015 11:38 PM

We're excited to launch all new PayPal Billing Gateway - please visit https://support.softsyshosting.com/index.php?/Knowledgebase/Article/View/181/2/enhanced-paypal-billing for details and steps on how to configure it.


Read more »