MeshWorks Wireless Oy

            Disturbances in Seemoto service

            8-9 February 2019

            Start time: 8 February 2019 20.00 CET time
            End time: 9 February 2019 11.00 CET time
            Still unknown reason caused a break in the Seemoto service. (Investigations ongoing.) Due to the weekend, the correction took longer than normally and the error was corrected in the morning of February 9th. Recovery took quite a long time because of the big amount of the sensor data.

            ------------------------------------------------------------
            5 February 2019

            Start time: 5 February 2019 07.00 CET time
            End time: 5 February 2019 11.00 CET time
            Problem in disk maintenance in the Seemoto servers caused a short break in the service. The break was corrected quickly, but the recovery started a bit slow. The data became up-to-date to all customers before 11.00 CET.

            ------------------------------------------------------------
            23 October 2018

            Start time: 22 October 2018 23.00 CET time
            End time: 23 October 2018 15.00 CET time
            Network connection problem of internet service provider affected to Seemoto service. 
            The measurement queue raised so high that the recovering to real time took longer than expected on 23rd. 
            A big amount of unnecessary "No data" -alerts were sent to some customers. 

            ------------------------------------------------------------
            17 October 2018

            Start time: 17 October 2018 14.00 CET time
            End time: 17 October 2018 15.00 CET time

            Network connection problem in Seemoto building. Problem was fixed quickly. Caused a short delay in the measurements.

            ------------------------------------------------------------

            16 October 2018

            Start time: 16 October 2018 14.00 CET time
            End time: 9 October 2018 15.00 CET time

            Temporary slow down in the measurement parsing. A big amount of data came in. Affected so that in some customers the data was late for a short time.

            ------------------------------------------------------------

            9 October 2018

            Start time: 8 October 2018 22.00 CET time
            End time: 9 October 2018 16.00 CET time

            Break in Seemoto service was not possible to correct during the night. It was corrected early in the morning on 9th. The measurement queue raised so high that the recovering to real time took longer than expected on 9th.

            -----------------------------------

            8 October 2018

            Start time: 9 October 2018 08.00 CET time
            End time: 9 October 2018 9.00 CET time

            Unfortunate SW update in service caused an error situation that it was not possible to log in to Seemoto service. This did not affect to the measurements, which were up to date all the time. SW update was removed quickly to solve the log in issue and updated later successfully.

            ----------------------------------------------------------

            18 July 2018

            Start time: 18 July 2018 02.00 CET time
            End time: 18 July 2018 04.00 CET time

            Maintenance break due to system maintenance activities.
            This caused reports to be unavailable during the maintenance break.

            ----------------------------------------------------------

            6 July 2018

            Start time: 6 July 2018 13.30 CET time
            End time: 6 July 2018 14.15 CET time

            Our service provider's problem was reflected to Seemoto -service also.
            This caused Seemoto webpage to be unavailable for a short period of time.

            ----------------------------------------------------------

            7 June 2018

            Start time: 7 June 2018 11.00 CET time
            End time: 7 June 2018 12.00 CET time

            Seemoto had a short maintenance break today at 11:00 - 12:00 CET.
            This caused increased backlog on servers, which can cause measurements to be delayed. Measurements should be back in real time withing this day.

            ----------------------------------------------------------

            30 May 2018,

            Tampere, Finland


            UPDATE REGARDING SEEMOTO SERVICE BREAK ON 29th of May 2018

            The root cause for problem was found. During April-May we have increased our data storing capability heavily in Seemoto cloud service. The break on 29th was caused as a result of partially incorrect configuration of Cassandra database, which consists of 24 nodes/servers. The erroneous configuration is now corrected and the system is working steadily. We believe the same issue has acted as a reason that recovering from network breaks (due to malfunctioning network switches) has taken so long.

            UPDATE REGARDING NETWORK SWITCHES

            We are also in process of replacing all our network switch stacks in order to improve service reliability. These new network switches, from new manufacturer, have just arrived to our service center and we are planning the change process.

            Marko Kyrölä

            CEO


            ----------------------------------------------------------

            To whom it might concern
            18 May 2018,
            Tampere, Finland
            STATUS OF SEEMOTO SERVICE
            By means of this document MeshWorks Wireless Oy, a corporation organized and existing under laws of Finland, proprietary owner of Seemoto brand confirms that

            Due to datacenter network switch stack malfunction, we have had several service breaks 25th – 27th of April, 30th of April and 14th -15th of May. Furthermore the problem appeared also earlier, 6th of March.

            The root cause of these breaks was very similar to previous reported service breaks on 15th, 17th, 20th and 24th of September.
            Since September 2017, network switch stack malfunction was tried to solve with the network switch stack manufacturer; the network switch stacks has been updated with latest firmware received from manufacturer, changes has been done to network topology to improve reliability and the manufacturer has been consulted several times about the case. All these actions improved reliability in some extent, but unfortunately the problem now reappeared in severe way.

            As a consequence, we have decided and already ordered new network switch stacks from another manufacturer. The delivery schedule of these new switches has not yet confirmed by the new manufacturer.

            Marko Kyrölä
            CEO
            -------------------------------------------------------------------

            14.5.2018 
            Disturbances in core network caused a short breakage in Seemoto service. This affected to the customers so that sensor data was not in real time for about two hours. 
            --------------------------------------------------------------------

            26.4.2018 - 29.4.2018
            Due to the previous disturbances in the service some of our customers faced during the end of the week a problem, where the data was not in real time all the time. This happened only in Finland, if the customer installation had a TGG type gateway. Customers with other gateways did not face these delays.  

            --------------------------------------------------------------------

            25.4.2018
            Start time: 24.4.2018 21.30 UTC time
            End time: 25.4.2018 7:30 UTC time
            One of the core switches in the Seemoto Data center was jammed and needed to be rebooted. That caused a break in the service and the data was not in real time for a couple of hours. Sensors used their own memories to store the measurements and reported them after the break. No data losses. To avoid this kind switch problems in the future Seemoto will change all switches to the server room. New switches will come from different manufacturer.

            ---------------------------------------------------------------------
            13.3.2018
            Start time: 11.00 UTC time
            End time: 14.30 UTC time
            A human error in Seemoto data center caused a short service break. It was corrected quickly, but the stored data from the sensors which came to the service all at the same time loaded the system. Emptying the queues took a bit longer than expected. Most of the customers were on real time at 13.15(UTC), but some customers did not come on line until 14.30 (UTC)

            ---------------------------------------------------------------------
            10-12.3.2018
            During the weekend some users faced occasional problems to log into the service. No username / password fields were seen on the login window. There was not any break in Seemoto servers and all measurements were continuously in real time. Also alerts were functioning normally. Error was fixed first time during the weekend and finally on 12th of March. 
            --------------------------------------------------------------------

            6.3.2018
            Start time: 3.05 UTC time
            End time: 7:30 UTC time
            One of the core switches in the Seemoto Data center was jammed and needed to be rebooted. That caused a break in the service and the data was not in real time for a couple of hours. Sensors used their own memories to store the measurements and reported them after the break. No data losses.

            Updated: 10 Feb 2019 05:16 PM
            Helpful?  
            Help us to make this article better
            1 0