Project Storage

Department Project storage

NetApp archive backups not working

October 17, 2019 | Lars Rohrbach

When fileservice was restored after the PG&E outage last Friday, the part of the service that handles backups became non-functional. Staff are working to restore backup service as soon as possible.

Snapshots remain functional and available, providing easy access to file versions from the last 7 days. Older archive backups, from before October 9, also remain available. The broken archiving process means that backups after Oct 9 will not be available.

We make nightly copies of current fileserver data to our Disaster Recovery (DR) site at the San Diego Supercomputing Center (SDSC). This mirror of current data remains operational, and is not affected by the archive process failure.

UPDATE

[2019-12-11 | robm ]

ftp service was restored around 11:35am.

Resolved as of 2019-12-11 11:35am

Potential Campus Power Outage

October 8, 2019 | Lars Rohrbach

We have been alerted to possible PG&E power outages due to fire danger, and that campus power may be shutdown at some point over the next 2-3 days. If we receive notice of imminent shutdown, we will work to safely shutdown our NetApp fileserver and other critical infrastructure systems; this may occur on very short notice.

We will endeavor to update this News item as we receive any updates.

We recommend you proceed with shutting down any non-essential machines in advance, e.g. shutting down workstations at the end of the work day.

[Read more…]

NetApp fileserver maintenance – Phase III: Mon, July 22, 6-8pm

July 6, 2019 | Lars Rohrbach

On Monday evening, July 22, we will be taking the next step in our fileserver migration: moving network interfaces from the old system to the new system. This will result in brief interruptions of access to home directories and project shares.

Our expectation is that each move of a network interface should only take a minute or so, and that most NFS and CIFS clients will likely recover just fine without a reboot. Any ongoing file copies or moves will be interrupted and terminated; any unsaved file editing should be closed before the maintenance period.

For any systems experiencing issues after maintenance is completed, a reboot would be recommended.

[Read more…]

NetApp fileserver maintenance – Phase II: Wed, June 19, 12-2pm

June 6, 2019 | Lars Rohrbach

As part of our NetApp fileserver refresh cycle, we have a NetApp technician scheduled to be on site June 19, to assist with installation of the new hardware, and transfer of storage racks from the old NetApp system to the new one. Work is planned to take place between noon and 2pm.

Our expectation is that this work will not be disruptive to existing fileservice, but it involves a good amount of work around the existing hardware, so there is potential for unexpected problems.

NetApp fileserver downtime in preparation for upgrade – Phase I: Monday, June 10, 6-8pm

June 3, 2019 | Pathma Venasithamby

NetApp filers are due for their 4 year refresh cycle. Work on this has commenced, with new NetApp hardware on site. New filers are going to be added and data migration will occur some weeks down the road. While most of the work will be non-disruptive, some elements will be disruptive.

The first of this will happen on 10th June Monday, and the scheduled downtime will be from 6-8pm, to incorporate some network changes. All NFS and CIFS access to the filer (home and project storage) will be down during that time.

[Read more…]

Most IRIS Services Offline, 7/31 11PM – 8/1 9AM

July 31, 2018 | Derek Calderon

Power will be shut down to our primary server room in order to complete tests of and apply load to our new UPS backup system. **Most services including file storage will be offline from 11PM Tues. 7/31 to 9AM Wed. 8/1**.

* **Network infrastructure** including wired and wireless networks and DHCP/DNS services will remain online (except in 290 Soda).

* **LDAP and Active Directory** authentication services will remain online.

* The **IRIS website** and portions of the department website including **personal homepages**, as well as other IRIS services that depend on file storage, will be shut down **starting at 11PM Tues. 7/31**.

* **Storage services**, including home and project file storage, will be shut down at **12 AM Wed. 8/1**.

* All services are scheduled to be **restored by 9AM Wed. 8/1**, but may be online earlier if maintenance is completed ahead of schedule.

As the IRIS website will be unavailable during this maintenance, https://status.eecs.berkeley.edu will be available for in-the-moment status updates.

Please contact help@eecs.berkeley.edu with any questions or concerns.
[Read more…] about Most IRIS Services Offline, 7/31 11PM – 8/1 9AM

IT and Electrical Maintenance Shutdown, Sat. 6/16/18

June 15, 2018 | Derek Calderon

On Saturday, June 16th, the campus high voltage team will be cutting power to **Soda, Jacobs, Etcheverry, and North Gate Halls** to complete critical power grid maintenance. The electrical shutdown is currently scheduled for **8AM to 1PM** and these buildings will be **closed** during this time to all but critical staff related to the maintenance projects.

We are also taking this time to perform safety tests in our server and network rooms, as well as replacing the failed battery backup system in our primary server room. As a result, **the outage window for most IT services** (other than basic network connectivity) **is 11PM 6/15 – 9PM 6/16**.

Anybody with computers and electronics in Soda and Jacobs is **STRONGLY RECOMMENDED** to power off their noncritical systems when leaving the office on Friday, June 15. Critical systems **MUST** be powered off by 6AM on Saturday the 16th. This includes machines hosted in the 340 Soda server room, and a separate message will be sent to those system administrators next week.

As most of EECS’ IT infrastructure is hosted in Soda Hall, this will have some impacts which are detailed below. This will affect all EECS locations which are not undergoing electrical maintenance: **Cory, Sutardja Dai and Blum Halls, the Calvin Lab, and BWRC** as well as remote users.

* **Network infrastructure** (wired and wireless connections, including DHCP and DNS services) **will be offline for a brief period between 6:00 and 6:30 AM** in order to perform annual Emergency Power Off and Fire Alarm tests in our network core. Basic networking (including DHCP/DNS) will be restored by 6:30 and should remain up for the remainder of the day for all locations outside of Soda and Jacobs Hall.

* The IRIS website and portions of the department website including personal homepages, as well as other IRIS services that depend on file storage, will be shut down **starting at 11PM Fri. 6/15**

* Storage services, including home and project file storage, will be shut down at **12 AM Sat. 6/16**

* All services are scheduled to be **restored by 9PM**, but may be online earlier if maintenance is completed ahead of schedule.

As the IRIS website will be unavailable during this maintenance, https://status.eecs.berkeley.edu will be available for in-the-moment status updates.

Please contact help@eecs.berkeley.edu with any questions or concerns.
[Read more…] about IT and Electrical Maintenance Shutdown, Sat. 6/16/18

Fileserver maintenance 10apr18

April 10, 2018 | Pathma Venasithamby

This should be mainly non disruptive, but be forewarned in case of mount issues.

Any CIFs shares will experience a disconnect but upon reconnect it should reconnect to the the other HA node. If someone is doing a copy over a share they will have to restart that.

Resolved as of 2018-04-10 15:15:00

Fileserver maintenance 30th Jan

January 30, 2018 | Pathma Venasithamby

Due to the power failures on 20th Dec, the department fileservice has suffered some damage, and there is a non functional management interface. This needs to be repaired or replaced. This should *not* be disruptive to users, as per Netapp. But it does involves reseating or replacing a motherboard on one of the nodes, so please be forewarned that one of the clusters will not be running in HA mode during the replacement.

Resolved as of 2018-01-30 14:26:00

Power Failure in Soda Machine room 11:40am Dec 20

December 20, 2017 | Rob McNicholas

Contractors attempting to repair the UPS in our primary machine room tripped a breaker which disabled most IRIS services. Service has been mostly restored as of around 2:05pm.

The lists.eecs.berkeley.edu mailing list server is still being rebooted but should be back shortly.

Resolved as of 2017-12-20 14:07:00