Medium Power Outage

• First draft by NTS from LASE list emails

• The preparation needed for a planned power outage varies based on how much power/which outlets will be lost. The following is a road map for a medium power outage: Mezzanine loses power, batcave loses power, MBE lab wall outlets lose power, BOS (plugged into wall outlet) loses power, PL lab loses power, DAS loses power; MBE racks don't lose power, emergency backup power remains on.

Bringing the Systems Down

• MBE lab - The MBEs and their electronics racks should not lose power during the outage. However, we will be losing power to the back room (Bat Cave), the outlets around the lab periphery, lights, and power to the mezzanine. Thus, the MBEs have been put in a low power idle state for tomorrow. The status of various MBE subsystems is as follows:

LN2

1. LN2 to the cryoshrouds has been shut off since the phase separators were to lose power.

2. The LN2 valves to each system has been closed in the MBE lab

3. the phase separator actuator controllers have been turned off in the mezzanine

4. the turbo controllers for the vacuum-jacketed lines in the mezzanine have been shut off

5. the oil backing pumps for the mezzanine turbos have been shut off

6. the LN2 valves feeding to each phase separator from the junction in the mezzanine have been closed.

7. The various mezzanine electronics were turned off the day before so that it could be completed gracefully before power was cut.

SF Chilled H2O

1. the shutoff valves on the pipes providing chiller water to the MBE source flanges were closed, cutting water flow to the source flanges.

2. The Neslab chiller that provides the chiller water from the mezzanine (by Echo's LN2 phase separator) was turned off the day before since the mezzanine will lose power.

Sources Chiller H2O

1. the shutoff valves behind the chiller in the Bat Cave were closed and the chiller depowered the day before since the Bat Cave will lose power.

2. The plan to put the chiller on emergency backup power for the outage was scrapped since it is fed by a 208 VAC line. Emergency backup is 110 VAC.

Cryopumps

1. Each cryopump was isolated from its attached system by closing its gate valve

2. The power line to the pump heads were removed and the cryos warmed up while being pumped on by scroll pumps through their pump ports

3. Once warmed and pressure-stable, the all-metal valves at the pump ports were closed and the scrolls turned off. Thus cryos are warm, isolated from chambers and pumps, and at rough vacuum

4. The He compressors in the mezzanine were turned off

5. The adsorbers for the Echo and Bravo SF cryopumps were changed. Kyle will be exchanging adsorbers tomorrow with Oxford so that we can swap the adsorbers on the other two He compressors (BOS and Bravo CAR cryos) tomorrow before bringing the cryos back online. It is a good idea to do this at this opportunity if the adsorbers are due for a change.

BOS

1. The BOS was back-filled with UHP N2

2. Because its electronics are plugged into the wall, it will lose all power tomorrow. This includes the ion gauge controller (the filament was already off since we vented), the cryo temperature monitor, and the actuator for the cryo gate valve. Turn them off gracefully beforehand.

RGAs

1. All RGA electronics boxes were fully depowered for protection, since we have a history of losing RGA electronics boxes.

Bravo and Echo chamber vacuum

1. The MBEs have been isolated from their cryos and have warm cryoshrouds, as detailed above

2. In order to maintain vacuum, the gate valves to the buffer chambers have been opened to the growth chambers to allow the buffer ion pumps to pump both systems.

3. The gate valve between the buffer chambers has been opened to allow for some redundancy and allow both ion pumps to pump on the whole volume.

Gallium

1. The Eurotherm box and Sorenson power supply for the Ga Tip filaments on each system was moved to emergency backup power. The cells should not have lost power tomorrow if they had been kept on regular outlets, but they were moved to backup "just in case."

Other MBE electronics

1. The electronics racks are on circuits that should not get affected by the outage. Thus all the Eurotherms were left on, in manual, at 0% output power.

2. All other Amber-controlled power supplies (non-Ga) are outputting no power to the MBE cells and CAR heaters.

3. The Ga Base Eurotherms were left in Auto with a nominal 50 degree C setting, but since the thermocouples are reading higher than 50 degrees C due to the heat from the Tip filaments, the corresponding Ga Base power supplies are outputting zero power to the cells.

4. The heated viewports were left on at their standard power setting.

5. RHEED controllers are off.

6. CAR controllers and cracker valve controllers are powered on. The cracker valves were left partly open (50-100 mils) so that in the event of a vacuum failure they will be at the same pressure as the growth chambers.

7. The building should still have internet connectivity to the outside, so we have left the PCs and Amber running and the systems can be monitored via LogMeIn tomorrow. NOTE: There was Wi-fi but no wired internet, so we couldn't monitor via Amber MBE growth computers are on wired internet.

Load locks

1. Both load locks were vented with UHP N2, turbos spun down, and scroll pumps turned off.

Personnel

1. Someone must be at MER prior to the power shutoff and just after to ensure that MBE systems left on are unaffected.

2. Once they have verified things are stable he or she will send out a group email stating so.

3. We should have LogMeIn access to the MBE Amber computers (if wired internet remains on) and can check in periodically through the day tomorrow to make sure everything is alright.

4. Once power is restored to the lab, those of us who are in Austin and available can head to MER to begin restoring order to the MBE lab.

Bringing the Systems Up

• Once the power outage is definitely over, we can begin to restore systems. The First Steps are

1. Mezzanine: LN2 phase separators, Neslab chiller, cryopump He compressors

2. MBE lab: scroll pumps on the load locks, cryopumps cooled and pumping behind gate valves, Echo turbo spun up and load lock pumped down, chiller water running, sources warmed up to ~100 degrees C

• MBE lab - The MBEs and their electronics racks DID NOT lose power during the outage. Overhead lights, some wall outlets, and wired internet (to which growth computers are connected) were off. The status of various MBE subsystems is as follows:

LN2

1. LN2 to the cryoshrouds are shut off.

2. The LN2 valves to each system are closed in the MBE lab.

3. The phase separator actuator controllers are: Bravo-ON; Echo-OFF; in the mezzanine.

4. The turbo pumps and controllers for the vacuum-jacketed lines in the mezzanine are ON

5. The oil backing pumps for the mezzanine turbos are ON.

6. The LN2 valves feeding to each phase separator from the junction in the mezzanine are mixed- Bravo: Open; Echo: Closed.

7. LN2 phase separators- Bravo: Full; Echo: Empty.

SF Chilled H2O

1. The Neslab chiller that provides the chiller water from the mezzanine (by Echo's LN2 phase separator) is ON.

2. The shutoff valves on the pipes providing chiller water to the MBE source flanges are OPEN.

3. All SF chilled H2O is back on.

Sources Chiller H2O

1. The chiller in the batcave is ON, and the shutoff valves behind the chiller are OPEN.

2. All Source chiller H2O is back on.

Cryopumps

1. Each cryopump was regened (pumped on by scroll pump).

2. He was never disconnected, so there was no need to recharge the cold head He or lines.

3. After regening, power was restored to each pump so it could pump on itself and the temperature (and pressure behind the gate valve) would drop.

4. BOS's Cryo is OPEN to the system once it drops below 20 K (it might burp a little, watch it)

5. Bravo's CAR cryo is OPEN to the system once it drops below 20 K (it might burp a little, watch it)

6. Bravo's SF cryo and Echo's cryo are NOT OPEN to the systems, as they continue to pump on themselves overnight.

7. The He compressors in the mezzanine are ON, and all adsorbers changed for all cryopumps.

8. We will monitor cryopumps overnight.

BOS

1. The roughing manifold was pump-purged, and the BOS was pumped down to the scroll's limit.

2. Once the cryo was under 15 K and stable (no longer rolling off/decreasing) on Amber, the gate valve was opened.

3. The ion gauge controller (after opening the operating cryo), the cryo temperature monitor, and the actuator for the cryo gate valve are all ON.

RGAs

1. All RGA electronics boxes are still OFF.

Bravo and Echo chamber vacuum

1. The chamber pressure was well-behaved today.

2. Now, the MBEs are mixed, as detailed above. Bravo: CAR cryo pumping on GC, Ion pump pumping on BC. Echo: Ion pump pumping on GC and BC.

3. Bravo gate valves: GC nearly full closed (normal); gate valve between buffer chambers closed. Echo gate valves: GC open to BC.

4. All LC gate valves are closed like normal (and were never opened).

Gallium

1. The Eurotherm box and Sorenson power supply for the Ga Tip filaments on each system are back on regular power. There were no issues.

Other MBE electronics

1. The electronics racks are on circuits that were not affected.

2. All of the Eurotherms for cells we are heating back to 100 C are in auto.

3. We are only heating cells for Bravo, are heating all cells except for As and Sb sublimator/crackers.

4. Echo's eurotherms are still in manual and cells are not outputting power (except for Ga).

5. The heated viewports were left on at their standard power setting.

6. RHEED controllers are off.

7. CAR controllers and cracker valve controllers are powered on. The cracker valves have been left partly open (50-100 mils).

8. The building did not have wired internet connectivity til 4:45 PM.

Load locks

1. Both load locks were pumped down by scroll.

2. Turbos were turned on and both LCs passed 5e-7 over an hour ago.

3. An LCLamp bake is running on both systems.

Personnel

1. Once power is restored to the lab, those of us who are in Austin and available can head to MER to begin restoring order to the MBE lab.

Items Done/Remaining

MBE Items turned back on/done

1. Mezzanine: Neslab chiller (SF H2O), cryopump He compressors, Bravo phase separator actuator controllers, the oil backing pumps for the mezzanine turbos are ON, the turbo pumps and controllers for the vacuum-jacketed lines in the mezzanine

2. MBE lab: scroll pumps and turbos on the load locks, cryopumps cooled and pumping behind gate valves (BOS and Bravo CAR cryos are OPEN), chiller water running, sources warmed up to ~100 degrees C. CAR controllers and cracker valve controllers are powered on

MBE Items remaining

1. Mezzanine: LN2 - The Echo phase separator actuator controllers in the mezzanine; the LN2 valves feeding to each phase separator from the junction in the mezzanine; filling the phase separators.

2. MBE lab: Open Bravo SF and Echo cryo pump; open the LN2 valves to each system in the MBE lab; Heat cells to full idle; As and Sb sublimator/cracker heating.

PL lab

• PL Lab (1.804)

Shut down

All electronics have been shutoff. PCs are shut off, spectrometers are turned off, and the powered lock-in amplifiers were each turned off. The lab will be without power (including lights) for the duration of the outage.

Bring up

PL Lab (1.804) - Overhead lights are ON, PCs are ON, spectrometers are OFF, and the powered lock-in amplifiers are ON. FTR, bench power did not turn off, overhead lighting was off; wall plugs weren't tested.

DAS

• DAS/D&S (1.820)

Shut down

Shut down and depower the FTIR (spectrometer and PC) and other equipment as needed. The lab will be without power (including lights) for the duration of the outage.

Bring up

1. Vacuum pumps are ON

2. electronics can stay OFF until needed