Unplanned power outages have the potential to cause catastrophic failure of a number of components on the MBE systems. If an unexpected power outage occurs there are several critical actions you should attempt to perform to minimize damage to the systems. Most importantly, remember that safety is paramount, when in doubt ask someone else before risking anything you're not comfortable with. Keep your head on a swivel.

1 Short to-do list (IMMEDIATELY - decreasing order of importance):
2 More detailed list of things that you should check immediately following a power outage
3 Lower priority things to check following a power outage
4 SRB thoughts
5 Old notes here:

Short to-do list (IMMEDIATELY - decreasing order of importance):

Call Seth (650-450-0060)
Keep Al melted if at all possible!
1. Note how long the UPS indicates you have.
2. If Al has frozen, power down the Al Sorensons and Al backup to prevent remelting.
3. Call Seth: if Al > 400C, things might still be OK and it's worth trying to save the
Loosen the As and Sb valves manually (Bravo and Echo)
1. Open each a couple turns
2. Direction is noted on the cell
3. DO NOT FORCE THEM OPEN
Find James/facilities to see how long of an outage to prep for.
Drain the CAR cooling water (Bravo and Echo)
(If building emergency power is out) drain the Bravo deep upward looking dopant cooling water
Begin to prepare machine for a graceful restart when power comes back online.

More detailed list of things that you should check immediately following a power outage

In an emergency, you can enter the MBE lab through the front door without gowning in. Put some gloves on before you start working on the machines.
Is the emergency backup power online?
1. If so, you have roughly 6 hours of backup power, you should be able to get the system into a stable state on backup power
2. Quick ways to determine if emergency backup power is online:
  1. Are there any lights on in the building?
  2. Does the APC uninterruptable power supply indicate that it has wall power?
  3. Does the Al interlock box light indicate that it's being powered by emergency backup power? (green power LED = building backup, orange power LED = battery backup)
Is the battery backup powering critical components?
1. ie: If emergency backup power is off, is the interlock box and growth chamber ion gauge controller on, is the Al charge still molten?
2. If battery backup is depleted the Al charge is frozen. Disconnect the power from the cell to prevent accidental remelting.
Is the Al charge molten?
1. Timeline for Al cell backup
  1. Immediately after power failure:
    1. Al Eurotherm turns off triggering Al backup interlock
    2. The interlock will turn off power to the cell in the event of a growth chamber pressure failure. If this happens, unplug power from the Al cell to prevent accidental remelting of the charge
    3. As long as emergency backup power is online, the backup Al power supply will keep the charge molten while the growth chamber pressure remains low enough to prevent the pressure interlock from triggering
      1. Try to get a pump on the growth chamber before the cryopanels stop pumping!
        Echo BC ion pump and GC cryo pump will be operational on emergency backup power. Carefully open gate valves to allow these to pump on the entire buffer-growth chamber vacuum system.
        The Gamma ion pump controller display will give you a very close estimate of the vacuum pressure in the ion pump. Monitor this while opening the gate valves between chambers with unknown pressures!!!
  2. After one hour without emergency backup power:
    1. APC will discharge completely turning off IGC 100 and Al backup power supply - the charge will freeze if left for more than an hour on battery backup. If this happens, unplug power from the Al cell to prevent accidental remelting of the charge
Have the cracker sublimators cooled down?
1. Without power the sublimators will cool rapidly. if you catch the power outage early, try to open the sublimator valves manually.
  1. Unhook the power from the valve motors. Manually rotate the valve positioner in the open direction (counter clockwise?) - the close direction is marked on the valve in sharpie
    1. If you feel resistance when trying to open the valve DO NOT FORCE IT! Permanent damage to valve can occur of it is deformed by excessive force.
2. If the system is no longer under vacuum do not open the valves! There will be a large pressure differential across the valve because the sublimator will still be under vacuum.
Is the system still under vacuum?
1. The cryo panel will keep the growth chamber under vacuum for two to three hours. If there is no power to the growth chamber ion gauge controller, it's safe to assume that the system is still under vacuum for two hours. Find a way to power the GC ion gauge controller. (DC Inverter from an automobile or backup gas generator + extension cords)
2. If the pressure interlock is triggered, the Al cell backup will turn off. Don't accidentally remelt the aluminum charge when you turn the ion gauge controller back on
Are any water lines frozen?
1. In-vacuum water lines in close proximity to the liquid nitrogen cryo panels will freeze after components cool down and the water flow stops.
2. When the power goes out the sources chiller (the little one in the bat cave) will turn off killing flow to the sources, CAR, and turbo pump. The source flange chiller (the big one in the mezzanine) will continue to run on emergency backup power.
3. Drain the water lines on sources that might potentially freeze. In order of importance these are:
  1. The CAR water lines
  2. The dopant cell water lines
    1. The dopants are powered by emergency backup power, which should buy you some time before they freeze
  3. The nitrogen plasma source water lines (if installed)
  4. The source cooling panel water lines (only if emergency backup power is not operational - check the interlock box flow meter displays if in doubt)
  5. Everything else
4. To drain the water lines remove the quick disconnect fittings from the lines. Remove the source quick disconnect first and place it in a bucked/jug. Then remove the return quick disconnect. If the lines drain on their own, it's a good sign that there is no blockage
  1. If the lines do no drain on their own, it most likely means that there is a blockage in the line somewhere. Do not pressurize the lines with CDA/N2 if you suspect there is a blockage! This can rupture the lines inside the system.
5. If any of the water lines have frozen
  1. The best course of action is to get another pump on the growth chamber and manually turn the LN2 fill valve off. After the cryo panels have warmed up, the ice should melt and allow the lines to drain. Once the system is back under vacuum and the lines are drained, leak check the waterlines to this component with He before reintroducing water!
Is there liquid nitrogen in the cryo panels?
1. The cryopanels will provide cold wall pumping for around two to three hours. While the panels are still cold if you can get at least one pump running on the system you should be able to maintain a vacuum while emergency backup power is operational
2. Try to get a pump on the growth chambers before the cryopanels stop pumping!
  1. Echo BC ion pump and GC cryo pump will be operational on emergency backup power. Carefully open gate valves to allow these to pump on the entire buffer-growth chamber vacuum system (Bravo GC, Bravo buffer, Echo buffer, Echo GC).
  2. The Gamma ion pump controller display will give you a very close estimate of the vacuum pressure in the ion pump. Monitor this while opening the gate valves between chambers with unknown pressures!!!
3. It is normal for the chamber pressure to rise as the cryo panels warm up. The adsorbed gasses will boil off as the panels warm up.
When is the power coming back on?
1. Find someone from facilities (James, Darren, Jesse, etc.)
2. If the power outage will persist for many hours, ask how long the backup power will be operational (typically 6-8 hours without a fuel refill).
3. If you're unsure about how long the power will be out, prepare things to fail gracefully. Read through this list again and make sure that everything is in a quasi-stable state.

Lower priority things to check following a power outage

Flip the LC turbo pump switches to the off position
Unplug the VPI valves on the LC scroll pumps
Turn all flowmeters on the water manifolds to zero flow
Disconnect Varian turbo controller controllers from Varian turbos (Auger, Bravo phase separator, Echo phase separator)
1. The Varian turbos will spin up as soon as power is restored. Make sure the backing pumps are operational before the turbos spin up.
Backfill the BOS with N2
Backfill the Auger with N2
Unplug cryo power cables, monitor the cryos to make sure they don't over pressurize and burst
Setup scrolls pumps to regen the cryos after power comes back on
Turn off non-critical power supplies so components don't turn on in an uncontrolled manner once power is restored.
Put up caution signs on the MBE lab doors
Grab your respirator in case you need it to enter the lab later

SRB thoughts

Does Neslab restart gracefully? (Emailed Wistey to ask)
Better timeline
1. First few min – save Al and group-V valves
2. First hour – prevent stagnant water freezing, prep machine for graceful restart
3. 3 hours as lose LN2 – get a pump or two running and pumping on Bravo and Echo

Old notes here:

Power Outage (Unplanned) Stanford Notes

Edited 16/11/10 by KMM from personal notes taken during 2016 power outage.