Disaster in the Cleanroom: Damage Control, Containment, and Recovery
Recovering from a cleanroom disaster involves overcoming two challenges: restoring the cleanroom environment and recovering the processing/ manufacturing equipment to cleanroom specifications.
By Bill Williams
You`ve just hung up the phone after a call from the facilities manager. He said that the fire in the building is out, but the wafer fab has been smoked. All the product on the fab floor is ruined, production has stopped, your product delivery schedule is shot–you are facing a huge loss. There is no way you can prepare for the feeling you have in your stomach right now, but you can prepare for the procedures you will initiate and execute in the next few minutes, hours, and days.
Disaster scenarios cover a wide range of possibilities–fire, smoke, flood, chemical contamination, earthquakes, employee sabotage, terrorism, power outages, etc. Depending on the disaster, a company`s responsibilities could include ensuring the safety of its employees and the well-being of their families, cooperating with emergency crews, planning and executing the business recovery, maintaining corporate security, handling public relations, re-instilling confidence in its customers, providing payroll for its employees, evaluating alternate production possibilities, and the list goes on and on.
Your company`s disaster recovery representative should have performed a risk impact analysis addressing the possibility of various disasters and determining an appropriate corporate course of action for each probability. The analysis should include identifying companies that can provide disaster recovery services for the various scenarios. If this has not been done, it needs to be done immediately.
There are two important aspects to the disaster recovery challenge: (1) restoring the environment of the cleanroom, and (2) recovering processing/manufacturing equipment and restoring it to cleanroom specifications in order to resume production.
Recovery procedures need to be planned and executed with a view to restoring a system, not just a room.
Systems Approach to Recovery
All aspects of cleanroom design collectively accomplish one objective: minimize and control particulate. This is done by controlling the quality, quantity, differential pressure, and speed of air. At the core of the cleanroom is the air-handling and filtration system. All other maintenance procedures in the cleanroom support the function of the air-handling system and serve to minimize the challenge to the HEPA/ULPA filters. This system is what makes a cleanroom, and all restoration efforts must be carried out from the perspective of a “systems approach” in re-establishing this dynamic entity.
Unlike other disaster recovery projects, cleanroom recovery is unique. Recovery procedures need to be restricted not only to a sequential order, but also to personnel who are qualified in cleanroom procedures and equipped to accomplish the necessary tasks. All of this requires intricate planning, coordination, timing, and excellent on-site manage ment. The restoration process is not just the successful completion of a set of required procedures. It involves accomplishing those procedures in a specific order, so that the most recent procedure complements the previous one, laying the groundwork for those that follow. This will not be a new concept for an experienced cleanroom disaster recovery company.
Assuming minimal structural damage, in a post-disaster environment, the viability of recovery options and the speed at which recovery occurs are determined by the source and extent of contamination. That being the case, some particulate considerations are in order.
Disaster-Related Particulate Considerations
Unwanted particulate cause faulty products, which translate to financial losses and customer dissatisfaction. Identifying and understanding the particulate to be eliminated is critical to system design and procedures implemented to eliminate particulate. In a cleanroom environment that has been compromised by smoke, fire, flood, or chemical contamination, the particulate possibilities are exponential. Of greater concern are the effects of that particulate on personnel and equipment. It is necessary, therefore, that the facilities manager overseeing the recovery be cognizant of the range of contaminant possibilities and how they impact recovery options and procedures.
Smoke and Fire
Environments affected by smoke and fire represent the greatest challenge to the cleanroom disaster recovery team. They are chemically active, aggressive, and time sensitive.
In a post-smoke/fire environment there are numerous chemical reactions taking place due to the by-products resulting from the fire, in addition to the particulate considerations surrounding the compromised manufacturing process. Chemical reactions begin taking place immediately, as the combustion by-products break down into primary elements– usually acids.
Ironically, the very process of putting the fire out creates two significant acids: hydrobromic (Hbr) and hydrochloric acid (HCl). The active chemical in fire-extinguishing systems is halocarbon. Halon 1211 and Halon 1301, manufactured by Dupont, are frequently used in commercial fire suppression systems. While these products are highly effective, both produce hydrobromic acid when in contact with hot metal or flame. This acid is very aggressive and corrodes ferrous metal as well as copper, brass, aluminum, zinc, and even gold. Hydrochloric acid is not only a byproduct of Halon but is also produced when Poly Vinyl Chloride (PVC) is heated to over 200°C or 392°F and then combined with water from the hoses of emergency fire crews or as a combustion byproduct.
Why is this significant? Because of the very corrosive nature of the chemicals. They immediately begin attacking electronic and intricate process manufacturing equipment. Fungi and bacteria will also produce corrosion, even on stainless steel and copper/nickel alloys, and on cast iron and aluminum. Even for a simple structure fire, the list of dangerous chemicals goes on: carbon monoxide, sulfur dioxide, nitrogen dioxide, phosgene gas, etc. Virtually all smoke, ash, and soot films are highly conductive. The post-smoke/fire atmosphere is extremely detrimental to those environments surrounding electronic, pharmaceutical, optics, and other precision manufacturing.
One piece of encouragement is that, for electronic and manufacturing equipment, a far greater degree of restoration is possible if immediate and proper actions are taken. This principle applies to all situations where the normal operating environment for the product and equipment in question is compromised, and where corrosion is taking place.
Fires and other disasters that occur outside the cleanroom and that generate airborne particulate potentially pose some of the same chemical and particulate hazards as a fire inside the cleanroom. Even though the cleanroom ventilation system is sealed from the rest of the building, the infusion of external makeup air for the air-handling machines (approximately 20 percent of total airflow requirements) creates a potential breach in the integrity of the cleanroom. Why? When smoke, fire, or chemical contamination occurs in close proximity to the intake for the makeup air, high concentrations of corrosive particulate present in the makeup air are then transported to the filtration system. This causes a greater upstream challenge to HEPA/ULPA filters than they were designed for, resulting in downstream particulate bypass in quantities significantly in excess of allowances for the affected cleanroom. In addition, if the particulate is corrosive, a compounding problem has been created. Even separate, sealed ventilation systems for cleanrooms are not a guarantee of the integrity of the cleanroom environment.
Chemical Contamination
Chemical contamination scenarios present some of the same problems as smoke/fire scenarios. However, the spectrum of potential chemical reactions is usually limited and more easily identified, due to the nature of chemical contamination. The most critical procedural step at this point is to first identify the chemical(s) in question and the range of secondary and tertiary reactions taking place as a result of the contamination. Once identification is accomplished, appropriate neutralizing procedures can be initiated. In certain chemical spills, as in all post-smoke/fire scenarios, the environment is chemically very active, aggressive, and extremely corrosive. However, even in these environments, equipment recovery can be rapid and straightforward if necessary preventive measures are taken quickly. The importance of immediately neutralizing the environment resulting from chemical contamination and smoke/fire cannot be overemphasized. Failure to accomplish this procedure can result in immediate corrosion, rendering equipment totally unrecoverable and adding significantly to a company`s financial loss. The use of improper techniques to address the neutralizing procedure can further complicate the problem. Generally, this procedure does not add to the recovery process in terms of cleaning or decontamination, because the cleaning process used in the recovery will also clean the neutralizing agents. Neutralizing the environment buys valuable time. Remember, the company is having to make literally thousands of decisions before and during the recovery. One of those decisions is recovery vs. replacement of affected equipment. If protective measures are not taken, recovery might not be an option.
Flood/Water Damage
Recovery from the presence of water in the cleanroom involves more than removal of water and the return to a 68-72°F 50 percent relative humidity environment. In fact, removal of the water is not the first objective. Securing all electrical power is the first order of business. In a wet environment, electricity–not water–is the greater risk to electronic equipment. Generally, once electrical equipment is saturated, more water will not cause further damage. The objective is not to let the various sediments in the water settle on the equipment and then dry. Caked-on sediment requires far more sophisticated recovery measures than equipment that has suffered only water damage. A potential complication again points to the need for a well-coordinated “systems approach” to recovery.
In a post-flood environment, the source of the water dictates the restoration measures to be taken. At one end of the spectrum, the source might be a broken freshwater pipe; at the other end, a river at flood stage, resulting in “black” water. Water quality variables that need to be addressed are: total alkalinity (pH), total dissolved solids (TDS), suspended material, dissolved gases, pathogens, organic material, microorganisms, electrolytes, oil, and chemicals, to mention a few. For instance, microbial activity is a significant consideration. Organic, living specimens can grow and multiply if not properly handled, creating an ongoing and escalating disaster long after the water has gone. Significantly then, total organic compounds (TOCs) are a closely-monitored type of particulate in many cleanrooms because they act like catalysts to other organic substances, causing the resultant particulate to grow (See Figure 1, p. 22). This is especially critical in pharmaceutical cleanrooms, where organic material is being combined to make drugs. Total alkalinity (pH) of the water needs to be monitored because it changes from a low pH (base) to a high pH (acid) as the water content decreases. Neither condition is desirable in the presence of sophisticated electronic equipment. Also, pH levels affect conductivity, as does the decay of organic materials.
A thorough understanding of particulate considerations is absolutely necessary on the part of the disaster recovery company providing such services. It, more than anything, impacts the company`s return to production and profitability.
Environment and Equipment Recovery Considerations
The environment recovery process is a function of the degree of damage experienced. In situations where there is minimal structural damage, recovery can be rapid for several reasons. To begin with, many companies have opted for the modular cleanroom concept, which allows for rapid replacement of parts or modular components. In more extreme circumstances, it allows for disassembly, relocation, and reassembly of the cleanroom, so production and revenue can resume. Modular construction also enables on-site disassembly, cleaning/decontamination, and reassembly, if needed. This is advantageous in situations where significant particulate contamination from smoke, fire, or chemical contamination is the primary consideration. Any disassembly and reassembly would initiate the required cleaning associated with construction Protocols 1-5, as necessary. In fact, much of the recovery can be accomplished under the guidelines of post-construction cleaning protocols.
Another form of new technology that can be utilized effectively in the recovery process is the use of mini-environments. Mini-environments are becoming more prevalent, decreasing the overall square footage requirements for critical clean areas. Additionally, since mini-environments are compact units, their recovery rate can be rapid as well, clearly shortening the recovery process. Because most mini-environments are portable, critical areas can be established in what were previously non-mini-environment cleanrooms, allowing production to resume on a limited basis, until the cleanroom can be reestablished at the required Class (1 to 100,000). For some process manufacturing companies, this provides welcome flexibility in the recovery process. Even in extremely large ballroom-style cleanrooms where minimal structural damage has occurred, recovery can proceed at a surprising speed. In these situations, the design of the cleanroom facilitates removal of equipment, allowing both the equipment and the environment to recover at the same time.
In addition to understanding particulate considerations, the accepted methodology for removing particulate must be understood. The scope of disaster recovery projects is such that post-construction cleaning protocols are required. These protocols establish a systematic cleaning of the entire cleanroom facility, including the fan deck, fab floor, subfloor plenum, ceiling, walls, piping, electrical conduit, bus and tag, fire suppression system, and the entire air-handling system. As discussed earlier, the air-handling system is the core of the cleanroom. Extreme care must be taken to ensure that all ducting, filters, fan assemblies, exhaust valves, pressure-modulating valves, and control mechanisms are completely cleaned or decontaminated. This involves strenuous work in confined spaces (ducting), and in some cases, may require robotic equipment. Depending on the source of contamination, there are, potentially, serious health and safety concerns for the personnel performing such work. The presence of PCBs and post-fire toxins can require the use of full-exposure suits and positive-pressure respirators. Cleanroom disaster recovery is not the place for a janitorial company. Disaster recovery is a highly technical discipline requiring specialized knowledge and equipment, and it should be performed only by those companies with the technical expertise.
The equipment recovery process can be rapid and happen concurrently with cleanroom recovery. In the absence of physical damage, extremes in temperature, or corrosive chemicals, equipment recovery is rather straightforward, and the use of portable cleanrooms offers the alternative of on-site recovery.
Most process manufacturing equipment used in the cleanroom is sealed to minimize contaminate. The protection offered by this design not only prevents contaminate from coming out of the equipment, it also keeps the disaster-related contaminate from going inside the equipment. Therefore, meticulous exterior cleaning/decontamination and application of post-construction Protocols 1-5 produce cleanroom-ready equipment in minimum time. If a piece of equipment must be disassembled in the process of cleaning, it should be done only by a technician trained on that equipment.
There are disaster recovery companies whose specialty is the recovery of electronic equipment.
Computer hardware and peripherals comprise a significant part of the equipment list in many cleanrooms. Restoration and recovery of computer equipment from smoke, fire, and chemical contamination can be equally successful. With the advent of quick-acting, low residue, and nontoxic chemistries, technicians can recover a whole range of computer equipment from PCs to mainframes.
Positive Leadership
A post-disaster environment is a highly emotional one. The normally clear waters of reason become clouded by the emotions surrounding such an event and by speculation about an unknown future. Recognizing the implications of this will contribute to a more discerning approach to managing what might seem to be extreme positions taken by management or fellow employees. A successful management approach to these issues entails instilling a sense of confidence and security in those who have direct contact with the disaster recovery representative. In a time of confusion, just knowing that:
someone is in charge
there is a viable solution to the disaster
it is being solved as fast as possible
the needs of the employees and the company are being addressed
provides emotional stability. Only then can the disaster recovery representative effectively marshal the talent at his disposal to address the challenges that lay ahead. n
Bill Williams is Director of the Disaster Recovery Division of TEC International (Roswell, GA). A graduate of the U.S. Naval Academy, he has extensive experience in military contingency planning, natural disaster response planning, and aircraft crash site management. He also has experience in the recovery of electronic equipment and is a consultant for IBM/ISSC for disasters involving cleanrooms, data centers, and mainframe computer equipment. TEC International has restored numerous high tech cleanrooms in Canada, Europe, South America, and throughout the United States. TEC International is a member of the IBM Critical Response Team. For more information contact Bill WIlliams at TEC International at (800) 526-1095.
 |
Before and after counts for Total Organic Compounds (TOCs) in rinse water for wafer manufacturing. Excessive counts result in contamination and, as a result, faulty parts.
 |
Application of a cleaning chemical is needed to remove smoke or fire damage to a circuit board.
 |
Meticulous cleaning procedures can be done in a controlled environment room.
 |
 |
A smoke and fire damaged circuit board before (far above) it has been chemically cleaned and restored. A clean board (above).
Do`s and Do Not`s…
Remember, the wafer fab is still smoked. Here is what you should and should not do.
Do …
1. Call your insurance company. Coordinate with the adjustor to contact your pre-selected disaster recovery company.
2. Instruct the facilities manager to secure all electrical power to the facility, if it is still on. (Operating electrical equipment in the presence of water or contaminant can cause significant additional damage not only to the equipment but to the integrity of the information stored in the computers.)
3. Secure all HVAC to contain the particulate problems.
4. Contact corporate security, if required.
Do Not …
1. Panic; the damage may not be as bad as it looks.
2. Employ a janitorial company. They do not have the technical expertise to effectively manage the disaster.
3. Initiate any procedures without coordinating them with the disaster recovery company.