Disaster Recovery
by Larry Mainers
AN OUNCE OF PREVENTION IS WORTH A POUND OF CURE. WHEN IT COMES TO CLEANROOM DISASTERS, HAVING A THOUGHTFUL CONTINGENCY PLAN IN PLACE WILL ACCELERATE YOUR RECOVERY TIME AND GET YOU BACK ON LINE.
 Click here to enlarge image
|
The recent tragedy at the World Trade Center in New York powerfully illustrates how an entire country can be caught off guard by an unexpected disaster. Disasters are personal; they affect people in ways you can't imagine.
The place in which you work can be transformed into something you don't even recognize. Expect shock. Expect confusion. Expect your people to become immobilized.
By arranging restoration groundwork today you can save yourself and your company tremendous time and money in the case of such an event.
More importantly, a well-thought-out restoration plan will quickly restore your company to normal operation, which in turn affects the lives of all who benefit from its production.
When you least expect it
Cleanroom disasters come from the least obvious sources and always at the most inopportune times. Because there's no predicting such an event, every cleanroom manager must consider a thorough contingency strategy, for all successful restorations stem from thoughtful planning.
Yet, history proves that disasters still occur even when the best contingency plan is in place. Take for example a production manager I recently worked with who had his ISO Class 5 (Class 100) cleanroom affected by an event far from the cleanroom floor despite his keen disaster-prevention program.
A fire started in an electrical room where cardboard boxes were stacked too close to a hot wire. Fire was kept to a minimum, but the smoke set off a succession of events that led to a power failure throughout the fab and a complete smoking of the cleanroom envelope. Nothing escaped the insidious gas—product, tools, support systems or employees.
True, this is the worst nightmare that any cleanroom production manager can have. Consequently, it is imperative to develop a well-designed preventive maintenance program along with an organized disaster recovery plan to help get things rolling should such an event strike.
Professionals familiar with cleanroom disaster preparedness can create a “risk impact analysis” to address the various possibilities surrounding your cleanroom production area. This form becomes the backbone of your business continuance program; and updated annually, this living document will go a long way in preventing a crisis and assuring timely reaction should one occur.
What do you do if you find yourself in the middle of a cleanroom disaster? How do you restore cleanroom production in the wake of a disaster? Let's take a look at some of the key elements and issues that will help you develop an effective disaster recovery plan.
Systems approach to production recovery
There are three key considerations on the road to restore cleanroom production: restoration of production equipment, tools, mini-environment and automation systems; restoration of environmental equipment/ systems such as filters, air handlers and duct systems; and restoration of all cleanroom envelope surfaces such as walls, floors, ceiling grids and furniture.
These three areas must receive specialized attention that meets the unique properties of each component. To achieve this, specialists that understand optimum operational conditions must work in concert with one another to meet the demands of each system being restored. Key restoration services include environment testing, tool, computer equipment, AC filter systems and facility restoration.
The need for a coordinated effort cannot be underestimated. Without coordination, the likelihood of cross contamination increases, as does damage of delicate electronic equipment.
The optimum system of response is one that is planned far in advance of a disaster. Here key restoration and planning personnel are identified and designed recovery schedules are created. One might see the similarities to sinking ship drills where the sea captain knows who goes where, when and how. Each life-saving action is thoroughly planned and rehearsed, and this is done each time a new crew person comes on board to ensure their understanding of what role each one will play.
Just as important as what you restore is when you restore it. An effective restoration must follow a sequential order, otherwise recontamination of cleaned surfaces may take place. Restoration is not simply a laundry list of areas to be cleaned. It involves a specific order so that the most recent procedure complements the previous one, laying the groundwork for the procedure that must follow. For example: cleaning sensitive optical points prior to securing ambient air cleanliness would prove to be futile.
A first step in restoration is to plan the logical order of cleaning each cleanroom system. What must be restored first, second, third? What gets attention prior to the effective operation of each item? Which comes first: cleanroom surfaces such as walls, floors and interstitial? Or is it operational equipment such as air filters and ducts? Each system will have an effect on the next item to be restored. In many cases steps must be repeated in stages until systems are restored to their optimal operating condition. This staged approach is familiar to those in the clean-build construction industry.
A pattern to follow
Similar to the construction method used in building cleanrooms, post-disaster recovery employs techniques to restore production environments. Clean-build construction utilizes an increasing stage of cleanliness as each system is deployed, as does the restoration of cleanroom production. Taking the post-disaster site from contamination to clean is just like the new construction site during the dry-in phase as it moves to the optimal operating order found when construction is complete.
Key components such as cleanroom protocol, top-to-bottom cleaning, new blow-down dates, systematic restoration of cleanroom components, all reflect familiar clean-build practices. This certainly includes various safety protocols such as confined space access, lock/tag programs, and analysis of chemical residue, which poses serious health risks.
The next important step is identifying what contaminate must be removed. Smoke, water or chemical contamination are by far the most common. Each carries its own peculiarity, and in most cases are found in combinations. For instance, if water or chemicals are used in concert to dose a fire, the water may interact with chemicals during the event through absorption.
Contaminate particulars
Fire/smoke: Smoke, by far, is the most difficult cleanroom contaminate. Because smoke is a gas, with heavy levels of particles, it represents the worst of all Airborne Molecular Contaminates (AMC). Because AMCs are routinely monitored in a normal working environment for parts per million or billion, imagine the effect when large-scale common sources are burned. The following statements are exacerbated when components in the cleanroom catch fire—imagine commonly found AMCs on steroids.
Notice what trace amounts can do to production. This should give you an idea of the volume of effect of a large-scale contamination due to fire.
Trace metals (SEMI F21-95 Class MM, molecular metals): Trace metals in cleanroom air can cause degradation of the electrical properties of the silicon substrate, such as carrier lifetime and leakage currents, and can affect gate oxide integrity and threshold voltages.
 Click here to enlarge image
|
Condensable organic compounds (SEMI F21-95 Class MC): Organic compounds in cleanroom air may adversely affect many processes in the fab, including high-temperature processes, metrology, cleaning, etching, oxide growth and film deposition. Organophosphorus compounds in air can adsorb onto silicon wafers and cause incorrect doping.
Acids and bases (SEMI F21-95 Classes MA and MB): Chemicals used in the fab are present in the air and can contaminate wafers, ultrapure water sinks, chemical baths and other processes within the fab. Corrosion, hazing and deterioration of DUV photoresist are all a result of acidic or basic cleanroom air contamination.
Since AMC’s are routinely monitored in a normal working environment for parts per million or billion, imagine its affect when large-scale common sources are burned as in this cleanroom disaster (left).Click here to enlarge image
|
Fire will transform common fixtures in the cleanroom into huge amounts of metals and organic compounds.
Ironically, the very process of putting the fire out creates two significant acids: hydrobromic (HBr) and hydrochloric acid (HCI). Halon 1211 and 1301 are frequently used in commercial fire suppression systems. While these products are highly effective, both produce hydrobromic acid when in contact with hot metal or flame. This acid is aggressive and corrodes ferrous metal as well as copper, brass, aluminum, zinc, and even gold. Hydrochloric acid is not only a byproduct of Halon but is also produced when polyvinyl chloride (PVC) is heated to over 200°C (392°F) and then combined with water either from the fire-suppression system or combustion byproduct.
These corrosive chemicals attack delicate electronic circuitry. The list of dangerous chemicals produced by fire is long: carbon monoxide, sulfur dioxide, nitrogen dioxide and phosgene gas. Additionally, virtually all smoke, ash and soot films are highly conductive, which is extremely detrimental to cleanroom precision manufacturing.
The fire/smoke scenario requires a well-planned recovery program. Because of smoke's ability to quickly affect all areas of the cleanroom, recovery programs must include a top-to-bottom response. This will require the services of professionals at every level of cleanroom operation, including planners, equipment manufactures, NEBB certified testers, air balancers, expert cleaning techs, AMC testers and computer restoration experts to mention a few. Coordination of each cleanroom discipline will ensure a thorough restoration.
Chemical contamination: Unlike smoke, chemical contamination does not necessarily require an entire cleanroom envelope restoration. One must determine the extent of the chemical affect on the cleanroom environment and quickly arrest the chemical's ability to distribute itself within the environment.
This action would be followed by a complete investigation of surrounding areas and the testing of air and surface contamination. Key to this restoration is the testing of porous surfaces such as filters and concrete substrate that can absorb chemicals. Cleanup requires neutralizing procedures along with removal of the neutralizer itself. Once again, the order of restoration steps is critical to prevent cross contamination.
Water: The very nature of water as it relates to electronic equipment requires special action. Water is one of the best agents for diluting chemical residue; in turn, it can become a carrier and a host for chemical reactors that can damage sensitive electronic equipment. Water sediment can dry onto circuitry leaving with it the potential to etch away protective coatings. Variable sediments include alkalinity (pH), total dissolved solids (TDS), suspended material, dissolved gases, pathogens, organic material, microorganisms, electrolytes, and oil. Significant are total organic compounds (TOSs), which are a closely monitored type of particle that can act like catalysts to other organic substances, causing the resultant particle to grow.
The value of a certification program
We have discussed the need for an orderly restoration that prevents cross contamination. Our next step concerns testing the effectiveness of the restoration. Tests should be performed on everything. The cleanroom must be recertified and each tool must be retested. For confidence to return to production, there must be a top-to-bottom testing program that utilizes a NEBB-certified company to ensure that particle count, airflow management, pressurization, and AMCs are all within operational standards.
For confidence to return to production, there must be a top-to-bottom testing program that utilizes a NEBB certified company to ensure that particle count, airflow management, pressurization and AMC’s are all within operational standards.Click here to enlarge image
|
Key tests can be performed by particle counters, minienvironment testing (Qveiw), surface particle detectors (QIII+), 3D anemometers, air foggers and AMC air sampling devices. At the end of the day, each item must be accounted for to indicate it is safe to return to operation.
Been there, done that
Hopefully, you have never been through a disaster. This is one of those life events you will not want to experience.
Still, there are those who make a living serving the needs of those who find themselves in these situations. Don't start your search for a solution after the cleanroom has been shut down. Instead meet with those familiar with cleanroom environment disaster recovery. They can help you create a risk analysis/restoration plan that can go a long way in preparing the response team for this unlikely event. Just as important, they may help you to identify ways you can prevent a disaster.
Larry Mainers is vice president of site services at Fremont, CA.-based Pentagon Technologies, a provider of products and services for defect reduction and yield enhancement. For the past 18 years he has been involved in the contingency and natural disaster response planning. An expert in the operation of cleanrooms he has managed key disaster recovery programs for several fortune 500 companies involved with cleanroom production.
