US Grounds Boeing 787, The Anatomy of a 20 Year Denial, and a Bubble of Self-Confidence

http://online.wsj.com/article/SB10001424127887323301104578257862879948292.html

An almost sad tribute to Boeing’s 787 above, attributes the recent Infernal Batteries problem of two 787 a week apart from each other, both while in normal operations, to the growing pain of “innovation”.

Except, this was not “innovation”.  Using such batteries in airplanes perhaps, but the battery technology, Lithium Cobalt Oxide type, is not new.  It was invented in the late 1970’s, and have been in prolific use in cell phones and laptops since 1990’s.

Back in the early 2000, there began rumors around the world of incidents where cell phones exploded.  First, it became an urban legend attributing such explosion to sparks generated by an operating cell phone near a gas refilling station.  But this was quickly denounced as “urban legend”.  https://canadasafetycouncil.org/safety-canada-online/article/cell-phones-risks-and-rumour

Then some users noticed that it was actually the batteries that were exploding or bursting into flames (even in laptops).  Companies attributed and blamed the problem on “counterfeit batteries” from China.   http://web.archive.org/web/20060109013055/http://www.kyocera-wireless.com/news/20041028_2.htm, recalling proactively batteries to look for “counterfeit”, but unfortunately, no report was ever made on the search.

Others simply dismiss the various documented incidents to “user error”, “external short” (another way of saying user error).

But then quietly and occasionally, Companies were aware of the problems, and began a few recalls.

http://en.wikinews.org/wiki/Nokia_issues_BL-5C_battery_warning%2C_offers_replacement

http://www.mukamo.com/nokia-n91-cell-phone-explodes/

http://pubs.acs.org/cen/news/84/i34/8434notw4.html

http://www.theinquirer.net/inquirer/news/1042700/dell-laptop-explodes-japanese-conference

In March 2007, Lenovo recalled approximately 205,000 batteries at risk of explosion. In August 2007, Nokia recalled over 46 million batteries at risk of overheating and exploding.  In December 2006, Dell recalled approximately 22,000 laptop batteries from the US market.  Approximately 10 million Sony batteries used in Dell, Sony, Apple, Lenovo/IBM, Panasonic,Toshiba, Hitachi, Fujitsu and Sharp laptops were recalled in 2006.

Governments, including US, bought into the “theory” of “external short” as the main cause of many Flaming batteries, and imposed restrictions on travelers carrying Li Ion batteries in 2008, http://www.robgalbraith.com/bins/content_page.asp?cid=7-9206-9211.  (with new revisions to be enacted in 2013).

But this theory didn’t explain many of the incidents where the batteries combusted while appearing in normal operations (ie. no “external short”).

And then, like magic, the reports of the incidents died down, public interests waned, and recalls became less public (while still occasionally happens:  http://www.cpsc.gov/cpscpub/prerel/prhtml11/11234.html HP recall in 2011, About 162,600 batteries (54,000 and 70,000 batteries were previously recalled in May 2010 and May 2009, respectively)).

*

If one does some careful search, one can find a continuous documented history of the SAME exploding batteries across industries of various applications and devices, from 2004 to today.

How can industries and companies continue to make such dangerous products without knowing exactly what’s causing the problems?

Well, it’s easy, call it a “bad batch” or “counterfeit”, and blame it on someone else, like China.

Yet, the problem keeps showing up again and again.

*

I’m not suggesting that there was some massive CONSPIRACY of silence, but I think it is a Bubble of over-confidence, similar to what Alan Greenspan once called the US economy at the height of its bubble, “irrational exuberance”.

The world electronic and battery industry was in a 20 year long “irrational exuberance”, celebrating our mobile gadgets packed with explosive energy canisters, marveling all the new fun things we could do with them, and dismissing the annoying “incidents” here and there.

*

But the batteries were growing EVERYWHERE.

The same battery type is used in the Chevy Volt, which encountered 2 separate incidents of battery explosions (1 spontaneously while parked) in 2011 and 2012.

US FAA recorded more than 100 incidents linking lithium ion batteries in “CARGO” hold to onboard fires over the last two decades, as more and more companies are shipping the batteries around the world in planes.  In at least 2 recent incidents, the planes’ cargo caught on fire, and the planes crashed (1 in the South China Sea), killing the pilots.  US even instituted (arbitrary) restrictions on shipping the batteries:  NO MORE than 66 lbs in EACH package, in a passenger flight’s cargo!!  (What makes more than 66 lbs of that stuff more dangerous than less than 66 lbs??)  http://www.bloomberg.com/news/2012-02-14/aircraft-fires-tied-to-lithium-battery-cargo-prompt-new-un-rule.html

A recent 2011 report on failure analysis of Lithium Ion Batteries seemed to placate the industry’s dismissive attitude about the dangers of the batteries, by attributing most of the Cargo plane incidents to “external shorts”, see Page 79 Table 7 in http://www.nfpa.org/assets/files/pdf/research/rflithiumionbatterieshazard.pdf.

Yet, contrarily, most of the personally transported battery explosions were attributed to “internal fault” (Ie. some battery defect), Table 8.

What is clear is that the Industries and the Companies, never accepted the idea that these batteries could actually “spontaneously” combust while not in operation (as in the case of the Chevy Volt), and even if they did combust IN operation, there must be some other reasons (such as “over charging”/ user error).

*

Unfortunately, sooner or later, this kind of place blame elsewhere will catch up with you.

Forward to today, why did Boeing 787’s catch fire in their batteries?  Because the Industry thought, based upon its own previous 2 decades of assumptions about the Lithium Ion batteries (Cobalt Oxide type), that these batteries were already “proven” technology, (discounting and dismissing all of the recalls and problems as someone else’s fault).

If I was an engineering manager with such beliefs, I would also be as confident about the Li-Ion batteries as the Boeing managers.

So for their showcase 787’s, Boeing contracted a reputable Japanese Battery maker that also supplies batteries for NASA and the International Space Station.  They took the battery design of the cell phones and effectively multiplied it by over 100 times, put all that highly energized chemical into a big metal box, and stuffed it in the underbelly of the new 787 (VERY CONFIDENTLY).

FAA was infected by the confidence of so many engineers and managers around the world.  A few cautious voices didn’t stop the course of the 787, because America needed it to compete with Europeans’ next Airbus, and to show off its “innovation”.

*

I don’t mean to sound to gloat.  Fortunately, no one was injured in these incidents with the 787’s.  But several airlines are being seriously impacted financially by this, among them, Japan Airline and Air India, and several others, like China Eastern, are canceling their orders of the 787’s.

Part of all this history, must place the burden of some blame on the Western media, who effectively stopped questioning the industries in the past 20 years, even though there are plenty of data to point to a pattern of problems in the technology.  And they believed too much in the superiority of their own “innovation” and quality, while becoming naively willing to place the blame of problems to “others”, such as China.

Those of us who worked in the industry and had our doubts, did our parts to highlight the problem.  (I personally sent many emails to managers and colleagues about this issue).

Boeing Executives today are still confidently announcing that they believe they can “fix” the problem “soon”.  I don’t know how many rational engineers and scientists can make such bold claims without actually knowing the EXACT nature of the problem.

I only hope that the 787 problems will spark a serious debate and investigation into the 20 year old technology issue, instead of just letting it slide again on some other “blame others” excuses.  (Some even blamed this on “outsourcing”, or “green technology” in general).

As a former engineer, I always believed that ALL technologies have problems, and one can ONLY solve them by facing them, not avoiding them.  Blaming this problem on “outsourcing” and others don’t get to the REAL cause of the problem, which has NOTHING to do with who made them (as 20 years of history can attest to).

While the engineers should work on this problem, the MEDIA needs to get real and stop deluding the public into believing a simplistic explanation.

The TRUTH is, that the tendency for fire in the Lithium-Cobalt-Oxide type batteries is a SERIOUS problem with yet unknown cause, for the last 20 years!  Sooner the media and the public understand the magnitude of that technological set back, the better we can appreciate the potential danger we are all in today.

*

Update 01/24/2013.

Another one talking about the event in 2006:

http://www.suntimes.com/business/17780000-420/ntsb-boeing-787-battery-shows-short-circuiting.html

 

NTSB spokeswoman also said there appeared to be also “short circuit”. But in this case, I think they mean an External short circuit.

““…These events should not happen as far as design of the aircraft,” she said of the battery problems. “There are multiple systems to protect against a battery event like this. Those systems did not work as intended. We need to understand why.””

She’s ASSUMING of course, external short. Yes, if it was an External short, the safety circuits in the system can detect them and prevent them.

But if it was NOT an external short, but an INTERNAL short, none of the existing safety systems can stop a thermal runaway reaction INSIDE the batteries.

From a US government study from NREL: http://www.nrel.gov/vehiclesandfuels/energystorage/pdfs/45856.pdf

“Thermal signature of the short is hard to detect from the surface.”

Additionally, page 24, showed a study of 1 type of internal short (Cathode layer bypassing) by GS Yuasa in 2008, the SAME battery company that made the BOEING 787 battery. Showing that they knew about this kind of problem.

*

https://batteryworkshop.msfc.nasa.gov/presentations/08_Safety%20Design%20Features%20of%20GS%20Yuasa%20Lg%20Li-Ion_TInoue.pdf

This is GS Yuasa’s marking pitch to NASA, about their large size Li-Ion cells designed to prevent some of their scenarios, but they continued to blame previous incidents to “indention process, cell structure, and operational condition such as charging,” While claiming at the same time, that “large Li-ion cells are designed in view of above points. They have never showed internal short accident through more than 10 years production. (>5000 cells).”

However, I note, their incident data was inherited from Sony, which also showed very low incidents numbers.  And GS Yuasa did not perform any real internal short tests, but merely noted that in their standard testing with impact tests, they didn’t see any internal shorts.  That would only be true if their theory is correct that internal shorts are caused by impacts (slide 21), but that theory is completely unproven.

*

Regarding the 2006 incident mentioned in comment, here is the administrative court paper detailing the fire and the events surrounding it.

http://victimsoflaw.net/Leon_Securaplane_Opinion.pdf

Particularly interesting, there was “a discrepancy between the top-level schematic and both the intended design for the BCUs and the actual design of the Battery Charging Units (BCUs) that were shipping”, admitted by Securaplane, the company that made the charging units for the battery.

While the discrepancy was ultimately corrected, allegedly, and none of the errors made into the Boeing 787 planes, there was 1 problem:  the discrepancy of design was in some or all of the TESTING BCU’s used, particularly during the FIRE.  Securaplane personnel were told to manually modify the BCU’s to correct the design discrepancy, and then test them.

That means, the entire testing using the modified BCU’s was tainted.  As it happens, a FIRE burnt down the building.  The investigation afterwards attributed the fire to the technician’s operating error in connecting the wrong cable.  Well, that’s just guessing.  They don’t really know if the modification of the BCU really did something, or perhaps even the batteries themselves were bad.

The point is, when you do testing with modified designs, you don’t really know if the actual design will work.  The result is tainted!

Even more interesting:  The Securaplane technician complained to the FAA requesting an investigation, and…

Stucky confirmed that FAA officials did visit; they called in advance to tell Stucky they were coming. The FAA investigators primarily met with Stucky, but did talk to a few production assemblers. During the site visit, investigators didn‘t ask questions about the schematic error, their focus was more on getting a “feel” for the company and how comfortable workers were with approaching Stucky about safety issues.

In other words, FAA didn’t take the fire that seriously, and accepted the investigative results of Securaplane and GS Yuasa, which blamed essentially “operator error” on the technician.