Hang around high-tech, or any other industry long enough and you learn that headline-worthy bad news comes in mostly predictable flavors.
There’s good old executive malfeasance, often complicated by breathtaking greed and/or egotism. Plus, don’t forget what might be called Stupid Employee Tricks which can range from simple misadventures to cluelessly earnest activities whose idiocy or sociopathology is utterly lost to those involved. To be fair, not all negative headlines are internally-created, so be sure to mention shady activities by associates, like contractors and partners.
Then there’s faulty/broken technology news. It’s difficult to speak of these events generally since they can range from marginal quality or manufacturing issues to catastrophic device failures. But despite their scope, what happens and how issues are corrected can get to the very soul of a company.
Why so? Because since those processes are often also controlled by executive fiat and board-level decision making, their impact on a company’s brand and core strategies can linger for months or years.
However, in best case scenarios, direct, intelligent action can help resolve the problems far more quickly and effectively than might be expected. That brings us to the current situation with Intel and the design points reported early in January that make many of its chips susceptible to security exploits called “Meltdown” and “Spectre.”
Bad news management
Before getting into Intel, consider how tech-related bad news moves through the media and marketplace and how a company’s response can impact that process. Generally, these stories break down into those that are easily understood and those that are not.
In 2006, dramatic videos of laptops exploding into flames led to investigations concluding that the batteries and/or charging systems were faulty. That resulted in Sony, which manufactured the batteries, and numerous partner OEMs eventually recalling nearly 10 million affected laptops and battery packs, resulting in the largest (till then) recall of consumer products.
Unfortunately, Sony’s investigation of the problem lagged both its partners’ responses and continuing news cycles. The company finally resolved the situation but took months to do so, eventually costing hundreds of millions of dollars and causing massive negative impact on Sony’s brand.
Apple faced a far less dramatic problem with the 2010 “Antennagate” scandal whose resolution was complicated by the hubris of company execs, including CEO Steve Jobs. In essence, the then-new iPhone 4 utilized a new antenna design that could be disrupted if it were held in a certain way.
Jobs and Apple vehemently insisted the problem was a non-issue, but customers and shareholders disagreed. The eventual resolution—offering free “bumper” cases and $15 rebate checks—cost Apple about $175M and unnecessary public embarrassment.
Unless they affect a large number of consumers, technically complex problems tend to follow a different path than high visibility scandals. While such stories may make an immediate splash in the mainstream media, those narratives often devolve into narrower coverage by specialized industry and/or financial news outlets. That’s especially true if the companies involved are forthright and transparent in handling the issue.
Project Zero and Meltdown and Spectre, oh my
Which brings us to Intel and its products’ vulnerability to potential “low probability, but very high impact” security dangers. These exploit a race condition design used in many modern CPUs, including numerous Intel Core and Xeon processors that optimizes memory access and privilege checking during instruction processing.
The theoretical potential for such exploits has been known for a while. However, by mid-2017 independent researchers, including some working with Google’s Project Zero identified a large class of security bugs called “Meltdown” along with a related CPU vulnerability they called “Spectre”. That resulted in a blog report posted on January 3, 2018 detailing the research and potential dangers.
I won’t delve into the technological whys/wherefores of Meltdown and Spectre or the specific systems impacted. In the former case, the Wikipedia entry offers a good, regularly updated summary. In the latter case, Paul Tiech, a principal analyst at TIRIAS Research authored a terrific overview (available on The Next Platform) that answers most relevant questions. However, it should be noted that despite their potential danger, no successful exploits of these vulnerabilities have been reported.
Instead, I’d like to consider Intel’s response. While it’s clear that Meltdown and Spectre are not (as some outlets initially reported or implied erroneously) Intel-only issues, the company’s market leadership in desktop and data center silicon mean that systems based on its solutions constitute the vast majority of those impacted. Thus, the company’s actions offer interesting insights into how vendors can and should approach significant and very public problems.
Transparency and timeliness
Intel’s first public comments on Meltdown and Spectre came on January 3rd, the same day Project Zero published a blog post on the vulnerabilities. The company addressed the subject in a straightforward manner, noting the work it is pursuing with partners, including AMD, ARM Holdings and OS vendors to fix the issues. Less than a week later, Intel CEO Brian Krzanich began his keynote at CES 2018 with comments on Meltdown and Spectre, by reaffirming the company’s dedication “to keep our customers’ data safe.”
Krzanich also described plans to issue updates for more than 90 percent of CPUs manufactured in the past five years “within a week and the remaining by the end of January.” That won’t be an easy task, mainly because of how initial software patches and firmware updates are impacting system and application performance. Since there are wide disparities in those effects (varying according to processor, memory and other system resources and the applications and workloads involved) there isn’t anything like a “quick fix” in the works.
In addition, the road hasn’t been free of potholes. Understandably, Intel and its partners are pushing ahead as quickly as possible. However, the first patches and updates issued on January 4th unexpectedly caused some systems with Broadwell and Haswell CPUs to repeatedly reboot. On January 11th, Intel addressed that point and promised it would be resolved. A report on the root cause of the reboot problem was published on January 22nd.
In all, Intel has issued nearly a dozen press releases and announcements in January related to Meltdown and Spectre. Those included a Security First pledge detailing its commitment to focusing on the core needs of customers and to communicating its progress in a clear and timely manner. In addition, company executives have hosted four calls with industry analysts to discuss Intel’s progress.
That’s all to the good for the short term but how does Intel’s situation look further down the road? There are obviously formidable challenges. That includes designing future CPUs that are immune to these exploits yet still deliver the levels of performance customers require. Those aren’t unattainable goals, but they will require Intel and other vendors to institute significant design and, possibly, manufacturing process revisions.
There are also separate yet still associated issues, including a sale of restricted Intel stock and exercises of share options by Brian Krzanich a few days before the Project Zero report posted. The unfortunate timing of the sale sparked speculative press stories and analyst commentaries and calls by two U.S. Senators for an SEC investigation into the sale. Intel says it will cooperate with any investigation, reflecting the same transparency the company is exhibiting elsewhere. There are also, unsurprisingly, at least four lawsuits related to Meltdown and Spectre in the works.
Is there any evidence that the specter of these or any other challenges is melting down Intel’s strength of will or good intentions? Hardly. In fact, the company appears positively energized and confident that it understands the central problems and how to successfully mitigate them. It is also intent on issuing corrective updates and patches for existing systems as quickly as possible and is working on next generation solutions that will avoid the vulnerabilities completely.
Overall, it’s difficult to see how Intel could have better addressed or managed such a highly complex, difficult and challenging situation. Could it have done worse? Even the briefest reading of technology history will reveal unforced errors and sorry paths taken by many vendors that Intel has understandably and successfully avoided.