Archive for metrics
Under the America Invents Act, the United States Patent and Trademark Office (USPTO) was given broad new powers to “correct” past mistakes in issuing patents through the power given to the PTAB, the Patent Trials and Appeals Board. The PTAB is an administrative law that decides issues of patentability, formed on September 16, 2012 under the America Invents Act. Their impact on patents, innovation, and the US economy has far exceeded anything contemplated when Congress debated this provision. They have become the “go to” route almost anytime significant patent litigation is underway, and the results have been devastating to patent holders. Large numbers of seemingly valuable patents have been invalidated and patent holders have faced huge costs and losses as opponents can launch repetitive assaults that need to be defended at great cost. For many of us in the innovation and IP communities, the term “patent death squad” sadly seems appropriate.
The PTAB consists of numerous “judges” who conduct trials on the patentability of patents that have already gone through years of examination at the USPTO. It’s a painful burden that often results in the USPTO saying, “Sorry — we messed up completely when we granted your patent that you struggled so many years to prosecute. Should never have been issued in the first place.”
But who are these “judges” that are causing such havoc? Gene Quinn of IPWatchdog.com has done some great investigative journalism and revealed that these judges are a far cry from what one would expect in terms of their legal experience. Many have just a few years of experience, which helps explain some of the surprising decisions they have rendered.
See “PTAB Judges Shockingly Inexperienced Compared to District Court Judges” by Gene Quinn at IP Watchdog, March 6, 2018. A short excerpt follows, but see the original article for key data and some incisive comments afterwards.
Inexperienced PTAB Judges
What was most astonishing is just how inexperienced many patent judges of the PTAB are compared to federal district court judges. For example, many PTAB judges were appointed to the PTAB at a time when they were associates, and in some cases junior associates.
This study uncovered several shocking revelations. First, 12.64% of PTAB judges were appointed with less than 5 years of experience prior to their appointment as APJs (i.e., 5 years or less removed from graduating from law school), while some PTAB judges were appointed with as little as 2 years of experience. Indeed, 7.47% of APJs had 4 or less years of experience when they were appointed to the PTAB. More than one-third (36.21%) of PTAB judges were appointed with 9 years or less of experience….
The America Invents Act (AIA) invests PTAB judges with extraordinary powers. For example, overwhelmingly institution decisions are not appealable. Yet, there have been numerous lawyers with shockingly little experience appointed to the position of patent judge, and vested with the power to make decisions that cannot be reviewed by any Article III federal court.
As we discuss in Conquering Innovation Fatigue, the profit motive can be important for inventors but is often not the real incentive behind the quest to invent. Steps that eliminate the opportunity to profit from invention, though, can be serious barriers to a nation’s innovation potential. The profit motive can be important for prospective innovators. However, a focus on profits can be utterly destructive to innovation within a corporation, where the incentives to those who lead other would-be innovators can create new barriers that kill the innovation future of the company. Ironically, what can be a helpful incentive for innovation to an individual can easily become a disincentive once distorted by the internal workings of a corporation. This is illustrated in recent analysis from Clayton Christensen. See an overview in the article “Clayton Christensen: How Pursuit of Profits Kills Innovation and the U.S. Economy” at Forbes.com. Christensen argues that ratio-based metrics for profitability distort corporate thinking and reward behavior that ultimately destroys the future of the corporation by creating short-term benefits in apparent profitability. We illustrate a related problem in the book with the Apple Tree Analogy, in which metrics for short-term profitability for an apple harvester get a dramatic boost when the apple trees are toppled, making it much faster to harvest the fruit. The future, though, becomes barren.
Corporations need to carefully consider the metrics they use for profitability, as Christensen teaches, and unlearn some of the sacred concepts they were given in business schools. They should also go one step further an consider the impact of their metrics on not just the long-term growth of the company as a whole, but also the individual innovator and the innovation culture within the company. Listening to the voice of the innovator inside the corporation should be an important exercise for its top leaders.
Maybe China is just too far from the smug innovation circles of the West. Maybe language and cultural barriers make the events unfolding in China too inaccessible to Western media. Maybe decades of concern about IP theft from Chinese companies has closed the eyes of the West to present realities. Whatever the reason, the West today seems generally blind to the innovation powerhouse that China is becoming. Witness, for example, the highly publicized list from Reuters-Thompson of the top 100 global innovators, based on “patent activity.” With China having become one of the world’s true hotbeds of patent activity, not to mention economic impact with innovation in many fields, one might expect Chinese institutions to be well represented on the list. Incredibly, the list has ZERO Chinese entities on. None from Mainland China, none from Taiwan, and none from anywhere else in Asia except for a heavy dose of Japanese companies (27) and 4 from South Korea. Tiny Switzerland makes the list 3 times, and its minute neighbor, Liechtenstein, makes the list with Hilti Corporation. But zero from China and Taiwan? The list is related to “patent activity,” but its compilers wisely recognize that patent volume alone is a poor metric for innovation. Instead, they have created other metrics based on patent data:
The Thomson Reuters 2011 Top 100 Global Innovators are companies that invent on a significant scale; are working on developments which are acknowledged as innovative by patent offices across the world, and by their peers; and, whose inventions are so important that they seek global protection for them.
Sounds fair. So sure, the manufacturing and supply chain innovation that has been a big part of China’s economic rise are not expected to make a showing on this list. That kind of innovation doesn’t show up in terms of granted US and European patents. And the tendency for many Chinese companies to mostly file patents in China doesn’t help them with the methodology Thompson Reuters has, which looks for measures of international impact and international patents. But did they miss all the international activity of some Chinese companies? For example, a couple of days before this list of innovators came out, I posted this on LinkedIn and Twitter (@jefflindsay):
Two Chinese companies, ZTE and Huawei Tech., are among the top 5 international (PCT) patent applicants. Lots of IP here! https://is.gd/t2tUb4
OK, so Thompson Reuters doesn’t follow me, but these companies should have shown up strongly in their patent searches. These are innovative companies with products marketed internationally, having strong economic impact, and loads of patents. Being in the top 5 for international filings wasn’t good enough to even place in the top 100 for Thompson Reuters. Huh? OK, it turns out the ZTE’s surge in patent filings is recent and their numbers prior to 2010 were probably too low to make the cut for this study–that’s fair. But Huawei had 445 US patents from 2005 to 2010, a number in greater than some other companies on the list. For 2011, by the way, Huawei isn’t just in the top 5 so far–they are Number One, the world’s leader in international patent filings (see the Nov. 2011 article in the Vancouver Sun). Think they’ll be on the next list of leaders in patent activity Thompson Reuters publishes? Perhaps, who knows?
How about the Liechtenstein firm that’s in the Top 100, Hilti AG. Heard of them? They have good products for the construction and building maintenance industries such as hammer drills and other tools. They have 20,000 employees, including 2,500 in the US, and market products and file patents internationally. For the 2005-2010 time frame of the Thompson Reuters study, Hilti had 327 patents. Not bad. Well below Huawei’s score, but still respectable.
Now let’s consider a little company that was not on the list: Foxconn. Heard of them? They have over 1 million employees and are the world’s largest producer of electronic components, including circuit boards. They are the ones who actually make Apple’s products such as the iPhone and iPad. This Taiwanese/Chinese firm (China considers Taiwan to be part of China, and much of Foxconn’s work is in China) is arguably the real powerhouse behind the success of Apple and several other companies on the Thompson Reuters list. Foxconn builds Apple’s products, and not just as a mindless executor, but as an innovative partner.
Ah, but what about real technological innovation expressed in patents? Surely Foxconn is just about cheap labor and low cost manufacturing, right? A quick search of Foxconn patents granted in the US from 2005 to 2010 shows they have over 700 patents. Some are design patents, but the vast majority are technological. Foxconn apparently is conducting serious R&D and spending millions on patents to find new ways to make leading edge high-tech products better, safer, faster, and cooler (both in terms of heat management and the “wow” factor). I have the privilege of interacting with some Foxconn people and from what I’ve seen and heard I can say that they have a world-class IP program to support innovation, and I feel that they are way ahead of many Western companies in these areas. Foxconn innovation and Foxconn IP may be the real key to Apple’s success. Foxconn innovation is abundantly expressed in patents, not just trade secrets and know how, with an estate twice as big as Hilti’s over 2005-2010 and an economic impact on the global market far in excess of Hilti. But Foxconn doesn’t make the list. How do none of these Chinese companies break into Thompson Reuters’ Top 100? Did they miss the 2010 story, “China Poised to Become Global Innovation Leader,” based on patent activity? That must be from another source they don’t follow.
Nov. 24, 2011 Update: My search on Foxconn patents needs to be updated. Yes, Foxconn has an impressive 700+ US patents for the 2005-2010 period, more than some companies in the TR list. But my search was deficient, failing to consider that many Foxconn patents might have been filed under the real name of the company that owns Foxconn, Hon Hai Precision Industry Co., Ltd. So I expanded my search term to be “Hon Hai Precision” or Foxconn. Now, instead of 700 patents, we’re looking at a massive estate of 5,872 US patents (perhaps a couple dozen more when typographical errors are considered). This estate now dwarfs MOST of the companies on the list such as Brother Industries (2873, searching for Brother Ind* or Brother Kogyo), BASF (2771, searching for BASF or Bayerische Akt*) Goodyear (1152), ABB (948), Airbus (926), Avaya (<600), Arkema (205), Cheil (116), etc. Oh, and what about innovation giant Apple Compute? A search for simply “Apple” (which might include some smaller companies unrelated to Apple Computer) returns 1809 issued US patents from 2005 to 2010, less than 1/3 of the US patent activity of the invisible innovator that makes Apple what it is.
Let’s return to Huawei for a moment. There should be little doubt about the innovation prowess of Huawei, even though they tend to be far more secretive and do much less P.R. than Apple. But this telecom company is big (with over 100,000 employees, they connect 1/3 of the world’s cell phones) and highly innovative. Read, for example, BBC’s story, “Innovation in China: Huawei – the secretive tech giant.” Maybe the Thompson Reuters methodology docks them for being on the young side. Over half of their 445 US patents from 2005 to 2010 came in 2010–but they still clearly outpace Hilti and others over the 5-year span of the study. So what gives? I suspect that the youthfulness of the estate means there has been less time for others to cite Huawei patents, and that may be part of the problem since patent citations are part of the methodology. But to miss Huawei completely?
Thompson Reuters will surely argue that their methodology was developed and implemented fairly, even blindly (a fair term), but someone should have immediately seen that something was wrong if top international filers and innovators like Huawei, and Foxconn didn’t make the list. But when it comes to innovation, innovation in China tends to be largely invisible to Americans, who are stuck in the old paradigm of US being the innovation leader and China just being a copier. That kind of blindness will catch the West by surprise in the very near future when US companies find themselves facing numerous patent barriers from the Chinese companies that will own much of the most valuable IP. China is creating and will create much of the most important global innovation for the future. Innovation needs to ramp up in the West in order to not be left completely behind.
Dec. 8 Update: Chinese computer giant Lenovo, the world’s 2nd largest producers of personal computers, may also deserve to be on the list.
- IAM Blog: “The US, Japan and France dominate new listing of the world’s most innovative companies“
- Nielsen.com: “Nielsen China Forum: Dispelling the Myths of Innovation in China“
- China Law Blog: “Innovation In China. It’s Happening, But Not How You Think“
- Tech Crunch: “Thomson Reuters Names â€˜Top 100 Global Innovatorsâ€™“
The Summer 2010 issue of American Educator (a publication of the American Federation of Teachers) ably illustrates one of the lessons we teach in Conquering Innovation Fatigue: metrics to drive performance can have unintended consequences that may actually hurt rather than help. Indeed, unintended consequences are a major theme of our book, as we explore the problems arising from metrics, corporate and government policies, corporate innovation initiatives, laws, taxation policies, and other factors, all of which can contribute to innovation fatigue.
In terms of education and the danger of improper metrics, Linda Perlstein’s article, “Unintended Consequences; High Stakes Can Result in Low Standards,” examines a highly celebrated school in Annapolis, Maryland that received media attention and praise for seemingly miraculous success in education. The new principal arrived in 2000 to find Tyler Heights Elementary School in a dismal state with only 17% of its students getting satisfactory scores on the state test. She began redirecting efforts in the school to address this problem. Eventually her laser-focus efforts paid off, delivering the stunning success of 90% of third-graders performing well on the Maryland State Assessment, when only 35% of third-graders did so two years before. Several newspapers recognized the amazing turn-around and people at the school celebrated the success. But was it real success?
To achieve good performance on the Maryland State Assessment, education for the children was largely focused on how to do well on the test. Students learned how to write BCR’s (“Brief Constructed Response”) to deal with expected questions about poems and plays, and practiced writing these short answers for many hours, without actually studying poems or plays. “What gets tested is what gets taught,” the principal told the teachers, even if that meant leaving behind the material that was supposed to be taught according to state standards. Bins of equipment for studying science were largely unused.
Tyler Heightsâ€™ third-graders got only the most cursory introduction to economics and Native Americans, and much of the curriculum was skipped altogether. The students were geographically ignorant. . . . The third-graders had heard Africa mentioned a lot but were not sure if it was a city, country, or state. (They never suggested â€œcontinent.â€) At the end of the year, the children in Johnsonâ€™s class were asked to name all the states they could. Cyrus knew the most: three. He couldnâ€™t name any countries, though, and when asked about cities, he thrust his finger in the air triumphantly. â€œHoward County!â€
The state standards required a broad curriculum, but the metrics for assessing that were based on one particular test and all the incentives were for helping students pass that test. In spite of the praise for the miracle at Tyler Heights, had the children really been helped?
The problem with unintended consequences from metrics such as tests is hardly unique to Tyler Heights. Daniel Koretz, also writing in the same issue of American Educator (see page 3 of the PDF file on unintended consequences), explains that in education and other fields, score inflation is a common and well known but widely overlooked problem. In the social sciences, a phenomenon that leads to score inflation is known as Campbell’s Law. While widely applied to education, it was developed while looking at business. Donald Campbell, a prominent social scientist, examined the role of corporate incentives on the performance of employees. His research led to this general formulation: â€œThe more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.â€ (Donald T. Campbell, “Assessing the Impact of Planned Social Change,” in Social Research and Public Policies: The Dartmouth/OECD Conference, ed. Gene M. Lyons, Hanover, NH: Public Affairs Center, Dartmouth College, 1975, p. 35. See also Can New York Clean Up the Testing Mess? by Sol Stern.)
Campbell’s Law is at work when schools game tests to get better scores, at the expense of education. It is at work when cardiologists choose not to operate on patients who might need surgery rather than risk hurting their own published statistics on mortality rates among their patients (Koretz refers to a 2005 story from the New York Times reporting the shocking results of a survey of cardiologists). It is at work when a company tries to boost innovation with metrics or incentives that result in game playing, while leaving the real problems from culture, systems, and vision unaddressed.
In our experience, metrics and incentives can play a valuable role in driving innovation, but only when the corporation has a culture that genuinely encourages innovation, when there is a shared vision of innovation and success, and when sound systems are in place to advance innovation. Without those, you can not only waste a lot of resources in attempting to drive innovation with metrics and incentives, you can actually make a weak culture become pathological and lethal, sometimes exacerbating fatigue factors like the Not Invented Here syndrome, theft of credit for innovation, and breaking the will to share. Adding incentives linked to metrics without the right culture and systems can be sort of like throwing raw meat into a school of sharks or piranhas. You can generate a lot of activity, a lot of exciting thrashing and splashing, but in the end there will just be a lot of blood in the water and fewer thinkers and producers in your school.
As always, innovation success requires that you carefully monitor for harmful unintended consequences from the policies, programs, and incentives you have in place. Innovation metrics, incentives of all kinds, and employee performance evaluation systems and other tools associated with metrics can backfire. Unless you are tuned to the voice of the innovator and understand the impact of unintended consequences, you can be like the company we treat in Chapter 8 of our book that felt like it was a rock star of innovation while they were actually squelching it. Don’t let the unintended consequences of well-intended policies and metrics crush your innovation success.
One of the lessons of Conquering Innovation Fatigue is that the choice of metrics business leaders use to track and drive innovation can contribute to innovation fatigue when the metrics drive bad decisions and poor behavior. A recent example of how metrics can actually achieve the opposite of the intended results comes from a Wisconsin grocery chain, where a friend employed there explained the unintended consequences of management’s good intentions. Management is now pushing for higher levels of IPM, items per minute, as a metric for the performance of cashiers. This is a measure of how many items per minute the cashier processes, and sounds like a valuable metric for productivity. Faster checkout means happier customers and shorter lines–of course we want IPM to be high.
However, as with all metrics, the details of how IPM is calculated come into play and may bring unintended consequences. For IPM, the clock doesn’t tick when a lane is closed or, more specifically, when the cashier’s terminal is in “secure” mode. Shut down the terminal to the “terminal secure” state and the clock stops, something that some cashiers use to their advantage while checking out a customer. A new manager at one store is pushing for IPM scores of at least 30 for all cashiers, but as one cashier explained, the only way that you can achieve that high of a score is to routinely go to “terminal secure.” If the cashier has to help with the bagging or do other tasks that reduce IPM, they can secure the terminal and then reactivate it before they continue scanning goods. That gives a higher IPM score, but the back and forth of securing and reactivating the terminals actually SLOWS DOWN the real work because it involves extra steps that eat up valuable time. By focusing on IPM as a proxy for productivity, productivity can actually decline.
A further consequence of securing a terminal is that the customer may need to swipe his or her credit card a second time. The card readers in each checkout lane allow customers to swipe their credit card during the scanning of goods, but when the cashier switches to terminal secure mode, the swiped credit card information is discarded and the customer will have the annoyance of having to swipe a second time. By focusing on IPM as a proxy for customer satisfaction, the annoyances to the customer and the time to check out actually increase.
Unintended consequences of metrics can easily follow similar patterns when it comes to innovation, intellectual assets, and new product development. Leaders need to step back and observe the impact of their metrics on those in the ranks and on the actual performance of the company. A carefully selected basket of metrics with frequent reality checks are needed to avoid hindering real productivity and innovation with your good intentions.
Innovation has often been killed by poor metrics, part of the innovation fatigue factor of flaws in judgment and vision. Poor metrics can include financial analysis that emphasizes short-term gains but overlooks the future. One way of illustrating the problem is with the “Apple Tree Analogy.” Imagine a fruit production company that is concerned with the rising costs of apple production and the poor efficiency of manpower at harvest time. A hot-shot expert steps in to shake things up and turn things around. Suddenly, executives in their distant offices see reports showing dramatic gains in the efficiency of harvesting (perhaps a measure of the time required to pick the fruit from a tree) and a sharp drop in harvesting costs such as expenses for ladders and labor. Wonderful news!
But what is the cause of the gains? If management is remote and not in tune with what is happening on the ground, they might rely on the metrics or key performance indicators to conclude that the business is healthier than ever and issue huge bonuses to the geniuses who transformed the apple business. In reality, the short-term gains might have come at a surprisingly high cost, namely, the future of the business, as shown below.
Sometimes dramatic cost-cutting efforts can lead to great short-term metrics and make a business seem healthy, when in fact, it is doomed by the very actions that gave it the short-term boost. Cutting down every apple tree in the orchard will greatly simplifying harvesting. No need for ladders at all! But the crops of the future have been wiped out.
Innovators in your company are the source of your future crops. Cutting innovation staff will always look attractive for the short-term, but can your future afford it?
Make sure you are using wise metrics in making decisions about innovation, research, and new product development. It’s a topic we address in Conquering Innovation Fatigue, and one of the important reasons why you and your team need to be familiar with this book.