NON-CONFIDENTIALNo. 2012-1252
United States Court Of AppealsFor The Federal Circuit
MOTIVA, LLC,Appellant,
v.
INTERNATIONAL TRADE COMMISSION,Appellee,
and
NINTENDO CO., LTD. AND NINTENDO OF AMERICA INC.,Intervenors.
_________________
ON APPEAL FROM THE UNITED STATES INTERNATIONAL TRADE COMMISSION
IN INVESTIGATION NO. 337-TA-743_________________
ANSWERING BRIEF OF INTERVENORS NINTENDO CO., LTD.AND NINTENDO OF AMERICA INC.
_________________
Joseph S. PrestaRobert W. FarisNIXON & VANDERHYE P.C.901 North Glebe Road11th FloorArlington, VA 22203(703) 816-4000
E. Joshua RosenkranzPeter A. BicksAlex V. ChachkesORRICK, HERRINGTON & SUTCLIFFE LLP51 West 52nd StreetNew York, NY 10019(212) 506-5000
Mark S. DaviesKatherine M. KoppORRICK, HERRINGTON & SUTCLIFFE LLP1152 15th Street, N.W.Washington, D.C. 20005(202) 339-8400
Case: 12-1252 Document: 46 Page: 1 Filed: 08/29/2012
i
CERTIFICATE OF INTEREST
Counsel for Intervenors Nintendo Co., Ltd. and Nintendo of
America Inc. certify the following:
1. The full name of the parties represented by us in this case
are:
Nintendo Co., Ltd. and Nintendo of America Inc.
2. The name of the real parties in interest (if the party named
in the caption is not the real party in interest) represented by us are:
Nintendo Co., Ltd. and Nintendo of America Inc.
3. The parent companies, subsidiaries (except wholly owned
subsidiaries), and affiliates that have issued shares to the public of the
parties represented by us are:
Nintendo Co., Ltd., whose stock is publicly traded in Japan, owns
100% of Nintendo of America Inc.
4. The names of all law firms and the partners or associates
that appeared for the parties now represented by us in the agency or
are expected to appear in this Court, are:
Case: 12-1252 Document: 46 Page: 2 Filed: 08/29/2012
ii
Peter A. BicksE. Joshua RosenkranzMark S. DaviesAlex V. ChachkesElyse D. EchtmanSarah E. WalcavichRichard A. RinkemaJordan L. CoyleNicholas H. LamLauren B. MuldoonCyrus P.W. RieckKatherine M. KoppSteven E. Adkins (no longer withfirm)ORRICK, HERRINGTON & SUTCLIFFE
LLP1152 15th Street, NWWashington, D.C. 20005(202) 339-8400
Joseph S. PrestaRobert W. FarisNIXON & VANDERHYE P.C.901 North Glebe Road11th FloorArlington, VA 22203(703) 816-4000
5. No other appeal involving this civil action was previously
before this or any other appellate court. There are no pending cases
known to counsel that would directly affect or be directly affected by
this Court’s decision in the pending appeal.
Date: August 29, 2012 /s/ Mark S. Davies
Mark S. Davies
Attorney for Nintendo Co., Ltd. andNintendo of America Inc./Intervenors
Case: 12-1252 Document: 46 Page: 3 Filed: 08/29/2012
TABLE OF CONTENTS
Page
iii
CERTIFICATE OF INTEREST ................................................................ i
TABLE OF AUTHORITIES.................................................................... vi
STATEMENT OF RELATED CASES .................................................... ix
INTRODUCTION..................................................................................... 1
JURISDICTIONAL STATEMENT.......................................................... 3
STATEMENT OF THE ISSUES.............................................................. 3
STATEMENT OF THE CASE ................................................................. 3
STATEMENT OF THE FACTS ............................................................... 5
A. Motiva’s Research And Marketing Efforts: The StartAnd Finish.............................................................................. 5
B. Motiva’s Patents..................................................................... 9
C. Motiva’s Patent Litigation ................................................... 16
1. Motiva Files An ITC Action Against Nintendo .......... 16
2. The ALJ Grants Nintendo’s Motion For SummaryDetermination ............................................................. 18
3. The ITC Reverses And Remands With Questions ..... 19
4. The ALJ Holds A Hearing, Answers The ITC’sQuestions, And Finds No Violation ............................ 20
5. The ITC Affirms .......................................................... 30
SUMMARY OF ARGUMENT ................................................................ 30
STANDARD OF REVIEW...................................................................... 33
ARGUMENT .......................................................................................... 34
Case: 12-1252 Document: 46 Page: 4 Filed: 08/29/2012
TABLE OF CONTENTS(continued)
Page
iv
I. SUBSTANTIAL EVIDENCE SUPPORTS THE ALJ’SFINDING THAT MOTIVA WAS NOT IN THE PROCESSOF ESTABLISHING ANY DOMESTIC INDUSTRY .................. 34
A. Motiva’s Stale Licensing Efforts Do Not DemonstrateAny Tangible Steps Toward Establishing A DomesticIndustry................................................................................ 35
B. Motiva’s Lawsuit Against Nintendo Is Not A TangibleStep Toward Establishing A Domestic Industry................. 41
C. The Wii System’s Success Does Not Suggest MotivaHas Any Chance Of Establishing A Domestic IndustryIn The Future....................................................................... 46
II. SUBSTANTIAL EVIDENCE SUPPORTS THE ALJ’SFINDING THAT THE WII SYSTEM DOES NOTINFRINGE MOTIVA’S PATENTS ............................................... 51
A. Substantial Evidence Shows That The Wii SystemDoes Not “Track” Movement Of A User .............................. 51
B. Substantial Evidence Shows That The Wii SystemDoes Not Determine User “Movement” ............................... 54
1. Three Credible Witnesses Demonstrated ThatThe Wii Sensors Cannot And Do Not DeterminePosition Or Orientation .............................................. 55
2. Motiva’s Hodgepodge Of Movement And PositionInformation Arguments Are All Meritless ................. 59
CONCLUSION ....................................................................................... 75
Case: 12-1252 Document: 46 Page: 5 Filed: 08/29/2012
TABLE OF CONTENTS(continued)
v
Material has been deleted from pages 1, 2, 6, 22, 26, 29, 43, 46-47, 51-
53, 55-64, 68-70, 72 and 73 of the Non-Confidential Answering Brief of
Intervenors Nintendo Co. Ltd. and Nintendo of America Inc. This
material is deemed confidential business information pursuant to 19
U.S.C. § 1337(n) and 19 C.F.R. § 210.5, and pursuant to the Protective
Order entered November 2, 2010. The material omitted from these
pages contains confidential deposition and hearing testimony,
confidential business information, confidential patent application
information, and confidential licensing information.
Case: 12-1252 Document: 46 Page: 6 Filed: 08/29/2012
vi
TABLE OF AUTHORITIES
Page(s)FEDERAL CASES
Am. Silicon Techs. v. United States,261 F.3d 1371 (Fed. Cir. 2001) ...........................................................50
Bally/Midway Mfg. Co. v. ITC,714 F.2d 1117 (Fed. Cir. 1983) ..................................................... 38, 48
Beneficial Innovations, Inc. v. Blockdot, Inc.,Nos. 2:07-CV-263-TJW-CE, 2:07-CV-555-TJW-CE,2010 U.S. Dist. LEXIS 54151 (E.D. Tex. June 3, 2010).....................74
Erbe Elektromedizin GmbH v. ITC,566 F.3d 1028 (Fed. Cir. 2009) ...........................................................33
Finnigan Corp. v. ITC,180 F.3d 1354 (Fed. Cir. 1999) ...........................................................60
In re Katz Interactive Call Processing Patent Litig.,07-ml-01816-BRGK, 2008 WL 4952454(C.D. Cal. Feb. 21, 2008)....................................................................74
In re Nintendo Co., Ltd.,589 F.3d 1194 (Fed. Cir. 2009) ...........................................................16
InterDigital Commc’ns, LLC v. ITC,No. 2010-1093, 2012 U.S. App. LEXIS 15893(Fed. Cir. Aug. 1, 2012)................................................................. 39, 41
John Mezzalingua Assocs. v. ITC,660 F.3d 1322 (Fed. Cir. 2011) .......................................... 32, 33,34, 40
Ninestar Tech. Co. v. ITC,667 F.3d 1373 (Fed. Cir. 2012) ...........................................................33
Nippon Steel Corp. v. United States,458 F.3d 1345 (Fed. Cir. 2006) ...........................................................50
Case: 12-1252 Document: 46 Page: 7 Filed: 08/29/2012
vii
Northrop Grumman Corp. v. Intel Corp.,325 F.3d 1346 (Fed. Cir. 2003) ...........................................................66
Retractable Technologies v. Becton, Dickinson & Co.,653 F.3d 1296 (Fed. Cir. 2011) ............................................... 65, 66, 67
SRAM Corp. v. AD-II Eng’g, Inc.,465 F.3d 1351 (Fed. Cir. 2006) ...........................................................74
St. Clair Intellectual Prop. Consultants, Inc. v. Canon Inc.,412 F. App’x 270 (Fed. Cir. 2011) .......................................................74
Tessera, Inc. v. ITC,646 F.3d 1357 (Fed. Cir. 2011) ................................................34, 71, 72
TianRui Group Co. v. ITC,661 F.3d 1322 (Fed. Cir. 2011) ........................................................... 33
Vita-Mix Corp. v. Basic Holding,581 F.3d 1317 (Fed. Cir. 2009) ..................................................... 70, 71
ADMINISTRATIVE CASES
Certain Coaxial Cable Connectors and Components Thereof andProds. Containing Same,Inv. No. 337-TA-650, USITC Pub. No. 4283 (Nov. 11, 2010).............41
Certain Digital Satellite Sys. Receivers and Components Thereof,Inv. No. 337-TA-392, USITC Pub. No. 3418 (Apr. 2011) ............. 45, 46
Certain Multimedia Display and Navigation Devices and Sys.,Components Thereof, and Prods. Containing the Same,Inv. No. 337-TA-694, USITC Pub. No. 4292 (Nov. 2011)............. 44, 45
Certain Stringed Musical Instruments & Components Thereof,Inv. No. 337-TA-586, USITC Pub. No. 4120 (Dec. 2009) ...................34
In re Certain Rotary Wheel Printing Sys.,Inv. No. 337-TA-185, USITC Pub. No. 1857 (May 1986)...................38
Case: 12-1252 Document: 46 Page: 8 Filed: 08/29/2012
viii
FEDERALRULES & STATUTES
Fed. Cir. R. 30(j) ........................................................................................1
19 U.S.C. § 1337 .............................................................................. passim
28 U.S.C. § 1295(a)(6)................................................................................3
LEGISLATIVE HISTORIES
S. Rep. No. 100-71 (1987) .................................................................. 20, 34
H.R. Rep. No. 100-40 (1987).................................................. 20, 32, 34, 40
REGULATIONS
75 Fed. Reg. 68,379 (Nov. 1, 2010)............................................................4
Case: 12-1252 Document: 46 Page: 9 Filed: 08/29/2012
ix
STATEMENT OF RELATED CASES
No other appeal from the proceeding below was previously before
this Court or any other appellate court.
The same parties and one of the patents involved in this appeal
are involved in a district court action, Motiva, LLC v. Nintendo Co.,
Ltd., et al., No. 2:10-cv-00349 (W.D. Wash. filed Nov. 10, 2008). That
action was filed in the Eastern District of Texas, but, on Nintendo’s
petition to the Federal Circuit, this Court ordered the case transferred
to the Western District of Washington. In re Nintendo Co., Ltd., et al.,
589 F.3d 1194, 1201 (Fed. Cir. 2009). In June 2010, the district court
stayed the action pending completion of the International Trade
Commission investigation underlying this appeal and completion of the
U.S. Patent and Trademark Office’s reexamination of a patent.
Case: 12-1252 Document: 46 Page: 10 Filed: 08/29/2012
INTRODUCTION1
The individuals behind Motiva, LLC (“Motiva”) dreamed of riches
resulting from an advanced type of exercise equipment that wirelessly
and precisely tracked an individual’s movements. But the dreamers
gave up that fantasy years ago once reality intruded. Creating a
workable device required “vast amounts of money” (A7811), and
Motiva’s only funder walked away. The individuals moved on to other
interests, and their “prototype,” which was “not close to being
production ready,” lay fallow. A7820.
A7836. Today,
only the dream of “‘winnings’ from a lawsuit against Nintendo” of
America Inc. and Nintendo Co., Ltd. (“Nintendo”) (A7822) keeps Motiva
going. But, as the ALJ found both before (A7647) and after (A7837) an
evidentiary hearing, Motiva “failed to demonstrate that a domestic
industry ‘is in the process of being established,’” (id.), a necessary
1 In this brief, all “A” cites are to the parties’ Joint Appendix, and “Br.”cites refer to the appellant’s opening brief. “SVA” refers to theSupplemental Video Recording Media Appendix, which includesdemonstratives submitted to the ALJ (Respondent’s DemonstrativeExhibit Nos. RDX-154, -156, and -158) and filed with this Courtpursuant to Fed. Cir. R. 30(j). For the Court’s convenience, the relevantpatents are reproduced at the back of this brief. The page numbersfrom the Joint Appendix are retained.
Case: 12-1252 Document: 46 Page: 11 Filed: 08/29/2012
2
requirement for bringing a complaint before the International Trade
Commission (“ITC” or “Commission”), 19 U.S.C. § 1337(a)(2).
The ALJ also went on to conclude that Nintendo’s Wii System
does not infringe Motiva’s patents. That conclusion is correct, and thus,
to say the least, supported by substantial evidence. The Wii System
does not “track” “movement” of a user, key limitations of Motiva’s
patents. Crediting Nintendo’s witnesses, the ALJ found that the
“evidence adduced at the hearing demonstrates that the three key
devices in the Wii Remote—the three-axis gyroscope, the three-axis
accelerometer, and the DPD—do not, in fact, track the movement of the
user or provide information regarding the position or orientation of the
user.” A7773. The ALJ found Nintendo’s witnesses credible:
A7780.
The ITC’s order adopting the careful factual determinations of the
ALJ should be affirmed.
Case: 12-1252 Document: 46 Page: 12 Filed: 08/29/2012
3
JURISDICTIONAL STATEMENT
Motiva invoked the ITC’s authority under Section 337 of the Tariff
Act of 1930, as amended. A8746. See 19 U.S.C. § 1337(b)(1). The ITC’s
final determination was issued on January 5, 2012. A7876. Motiva
filed a petition for review with this Court on March 5, 2012. A30,562.
Motiva notes this Court’s jurisdiction to review ITC final
determinations under 28 U.S.C. § 1295(a)(6). Br. 1.
STATEMENT OF THE ISSUES
1. Whether the ITC correctly found that Motiva failed to
establish that “an industry in the United States, relating to the articles
protected by” Motiva’s two patents (U.S. Patent No. 7,492,268 and U.S.
Patent No. 7,292,151) was “in the process of being established”
(19 U.S.C. § 1337)?
2. Whether substantial evidence supports the ITC’s factual
finding that Nintendo’s Wii System does not track user movement nor
determine the position or orientation of the user and thus does not
infringe Motiva’s two patents?
Case: 12-1252 Document: 46 Page: 13 Filed: 08/29/2012
4
STATEMENT OF THE CASE
In September 2010, Motiva filed a proposed complaint at the ITC
naming as respondents Nintendo Co., Ltd. and Nintendo of America
Inc. A8746. The ITC issued a Notice of Investigation. See 75 Fed. Reg.
68,379 (Nov. 1, 2010). Administrative Law Judge Robert K. Rogers, Jr.
issued an Initial Determination granting Nintendo’s motion for
summary determination. A7631-49. The ITC vacated the summary
determination, posed a series of factual questions, and remanded for an
evidentiary hearing. A7858-69. The evidentiary hearing took place
from August 1 through August 5, 2011. A7662. The witnesses included
the two named inventors, six fact witnesses (including two Nintendo
employees), and six experts. A7663. Thereafter, the ALJ issued a
second Initial Determination, again finding no violation. A7855.
Motiva petitioned the ITC for review. A8537. The ITC rejected
the petition and ruled that “[t]he ALJ’s conclusion that Motiva has not
proven a violation of section 337 is correct” and “is the Commission’s
final determination.” A7876.2
2 Although the ITC determined to review portions of the InitialDetermination on its own initiative, see A7871, those issues are notraised in this appeal.
Case: 12-1252 Document: 46 Page: 14 Filed: 08/29/2012
5
STATEMENT OF THE FACTS
A. Motiva’s Research And Marketing Efforts: The StartAnd Finish
Kevin Ferguson and Donald Gronachan had a dream. But like so
many get-rich dreams, theirs never even came close to fruition. They
had exactly one investor who abandoned them, got no traction from any
identified licensee, and never created a product—or even a passable
prototype of a product.
The dream was to develop an advanced technology that would
precisely measure human movement for the fitness and rehabilitation
market. A7810. The product they had in mind was a system consisting
of a screen, a base station, and a handheld controller. A9519. They
thought the product might be used for general exercise, athletic
performance training, and physical therapy and research. A30,638.
The Ferguson-Gronachan collaboration began in October 2003.
They formed Motiva as an “informal partnership.” A7810. Mr.
Gronachan had a day job at a company called Biodex Medical Systems.
A7811. Mr. Ferguson devoted time to developing some prototypes—
largely compressed in a 16-month period between October 2003 and
April 2005. A7823. During that period he tried to build a “proof-of-
Case: 12-1252 Document: 46 Page: 15 Filed: 08/29/2012
6
concept” prototype and a “demonstration” prototype. A10,490; see also
A10,179-80, 20,717. The proof-of-concept prototype no longer exists.
A20,717. “The demonstration prototype cannot be described as
anything close to production-ready. . . . [T]he prototype lacks
industrial design, fine-tooling, functional engineering, and test phase
manufacturing.” A7819-20, 10,506-07. See also A7819 (describing the
prototype as having “exposed circuit boards, wiring, and sensors”). It
was the engineering equivalent of a stick figure, albeit an expensive
one: Motiva planned to sell the device for See, e.g., A20,708-
09.
Motiva’s development work was funded—albeit only briefly—by
the only investor the duo ever attracted. His name was David Smith.
Motiva claims that Mr. Smith contributed a sum total of to
engineering, research, and development. A7807-08, 20,523-24. Mr.
Ferguson accepted the majority of this funding— —as salary.
A7807, 7811. Motiva spent the rest on supplies for the prototypes. Id.
Within a year, Mr. Smith pulled the plug on the investment. He
told the duo in 2004 that he had lost interest in supporting the project.
A7811. He realized that Mr. Ferguson and Mr. Gronachan had no hope
Case: 12-1252 Document: 46 Page: 16 Filed: 08/29/2012
7
of bringing anything to market without “vast amounts of money”—far
“more money than he could afford to invest.” Id. By early 2005, Mr.
Smith abandoned the venture without even recovering his investment.
A7814. He did not bother retaining any ownership rights in the
inventors’ work. A7810, 20,532, 20,909, 30,403, 30,404-06.
With their only investor gone, Motiva ground to a halt. Mr.
Ferguson took a job at Liebert Corporation. Now that he had a day job,
Mr. Ferguson had virtually no time to devote to developing the product.
A9534, 10,179-80. While Mr. Ferguson continued for a couple of years
to dabble in the project when time permitted, by 2007 he had
completely abandoned the pretense of working on the invention. A9539.
In the entire four-year period of their collaboration—from 2003 to
2007—Mr. Ferguson and Mr. Gronachan made exactly three pitches in
the hopes of attracting a licensee.
The first pitch, in January 2005, was to James Reiss. Mr. Reiss
was Mr. Gronachan’s boss at Biodex. A7811. The meeting took place in
the basement of Mr. Ferguson’s home. A10,447. The pitch went
nowhere because the technology was not compatible with Biodex’s
products. A7812.
Case: 12-1252 Document: 46 Page: 17 Filed: 08/29/2012
8
Fifteen months elapsed before Motiva had an opportunity for a
second pitch. In March 2006, Mr. Gronachan had several conversations
about the Motiva technology with Gregory Highsmith. A7813. Mr.
Highsmith is one of Mr. Gronachan’s “oldest, long standing friends.”
A20,668. Mr. Highsmith was employed by a commercial fitness
equipment manufacturer known as Life Fitness. “As a member of his
company’s innovation committee, Mr. Highsmith was in a perfect
position to introduce the Motiva technology to Life Fitness, but he did
not do so.” A7814. Life Fitness just was not interested. Id. Mr.
Highsmith stated that he never would have even bothered talking about
the Motiva device but for his “personal relationship” with Mr.
Gronachan. A20,668. See also A7814.
A good nine months later, in January 2007, came the third, and
last, pitch. Mr. Ferguson and Mr. Gronachan met with Michael
Lannon, the CEO of Koko Fitness. A7816. The meeting was held on a
Saturday because, as Mr. Lannon explained, it “wasn’t important
enough to take time out of our actual workweek.” Id. (quoting A10,458);
see also A10,456. It was not even a real opportunity. Koko Fitness was
Case: 12-1252 Document: 46 Page: 18 Filed: 08/29/2012
9
“a small company” with “limited development resources.” A7816
(quoting A10,456).
After this third strike, having already abandoned their
development efforts, Mr. Ferguson and Mr. Gronachan abandoned all
further marketing efforts.
B. Motiva’s Patents
What Mr. Ferguson and Mr. Gronachan did manage to do was to
secure two patents. In July 2004, they filed a provisional patent
application. A7810. The application resulted in U.S. Patent No.
7,292,151, “Human Movement Measurement System,” A1-35, issued on
November 6, 2007, and U.S. Patent No. 7,492,268, A36-67, a
continuation of the ‘151 patent, issued on February 17, 2009.
Both patents relate “to a system and methods for setup and
measuring the position and orientation (pose) of transponders” for
the purposes of “exercise” and “rehabilitation.” A14 at 1:11-13
(emphasis added); A50 at 1:13-15 (emphasis added). Before setting out
the patent more fully, a few words are in order about the bolded terms.
The “position” of any object relative to another object at any
particular time can be described by “x, y and z” coordinates. The three
Case: 12-1252 Document: 46 Page: 19 Filed: 08/29/2012
10
coordinates describe an object’s location in space relative to an origin
point. Here is a graphical image of this coordinate system:
So, for example, using this Court as the point of origin, one can describe
the position of the flag atop the White House in terms of its x, y, and z
relative to the front door of 717 Madison Place. If west-east is the
x-axis, north-south the y-axis, and up-down the z-axis, the position of
the flag relative to the Court would be (in feet) x=+413, y=+734, and
z=+95.
As paradoxical as it might at first sound, an object can move
without changing its position. That is because a change in location is
not the only sort of movement. A pigeon flying from the White House
across Lafayette Square to the Court is obviously moving in the sense of
position. But if the pigeon sits atop the flagpole and spins around, or
Case: 12-1252 Document: 46 Page: 20 Filed: 08/29/2012
11
tilts forward or to the side, it is moving even though its position has not
changed.
An object’s rotation about an axis is referred to as “orientation.”
Orientation is measured as the amount of rotation along each of three
axes at any particular time. “Roll” is the degree of rotation around the
x-axis (e.g., a bird leaning left or right); “pitch” is the degree of rotation
around the y-axis (e.g., a bird looking up or down); and “yaw” is the
degree of rotation around the z-axis (e.g., a bird balancing on one leg
and spinning in circles). See, e.g., A10,306.
“Pose” refers to measurement of both “position” and “orientation.”
See A14 at 1:11-13 (“This invention relates to a system and methods for
setup and measuring the position and orientation (pose) of
transponders.”).
In sum, the pose of an object is described by using six
coordinates—three coordinates specific to position (x, y, z), and three
coordinates specific to orientation (roll, pitch, yaw).
Although not a disputed term, by way of background a
“transponder” is a wireless device that responds to a signal with a
signal of its own. The patent’s specification defines a transponder as a
Case: 12-1252 Document: 46 Page: 21 Filed: 08/29/2012
12
“wireless communication and monitoring device that receives a specific
signal and automatically responds with a specific reply.” A14 at 2:18-
20. A transponder is often “hand-held.” A15 at 4:34-35.
The ‘151 patent discloses an invention that trains the user to
manipulate “the pose of the transponders through a movement
trajectory” for the purposes of exercise and rehabilitation. A14 at
1:13-17 (emphasis added). Immediately below is an image (a composite
of patent Figures 4A, 4B, and 6A, see A10, 12) illustrating how the
system determines the transponder’s position and orientation:
Case: 12-1252 Document: 46 Page: 22 Filed: 08/29/2012
13
In the diagram above, the user (represented by the figure) holds a
transponder. A10. In response to a request from the computer, the
transponder sends a signal (indicated by the blue lines) to the receiver.
To determine the transponder’s position relative to the receiver, the
computer engages in a two-step process: (1) it measures how long it
takes the signal to reach the different points on the receiver; and (2) it
performs certain geometric and algebraic calculations to translate those
measurements into the distance on the x, y, and z axes. A17 at 8:9-11,
23 at 20:12-16, 27 at 27:55-60, 28 at 29:6-7. With slightly different
calculations, the computer can also determine the position of the
transponder relative to an “origin” point other than the receiver. A28 at
29:31-32.
To determine the orientation of the transponder, the computer
calculates the position of three separate points on the transponder. See
A28 at 29:39-45. The three positions define a unique plane, and that
plane reflects the transponder’s roll, pitch, and yaw. Id.
Case: 12-1252 Document: 46 Page: 23 Filed: 08/29/2012
As explained abo
measurements at any
a “movement trajector
the user moves the tra
stores the position and
By recording the trans
user’s movement.
Case: 12-1252 Document: 46 Page: 24 Filed: 08/29/2012
Transponder →
14
ve, position and orientation, or pose, are
particular moment in time. The patents refer to
y.” See, e.g., A14 at 1:14-15; A50 at 1:16-17. As
nsponder, the system repeatedly generates and
orientation measurements:
ponder’s poses over time, the system tracks the
15
The first claim in the patent reads as follows:
A system for tracking movement of a user, comprising:
a first communication device . . .;
a processing system . . . said processing system adapted todetermine movement information for said first communicationdevice and sending data signals to said first communication devicefor providing feedback or control data; and
wherein said first communication device receives and processessaid data signals from said processing system and wherein theoutput device provides sensory stimuli according to the receiveddata signals.
A31 at 35:39-54 (emphasis added). Claim 50 claims an “apparatus for
use in tracking movement of a user.” A32 at 38:47.
The ‘268 patent is a continuation of the ‘151 patent. Claim 1
claims a system for “tracking position of a user.” A66-67 at 34:59-35:8.
Claim 10 claims an “apparatus for use in conjunction with a remote
processing system for tracking position of a user.” A67 at 36:7-8.
Claim 15 is for a “system for tracking movement of a user.” A67 at
36:35.
In 2010, in two separate orders, the PTO ordered reexamination of
each patent. The examiner canceled some claims and confirmed
Case: 12-1252 Document: 46 Page: 25 Filed: 08/29/2012
16
others.3 Nintendo has appealed to the Board of Patent Appeals as to
both patents.
C. Motiva’s Patent Litigation
1. Motiva Files An ITC Action Against Nintendo
In November 2008, Motiva commenced a patent infringement
action against Nintendo in the United States District Court for the
Eastern District of Texas. This Court ordered the litigation transferred
to the Western District of Washington, where Nintendo of America is
located. In re Nintendo Co., Ltd., et al., 589 F.3d 1194, 1198
(Fed. Cir. 2009). In June 2010, the district court in Washington stayed
the action pending the outcome of the reexamination proceedings at the
Board of Patent Appeals. A few months later, Motiva sued Nintendo in
the ITC, alleging that Nintendo violated Section 337 by importing and
selling the Wii game system and related items. A8746.
Motiva’s allegations center on the Wii Remote, the primary
controller for the Wii System. Here is a picture of the Wii Remote Plus:
3 As to the ’151 patent, the examiner canceled claims 1-11, 17-20, 26, 35,36, 38-43, 45-56, 58-64, 66, 67, 69-72, 77, 78, 82, 83, and 85-91. Theexaminer confirmed claims 12-16, 21-25, 27-34, 37, 44, 57, 65, 68, 73-76,79-81, and 84. As to the ‘268 patent, the examiner canceled claims 1, 3,and 13, confirmed claim 4, and found claims 2, 5-12, 14, and 15patentable.
Case: 12-1252 Document: 46 Page: 26 Filed: 08/29/2012
17
As relevant here, the Wii Remote Plus has three key devices:
(1) a three-axis gyroscope
(2) a three-axis accelerometer
(3) a particular type of camera referred to as a Direct Pointing
Device (“DPD”).
The Wii System was released in the United States in November
2006, A8276, two full years after Mr. Smith stopped funding Motiva.
A7811. The Wii sells for approximately $149. A8278.
“Motiva argues that these three devices contained within the Wii
send movement and position information” to the Wii console. A7773.
The complaint asserted that the Wii System infringes three
independent claims and various dependent claims of Motiva’s patents.
Case: 12-1252 Document: 46 Page: 27 Filed: 08/29/2012
18
All the asserted ‘151 claims require “tracking movement of a user.”4
A8746. All but one of the asserted ‘268 claims require “tracking
position of a user.”5 Id.
The ITC opened an investigation.
2. The ALJ Grants Nintendo’s Motion For SummaryDetermination
In Order No. 12, the ALJ granted Nintendo’s “motion for summary
determination that the economic prong of the domestic industry
requirement is not satisfied.” A7631. The ALJ noted that a patent
complainant invoking the ITC’s authority must establish that an
industry “relating to the articles protected by the patent . . . exists or is
in the process of being established” in the United States, A7634
(quoting 19 U.S.C. § 1337(a)(2)).
After summarizing the research and marketing effort set forth by
Motiva, the ALJ ruled that “Motiva’s investments do not support a
4 The complaint, as corrected, asserted eight claims dependent onclaim 1 (claims 16, 27-32, 44) and four claims dependent on claim 50(57, 68, 81, and 84).
5 The complaint asserted claim 1 of the ‘268 patent and 7 claims(claims 2-6 and 8-9) dependent on claim 1, as well as claim 10 and 4claims (11-14) dependent on claim 10. The complaint also assertedclaim 15, which refers to “tracking movement of a user.”
Case: 12-1252 Document: 46 Page: 28 Filed: 08/29/2012
19
finding of either the existence of a domestic industry or that a domestic
industry was in the process of being established at the time of filing of
the complaint in this investigation.” A7637. The ALJ found that
“Motiva ceased investing in research and development, marketing, or
any step towards creating a domestic industry, in 2007.” A7648. The
ALJ reasoned that: (i) “Motiva does not directly dispute that it had
stopped investing in research and development and marketing for the
asserted patents in 2007”; (ii) “Motiva’s patent litigation activities are
not to be considered as part of the domestic industry analysis, and its
prosecution of ‘related applications’ is not relevant to the domestic
industry analysis”; and (iii) Motiva’s “assertion that once it wins its
patent infringement litigations” it will “reenter the market” is
“insufficient” to establish a domestic industry because to “find otherwise
would render the domestic industry requirement a nullity.” Id. The
ALJ granted summary determination to Nintendo. A7631.
3. The ITC Reverses And Remands With Questions
Motiva and the Office of Unfair Import Investigations petitioned
for review. The ITC reversed and remanded. The ITC explained that
the “legislative history indicates that an industry is ‘in the process of
Case: 12-1252 Document: 46 Page: 29 Filed: 08/29/2012
20
being established’” if the patent owner “can demonstrate that he is
taking the necessary tangible steps to establish such an industry in the
United States,” A7863-64 (citing S. Rep. No. 100-71, at 130 (1987)), and
there is a “significant likelihood that the industry requirement will be
satisfied in the future,” A7864 (citing H.R. Rep. No. 100-40, at 157
(1987)). The ITC ruled that the “ALJ erred in declining to consider
Motiva’s activities that occurred before the issuance of the asserted
patents.” A7863 (citing A7672).
The ITC also found that Motiva had raised a “genuine issue of
material fact as to whether its district court litigation activities between
2007 and the present are related to licensing and/or product
development.” A7865. The ITC directed the ALJ to address “to the
extent necessary” several questions “relevant” either “to whether a
domestic industry exists” or “is in the process of being established.”
A7867.
4. The ALJ Holds A Hearing, Answers The ITC’sQuestions, And Finds No Violation
The ALJ holds the hearing. At the hearing, the ALJ heard
from nine witnesses called by Motiva: the two inventors; the one short-
term investor; two of the three targets of pitches (Mr. Highsmith and
Case: 12-1252 Document: 46 Page: 30 Filed: 08/29/2012
21
Mr. Reiss); three experts; and a Nintendo engineer. A7663. Nintendo
called three experts, a Nintendo manager, and Mr. Barry French (the
CEO of Trazer Technologies and Mr. Ferguson’s former boss). Id.
At the hearing, Nintendo’s expert testified that “Motiva
abandoned any domestic industry that it might have had in process.”
A10,488. He based his conclusion “on the fact that Mr. Ferguson’s work
. . . waned significantly . . . in early 2005, and then stopped entirely in
January 2007” and that “Mr. Gronachan . . . did not put in any
significant effort at any time.” Id. In response to one of Motiva’s
experts, Mr. Bakewell prepared the following graph to illustrate the
point:
Case: 12-1252 Document: 46 Page: 31 Filed: 08/29/2012
22
The ALJ finds no ITC authority over Motiva’s suit. Based on
a careful review of the trial record, the ALJ concluded that “Motiva has
failed to prove that a domestic industry ‘exists’ pursuant to Section
337(a)(2).” A7832. The ALJ summarized the record:
(1) well before the release of the Wii, Motiva was facing alack of funding due to the departure of David Smith;
Case: 12-1252 Document: 46 Page: 32 Filed: 08/29/2012
23
(2) prior to the Wii being released, Motiva met withpotential partners, but they showed little to no interest ininvesting in the Motiva technology;
(3) the one company that Motiva met with after the releaseof the Wii—Koko Fitness—was not interested in investing inthe technology for reasons other than the fact that theWii was already on the market;
(4) Motiva has offered no evidence that any potentialinvestor, partner, or licensee was dissuaded in investing inthe Motiva technology due to the Wii;
(5) the Wii was not in the same market as the Motivaproduct, and the two products would not have competed;
(6) Motiva’s actions in litigation are not consistent with acompany whose main purpose is to remove Nintendo fromthe market so that it can enter the market; and
(7) Motiva has not demonstrated that litigation was its onlyrecourse to salvage its business in the face of Nintendo’srelease of the Wii.
A7830-31 (emphasis added).
The ALJ also provided the following answers to the ITC’s
questions relevant to the existence of a domestic industry (with all
emphasis added):
Commission Question ALJ Findings
1[a]. What was the level of interestfrom potential manufacturers,investors, and licensees in Motiva’stechnology prior to the release ofthe Wii?
“I find that there was littleinterest from potentialmanufacturers, investors, andlicensees in Motiva’s technologyprior to release of the Wii.” A7814.
Case: 12-1252 Document: 46 Page: 33 Filed: 08/29/2012
24
1[b]. Did Nintendo’s release of theWii cause this interest to decrease?
“I now find that Motiva has failedto demonstrate that there wasany decline in interest causedby the release of the Wii.” A7815.
1[c]. To what extent would theproduct(s) being developed byMotiva compete with Nintendo’sWii?
“I find that the product developedby Motiva would not competewith the Nintendo Wii.” A7817.
2. How close was Motiva’stechnology to beingcommercialized and/or productionready?
“I find that Motiva’s technologywas not close to beingincorporated in a commercial orproduction-ready product.” A7819.
3[a]. To what extent was Motiva’sshift in production-orientedactivities to litigation-orientedactivities a strategic businessdecision not caused by Nintendo’sactivities?
“I find no evidence to supportMotiva’s claim that the turn tolitigation was the result of beingrejected by potential investors dueto the presence of the Wii.” A7822.“Motiva’s litigation tacticsstrongly suggest that thepurpose behind the litigationwas to extract a monetaryaward either through damages ora financial settlement.” A7823.
3[b]. Could Motiva have continuedits commercialization effortswithout resorting to litigation?
“It was possible that Motiva couldhave continued itscommercialization efforts withoutresorting to litigation, but it wouldhave taken a new source ofmoney to do so.” A7823.
In addition to finding that no domestic industry existed, the ALJ
concluded that “Motiva has failed to demonstrate that a domestic
Case: 12-1252 Document: 46 Page: 34 Filed: 08/29/2012
25
industry ‘is in the process of being established.’” A7832-37. The ALJ
found that “[f]rom 2003 to 2007, Motiva was taking tangible steps to
establish an industry in the United States.” A7833. But the “building”
and “market[ing]” “activities ended in 2007.” A7833-34. “Therefore,”
the ALJ found, “after 2007, Motiva abandoned its efforts to establish an
industry in the United States.” A7834 (citing A10,488-90). “Mr.
Bakewell’s credible testimony supports this conclusion.” Id. The ALJ
also rejected Motiva’s argument that the “litigation against Nintendo is
evidence of Motiva taking the necessary tangible steps to establish an
industry.” Id.
Not only was Motiva not taking tangible steps to establish an
industry, the ALJ found that “Motiva has not demonstrated that there
is a significant likelihood that the industry requirement will be satisfied
in the future.” Id. “Motiva could not demonstrate that there was any
significant interest in its technology prior to the Wii’s existence, and I
find that there is no reason to believe that manufacturers of fitness and
rehabilitation equipment will suddenly become interested in the
technology because Nintendo has been excluded from the market.”
A7835. Moreover, the ALJ found that “Motiva has not shown that it
Case: 12-1252 Document: 46 Page: 35 Filed: 08/29/2012
26
has ever attracted interest from the video game industry.” Id. In
addition, the ALJ found that
A7836. The ALJ found
that “Motiva failed to demonstrate that a domestic industry ‘is in the
process of being established,’ pursuant to Section 337(a)(2).” A7837.
In so holding, the ALJ answered the ITC’s questions relevant to
any future industry:
3[c]. Was Motiva taking the“necessary tangible steps toestablish” a domestic industry?
“I find that after 2007, Motivaabandoned its efforts toestablish an industry in theUnited States.” A7834. “I do notfind that the litigation againstNintendo is evidence of Motivataking the necessary tangible stepsto establish an industry.” Id.
4[a]. Do the steps “taken [byMotiva] indicate a significantlikelihood that the industryrequirement will be satisfied in thefuture?”
4[b]. How likely is it that Motivawill have a domestic industry inthe future (1) if no relief is issuedagainst Nintendo or, alternatively,(2) if relief is granted againstNintendo?
“I conclude that Motiva has notdemonstrated that there is asignificant likelihood that theindustry requirement will besatisfied in the future, and thisdetermination is not dependent onthe Commission’s actions in thisinvestigation.” Id.
Case: 12-1252 Document: 46 Page: 36 Filed: 08/29/2012
27
The ALJ also finds no infringement. In addition to rejecting
Motiva’s complaint for failure to establish a condition for an ITC action,
the ALJ separately found that Nintendo’s Wii System does not infringe
the Motiva patents. A7854-55.
In so doing, the ALJ’s Initial Determination construed certain
claim terms. The phrase “tracking movement of a user” appears in the
preamble of independent claims 1 and 50 of the ‘151 patent. A31-32.
The ALJ ruled that the phrase means “tracking changes of position
and/or orientation of a user.” A7674.
The ALJ’s construction was different from Nintendo’s reading of
the patent. Nintendo argued that “tracking movement of a user”
required tracking the position and orientation of the user. Among
other things, Nintendo noted that the patent describes the invention as
“measuring the position and orientation (pose) of transponders.” A7673
(emphasis added). Nevertheless, the ALJ found that “tracking
movement of a user” can mean tracking either the position or
orientation of the transponder. A7672.
In addition to construing the claims to require “tracking changes
of position and/or orientation of a user,” the ALJ construed “position
Case: 12-1252 Document: 46 Page: 37 Filed: 08/29/2012
28
information.” The ALJ explained that the “parties’ proposed
constructions do not vary greatly,” and “[t]he primary dispute centers
on Nintendo’s inclusion of ‘3D space.’” A7678. The ALJ found that
“[t]he specification explains that the invention may be used ‘for the
purposes of functional movement assessment for exercise, and physical
medicine and rehabilitation,’” Id. (citing A14 at 1:16-17), and “[s]uch
tracking requires knowledge of the user’s location in 3D space,” id.
(citing A10,330-31). Although Motiva invoked certain statements by
the examiner made during the reexamination of the ‘151 patent, the
ALJ found them “not persuasive” in light of the “broader standard for
claim construction during reexamination” and because the
“reexamination of the ‘151 patent is not complete.” A7679.
After construing the key terms, the ALJ found that “Motiva has
failed to show by a preponderance of the evidence that the accused
products infringe claim 1 of the ‘151 patent.” A7782. At the hearing,
the ALJ viewed live demonstrations in which Nintendo expert
Dr. J. Edward Colgate and Keizo Ohta, Nintendo’s Manager of the
Technology Group in the Entertainment Analysis and Development
Division (A7663), played Wii games and explained how the Wii creates
Case: 12-1252 Document: 46 Page: 38 Filed: 08/29/2012
29
the appearance of tracking the user’s movement without knowing the
user’s position or orientation. A21,593-618, 30,447-56, 30,552-53,
30,576-79, 30,581-83; see generally, SVA at RDX-154, -156, and -158.
The ALJ found that the “evidence adduced at the hearing demonstrates
that the three key devices in the Wii Remote—the three-axis gyroscope,
the three-axis accelerometer, and the DPD—do not, in fact, track the
movement of the user or provide information regarding the position or
orientation of the user.” A7773. The ALJ concluded:
A7780. Instead,
the Wii
Id.
(emphasis added); see id. (“After careful review of the record evidence, I
find that Motiva has failed to meet its burden to demonstrate that the
Wii accused products track the movement of a user.”).
The ALJ applied the same analysis to find no infringement of the
‘268 patent.
Case: 12-1252 Document: 46 Page: 39 Filed: 08/29/2012
30
5. The ITC Affirms
Motiva appealed again to the ITC, but this time the ITC affirmed.
A7871. The Commission determined that “[t]he ALJ’s conclusion that
Motiva has not proven a violation of section 337 is correct and is the
Commission’s final determination. The investigation is terminated.”
A7876.
SUMMARY OF ARGUMENT
The Court should affirm the ITC’s decision to adopt the ALJ’s
careful determination. The ALJ gave two independent reasons for
rejecting Motiva’s complaint. Nothing in Motiva’s brief calls these
reasons into question.
I.A. Substantial evidence supports the ALJ’s finding that Motiva
was not “in the process” of “establishing” any domestic industry related
to these patents. In a detailed opinion issued after an extensive
hearing, the ALJ found that “after 2007, Motiva abandoned its efforts to
establish an industry in the United States.” A7834.
Motiva makes no effort to challenge the ALJ’s finding that its
licensing efforts have “ended.” Id. Motiva does not deny that its main
investor walked away, does not claim to have another funding source,
Case: 12-1252 Document: 46 Page: 40 Filed: 08/29/2012
31
and does not claim to have done anything since 2007 (except sue
Nintendo). Motiva has stopped trying to establish any industry, and its
completed actions are not evidence that it will do any research or
marketing in the future.
B. Against these and similar dispositive factual findings, Motiva
makes a variety of limited and erroneous points. For example, much of
Motiva’s brief turns on its contention (at 44) that “[b]ut for the presence
of the Wii in the market, Motiva was—and still is—extremely close to
realizing its product-driven licensing goals.” The ALJ specifically
rejected this fiction. A7812-13 (citing A20,633-34). The ALJ also ruled
that “Motiva could not demonstrate that there was any significant
interest in its technology prior to the Wii’s existence,” and “that there is
no reason to believe that manufacturers of fitness and rehabilitation
equipment will suddenly become interested in the technology because
Nintendo has been excluded from the market.” A7835.
This Court has explained that “Congress recognized that the
Commission is fundamentally a trade forum, not an intellectual
property forum, and that only those intellectual property owners who
are ‘actively engaged in steps leading to the exploitation of the
Case: 12-1252 Document: 46 Page: 41 Filed: 08/29/2012
32
intellectual property’ should have access to the Commission.” John
Mezzalingua Assocs. v. ITC, 660 F.3d 1322, 1327-28 (Fed. Cir. 2011)
(quoting H.R. Rep. No. 100-40, at 157 (1987)). Here, Motiva’s bald
assertion that it is a “licensing company” is not sufficient to allow it to
invoke the ITC’s jurisdiction. Motiva has nothing but its patents. If
Motiva can invoke the protections of the ITC, any and every patent
owner can do so. This Court does not permit such a lax definition of the
ITC’s role.
II. The Wii System does not infringe Motiva’s patents. Because
Motiva did not establish any basis for bringing its complaint to the ITC,
the ITC decision should be affirmed on that basis alone. Nevertheless,
the ALJ reached, and the ITC adopted, a second rationale for rejecting
Motiva’s complaint: The Wii System does not infringe Motiva’s patents.
A. Substantial evidence supports the Commission’s determination
that the Wii System does not “track” movement of a user. A7777-82.
The ALJ found that the Wii System does not take “complete
measurements” of user movement, does not measure user movement
with “reasonable accuracy,” and does not keep any “representation of
the movement history in memory.”
Case: 12-1252 Document: 46 Page: 42 Filed: 08/29/2012
33
Motiva argues (at 25-26) that the ALJ’s construction of “tracking”
“directly contradicts the Commission’s apparent construction of
‘tracking movement of a user.’” Br. 24-25 (citing A7674 at ¶¶ 3-4). This
is inexplicable. The cited paragraphs do not construe the term
“tracking.” In any event, the Wii System does not “track” movement.
B. Moreover, and independently, substantial evidence supports
the ITC’s finding that the Wii System does not determine user
movement. As three witnesses that the ALJ found persuasive
explained, the Wii sensors cannot and do not determine the position or
orientation of the transponder. Motiva’s hodgepodge of movement and
position information arguments are all meritless.
STANDARD OF REVIEW
“[F]indings of fact [by the ITC] are reviewed to ascertain whether
they were supported by substantial evidence on the record as a whole.”
Erbe Elektromedizin GmbH v. ITC, 566 F.3d 1028, 1033
(Fed. Cir. 2009). The Court routinely affirms ITC decisions that are
supported by substantial evidence. See, e.g., Ninestar Tech. Co. v. ITC,
667 F.3d 1373, 1379 (Fed. Cir. 2012); TianRui Group Co. v. ITC, 661
F.3d 1322, 1337 (Fed. Cir. 2011); Mezzalingua, 660 F.3d at 1330.
Case: 12-1252 Document: 46 Page: 43 Filed: 08/29/2012
34
Claim construction is an issue of law this court reviews de novo.
Tessera, Inc. v. ITC, 646 F.3d 1357, 1364 (Fed. Cir. 2011). Infringement
decisions are a question of fact reviewed for substantial evidence. Id.
ARGUMENT
I. SUBSTANTIAL EVIDENCE SUPPORTS THE ALJ’SFINDING THAT MOTIVA WAS NOT IN THE PROCESS OFESTABLISHING ANY DOMESTIC INDUSTRY
As the ALJ realized from the get-go, Motiva is not entitled to
invoke the ITC’s authority. Motiva could not prove that a domestic
industry is “in the process of being established,” 19 U.S.C. § 1337(a)(2),
without establishing: (1) “that [it] is taking the necessary tangible
steps to establish such an industry in the United States,” and (2) that
“there is a significant likelihood that the industry requirement will be
satisfied in the future.” Certain Stringed Musical Instruments &
Components Thereof, Inv. No. 337-TA-586, USITC Pub. No. 4120, at 13
(Dec. 2009) (Comm’n Op.) (quoting S. Rep. No. 100-71, at 130 (1987) &
H.R. Rep. No. 100-40, at 157 (1987)).
Motiva’s brief reads as if the ALJ made no findings. But in a
detailed opinion issued after an extensive hearing, the ALJ found that
“after 2007, Motiva abandoned its efforts to establish an industry in the
Case: 12-1252 Document: 46 Page: 44 Filed: 08/29/2012
35
United States.” A7834. The decision is supported by substantial—
indeed, overwhelming—evidence and should be affirmed.
A. Motiva’s Stale Licensing Efforts Do Not DemonstrateAny Tangible Steps Toward Establishing A DomesticIndustry
The ALJ’s two opinions here call to mind the Monty Python “dead
parrot” sketch. There, John Cleese returns to the pet shop to register a
complaint about his dead Norwegian Blue parrot that had been nailed
to its perch. The shop owner insists that the bird is “resting,” “pining
for the fjords,” or simply “stunned.” Cleese responds by banging “Polly
Parrot” on the counter and rattles off several metaphors for death: the
bird “is no more,” “has ceased to be,” “bereft of life, it rests in peace.”
The sketch is one of the most famous in the history of British television
comedy.
The propped-up parrot here is Motiva’s technology, and the ALJ is
playing the role of Cleese by repeatedly and emphatically declaring the
technology dead. As the ALJ found both before (A7648) and after
(A7835-36) an evidentiary hearing, Motiva’s technology was “bereft of
life” by the time Motiva filed its complaint in 2010. Motiva’s only
investor quit in 2004. The ALJ noted that the “Motiva technology has
Case: 12-1252 Document: 46 Page: 45 Filed: 08/29/2012
36
not been updated or improved since at least December 2007.” A7820
The ALJ found that Motiva’s licensing efforts “ended in 2007.” A7834-
35. “Thus, on October 1, 2010, the relevant date for determining a
domestic industry, it had been at least 3.5 years since the end of the
[sic] Motiva’s engineering, research and development, and
commercialization activities.” A7826. Motiva has no future products
and no potential customers. Instead, Motiva exists solely to seek a cut
in the hoped-for contingent lawsuit “‘winnings.’” A7822. The ALJ notes
that Motiva does not “directly” challenge this version of events. A7648.
Instead, Motiva emphasizes (at 40) the ALJ’s ruling that “Motiva
was taking tangible steps to establish an industry in the United States”
from 2003 to 2007.6 Motiva argues this finding is evidence that it was
seeking to establish a domestic industry when it filed this complaint in
2010.7 And these same steps form the basis of Motiva’s claim (at 48)
that the “public interest” supports ITC authority here.
6 On appeal, Motiva has abandoned its argument that ITC authoritycould be based on the “existence” of a relevant domestic industry.Compare A8023 with Br. at 10.
7 For purposes of this appeal, Nintendo does not dispute the ALJ’sfinding that Motiva’s activities before the end of 2007 were steps towardestablishing a licensing program. But the record is far from clear thatthe sporadic effort set forth by Motiva amounts to a “substantial”
Case: 12-1252 Document: 46 Page: 46 Filed: 08/29/2012
37
But Motiva cannot turn old news into future prospects. The ALJ
found that “all” of Motiva’s licensing efforts “ended in 2007.” A7834.
“Motiva lost its primary source of funding when David Smith ended his
investment in late 2004, and Motiva has never found a suitable investor
to replace him,” id., nor did it try, A10,489. “Since the end of 2007,
Motiva has focused on nothing except suing Nintendo.” A7834. Motiva
did not have any executed licensing agreements, term sheets, letters of
intent, confidentiality agreements, licensing negotiations, trade show
demonstrations, or any offers made or received to license Motiva’s
technology. See A8751 (“Motiva has not licensed the Asserted
Patents.”). See also A20,686, 20,687, 20,689, 20,690, 20,706, 20,722,
20,723. The ALJ found that “after 2007, Motiva abandoned its efforts to
establish an industry in the United States.” A7834; see also id. (“Mr.
Bakewell’s credible testimony supports this conclusion” (citing A10,488-
90)).
Motiva makes no effort to challenge the ALJ’s finding that its
licensing efforts have “ended.” A7834. Motiva does not deny that Mr.
Smith walked away, does not claim to have another funding source, and
investment in licensing. Cf. Br. 51 (asserting there is “no dispute”regarding pre-2008 activities).
Case: 12-1252 Document: 46 Page: 47 Filed: 08/29/2012
38
does not claim to have done anything since 2007 (except sue Nintendo,
discussed below in section I.C). Motiva has stopped trying to establish
any industry, and its completed actions are not evidence that it will do
any research or marketing in the future. Compare Bally/Midway Mfg.
Co. v. ITC, 714 F.2d 1117, 1122 (Fed. Cir. 1983) (“if there was an
existing . . . [domestic industry] when the complaint was filed, section
337(a) was satisfied.”) (emphasis added); see In re Certain Rotary Wheel
Printing Sys., Inv. No. 337-TA-185, USITC Pub. No. 1857, at 42 (May
1986) (holding that Bally/Midway “was attempting to take account of
the situation where an industry is destroyed in the course of a
Commission investigation”).
Although it is not even attempting to license its patents, Motiva
continues to refer to itself (at 39, 43) as a “product-driven licensing
company.” But Motiva has no products, no licenses, and no drive. It is
done. Motiva began marketing its technology as early as January 2005,
yet it has never come close to negotiating a license with a third party,
let alone having its technology incorporated into a commercial product.
A7820, 8751, 10,486-87, 20,686, 20,687, 20,689, 20,723. After Nintendo
observed that “Motiva has never offered to license its patents to
Case: 12-1252 Document: 46 Page: 48 Filed: 08/29/2012
39
Nintendo,” Motiva wrote: “Nintendo has it right. Motiva’s litigation is
not about licensing Nintendo.” A30,364 (emphasis added).
Also meritless is Motiva’s claim (at 39) to a focus on the “video
game environment.” Its patents disavow any real interest in the video
game business. See A21 at 15:38-43 (“It is important to note that only
primitive forms of video game challenges would be considered, to take
into account the user’s cognitive awareness and physical limitations,
and the economics of software development for photo realistic virtual
environments and animation.”) (emphasis added); A57 at 15:29-34
(same). Motiva has made no effort to compete with Nintendo in the
video game market. A20,708-09.
But even if Motiva is a licensing company, it has not shown the
required “substantial licensing activities related to the asserted patent”
necessary to satisfy the future domestic industry requirement.
InterDigital Commc’ns, LLC v. ITC, No. 2010-1093, 2012 U.S. App.
LEXIS 15893, at *32 (Fed. Cir. Aug. 1, 2012). Since 2007, the ALJ
found that Motiva engaged in no licensing activities related to any
patents. A7834. Motiva complains that the ALJ “faults Motiva at
length for its lack of consummated licensing deals.” Br. 46 (citing
Case: 12-1252 Document: 46 Page: 49 Filed: 08/29/2012
40
A7814) (emphasis added). That is rather like referring to a monk’s lack
of consummated marriages. The ALJ’s finding was not just that the
deals were not “consummated,” but that they never got past the first
date: The ALJ found that “nothing ever came of those meetings.”
A7814. They do nothing to show the required tangible steps toward a
future domestic industry.
As for public policy, “Congress recognized that the Commission is
fundamentally a trade forum, not an intellectual property forum, and
that only those intellectual property owners who are ‘actively engaged
in steps leading to the exploitation of the intellectual property’ should
have access to the Commission.” Mezzalingua, 660 F.3d at 1328
(quoting H.R. Rep. No. 100-40, at 157 (1987)). Motiva’s bald assertion
that it is a “licensing company” is not sufficient to allow it to complain
to the ITC. Motiva has nothing but its patents. If Motiva can invoke
the protections of the ITC, any and every patent owner can do so. That
is not the scheme Congress established. Id.
Case: 12-1252 Document: 46 Page: 50 Filed: 08/29/2012
41
B. Motiva’s Lawsuit Against Nintendo Is Not A TangibleStep Toward Establishing A Domestic Industry
The ALJ rejected Motiva’s argument that its district court
“litigation against Nintendo is evidence of Motiva taking the necessary
tangible steps to establish an industry.” A7834.
In Coaxial Cable, the Commission held that for litigation activities
to constitute “exploitation” of a patent under Section 337(a)(3)(C), the
complainant must “clearly link” its litigation efforts to licensing.
Certain Coaxial Cable Connectors and Components Thereof and
Products Containing Same, Inv. No. 337-TA-650, USITC Pub. No. 4283,
at 51 (Apr. 14, 2010) (Comm’n Op.). As the Commission explained,
“[a]llowing patent infringement litigation activities alone to constitute a
domestic industry would place the bar for establishing a domestic
industry so low as to effectively render it meaningless.” Id. at 46.
Motiva does not dispute that this settled agency practice is the correct
understanding of Section 337(a)(3)(C). See InterDigital, 2012 U.S. App.
LEXIS 15893, at *32 (noting that “[i]f there were any ambiguity” about
the scope of Section 337(a)(3)(C), the Commission’s “consistent
interpretation of the statute . . . would be entitled to deference”).
Case: 12-1252 Document: 46 Page: 51 Filed: 08/29/2012
42
Here, the ALJ concluded that “the litigation against Nintendo
does not relate to exploitation of the asserted patents.” A7832. Among
the ALJ’s reasons for so holding were:
“Emails between the inventors before the start of any litigation showthat they were interested in their potential ‘winnings’ from alawsuit against Nintendo.” A7822 (citing A10,548); A30,547 (“Themoney is suppose [sic] to come from Laniers [sic] cut of the winningsand not from our cut.”) (emphasis added).
“Motiva’s decision not to file a complaint at the ITC from the outsetor seek a preliminary injunction against Nintendo shows that Motivawas not concerned with taking swift actions to remove Nintendo fromthe market.” A7823.
“The inventors could have continued to seek investors, licensees, orpartners instead of, or in conjunction with, suing Nintendo.” A7824.
Moreover, the ALJ held that even if the litigation is related to the
patents, Motiva has not made a significant investment in the district
court litigation. “Motiva’s lawyers are working under a contingent fee
agreement.” A7831. “Motiva has not paid any attorney’s fees or
expenses related to Motiva’s litigation against Nintendo.” Id.
Despite all this, Motiva writes (at 52) that “the undisputed
evidence establishes that Motiva did not pursue litigation as a mere
patent owner.” According to Motiva, “it is important to remember that
Motiva’s litigation effort was in further service of its licensing goals.”
Case: 12-1252 Document: 46 Page: 52 Filed: 08/29/2012
43
Br. 52. But it is Motiva that has amnesia. The ALJ found that “the
litigation against Nintendo does not relate to exploitation of the
asserted patents.” A7832. If this litigation is “not about licensing,”
A30,364, then it cannot be evidence of Motiva’s “substantial investment
in . . . licensing,” 19 U.S.C. § 1337(a)(3)(C).
Motiva’s litigation-inspired arguments also forget a host of other
adverse factual findings. Motiva says (at 54) that the evidence showed
“Motiva and several business partners alike all saw the Wii as a
substantial roadblock to Motiva’s success.” But, as detailed further
below (at 47-49), the ALJ rejected this view of events. Motiva “failed to
demonstrate that there was any decline in interest caused by the
release of the Wii.” A7815. Motiva invokes the wisdom of its founders
on the necessity of litigation. Br. 54. But the ALJ specifically rejected
this assertion: “I do not concur with the inventors’ assertion that
because of the Wii, they had no other recourse but to sue Nintendo.”
A7824. Motiva notes that it “invested in the litigation,” Br.
58, but the ALJ ruled this time “insubstantial.” A7831; see also A7817
(“I give no weight to this wholly unsupported testimony from clearly
interested witnesses.”).
Case: 12-1252 Document: 46 Page: 53 Filed: 08/29/2012
44
Motiva’s “public policy” points in support of its ‘litigation alone is
enough’ theory of ITC authority misconstrues what the ALJ held.
Motiva states that the ALJ “implicitly held that when a company seeks
financial compensation for patent infringement, as well as injunctive
relief, it necessarily belies a bona fide domestic industry.” Br. 55. But
the ALJ made no such global ruling, “implicitly” or otherwise. The ALJ
pointed to Motiva’s failure to invoke certain procedures, such as a
preliminary injunction or exclusion order. Those procedures are
designed to protect industry. Motiva’s decision not to invoke those
protective procedures is a decision that confirms the lack of an industry
to protect. Where, as here, the complainants are on record as excited
about their litigation “winnings” and have ended any efforts to license
or commercially exploit the patents, a patent owner cannot cite
litigation focused on monetary relief as the sole evidence of steps toward
establishing an industry.
Motiva cites Certain Multimedia Display and Navigation Devices
and Systems, Components Thereof, and Products Containing the Same,
Inv. No. 337-TA-694, USITC Pub. No. 4292 (Nov. 2011) (Final), for the
premise that its litigation efforts should have been given more weight in
Case: 12-1252 Document: 46 Page: 54 Filed: 08/29/2012
45
the ALJ’s domestic industry analysis. In Certain Multimedia, the ITC
held that “section 337(a)(2) . . . requires complainant’s activities relate
to licensing.” Id. at 13. Here, however, the ALJ found as a matter of
fact that Motiva failed to establish any connection between its litigation
efforts and any licensing. No matter how frequently Motiva asserts
that it is in the licensing business, the ALJ found as a matter of fact
that its litigation against Nintendo had nothing to do with licensing.
A7830-31. Litigation unrelated to licensing, as Motiva concedes, cannot
establish a domestic industry. Br. 52.
Motiva’s reliance on Certain Digital Satellite System Receivers and
Components Thereof is similarly misplaced. Inv. No. 337-TA-392,
USITC Pub. No. 3418, at 11 (Apr. 2001) (Initial and Recommended
Determinations). The patentee in Certain Digital Satellite had reached
license agreements to the patent-in-suit with four licensees and every
single employee of the patentee was “responsible for maintaining [its]
system of identifying, approaching, and negotiating with prospective
licensees.” Id. at 10-11. As Motiva acknowledges, the Certain Digital
Satellite litigation efforts were “an extension of its licensing program.”
Case: 12-1252 Document: 46 Page: 55 Filed: 08/29/2012
46
Id. at 11 (emphasis added). But here, there is no licensing program
other than the litigation effort.
C. The Wii System’s Success Does Not Suggest MotivaHas Any Chance Of Establishing A Domestic IndustryIn The Future
The ALJ ruled that “Motiva could not demonstrate that there was
any significant interest in its technology prior to the Wii’s existence,”
and “there is no reason to believe that manufacturers of fitness and
rehabilitation equipment will suddenly become interested in the
technology because Nintendo has been excluded from the market.”
A7835. In addition, the ALJ found that
A7836.
Against these dispositive findings, Motiva makes three limited
and erroneous points.
Much of Motiva’s brief turns on its contention that “[b]ut for the
presence of the Wii in the market, Motiva was—and still is—extremely
close to realizing its product-driven licensing goals.” Br. 44; see Br. 41-
42, 58-59. For example, Motiva cites the testimony of Reiss (he “would
still be willing to invest in a license to Motiva’s technology today if
Motiva has the power to exclude Nintendo”) and of Highsmith (“there is
Case: 12-1252 Document: 46 Page: 56 Filed: 08/29/2012
47
potential for a product that has [Motiva’s] characteristics”). Br. 44.
Motiva argues that “[s]ubstantial evidence supports a conclusion that
Motiva’s invention competes with the Wii.” Br. 50.
The ALJ explained why “the Wii Fit [an exercise game] would not
compete with the Motiva product.” A7818. “Even though the Wii Fit
may be viewed by some as a fitness product, it is not a serious fitness
product like the one described in the Motiva documentation.” Id.
“Based” on this review of the evidence, the ALJ found “that the Motiva
product was intended to be an expensive tool used by people in the
physical rehabilitation and fitness industries, while the Nintendo Wii is
a relatively inexpensive video game system for home consumers that is
intended to appeal to a mass market.” A7819. See, e.g., A20,708-09 (“Q:
Now, do you think a normal American consumer is going to pay
for a device and put it in their living room to play video games? A: No.”).
Motiva makes no effort to dispute these substantial facts the ALJ relied
on to find no competition between the Wii System and any Motiva
product.
Moreover, any argument that the November 2006 introduction of
the Wii System doomed Motiva’s domestic industry ignores the ALJ’s
Case: 12-1252 Document: 46 Page: 57 Filed: 08/29/2012
48
findings. “The proper date for determining whether [Motiva’s licensing
efforts] constituted an ‘industry’ entitled to protection under section 337
was the date on which the complaint was filed.” Bally/Midway, 714
F.2d at 1121. Motiva’s complaint was filed in September 2010. A8759.
But, as detailed above at 25, the ALJ properly found that Motiva’s
efforts, if any, to establish a domestic industry had ceased by 2007.
A7829. Any effect of the Wii System on Motiva’s prospects must have
likewise ended by 2007, three years before the complaint was filed here.
The ALJ also specifically rejected the fiction that Motiva’s
licensing efforts were hurt by the introduction of the Wii System.
“Neither Mr. Reiss nor Motiva has offered any evidence to support the
assertion that [Mr. Reiss’s company] would license the Motiva patents,
but for the presence of the Wii.” A7812. Instead, the ALJ found that
Mr. Reiss’s “real interest lies in excluding Nintendo from the market” to
protect his “physical therapy device” from competition from the Wii Fit
game. A7812-13 (citing A20,633-34). And the ALJ found that Mr.
Highsmith had “little interest” in Motiva; he never viewed the prototype
or patent application. A7814; see also A7815 (“[E]ven though Mr.
Highsmith was aware of Motiva’s technology prior to the Wii’s release,
Case: 12-1252 Document: 46 Page: 58 Filed: 08/29/2012
49
he never showed enough interest to view the prototype, view Motiva’s
patents or patent applications, hold a meeting with the inventors in
[his] offices, or present the invention to [his company’s] innovation
committee.”).8
Motiva incorrectly states (at 59-60) that the ALJ’s findings are
“premised largely” on its noninfringement ruling. According to Motiva,
should this Court find that the Wii System infringes the Motiva patents
(it does not, see below at Section II), then the ALJ’s analysis of the
prospects of Motiva’s products also falls. But the ALJ expressly stated
that its factual findings were independent of the infringement ruling:
“Motiva has not demonstrated that there is a significant likelihood that
the industry requirement will be satisfied in the future, and this
determination is not dependent on the Commission’s actions in
this investigation.” A7834 (emphasis added). The ALJ noted
testimony by Motiva’s inventors to the contrary, but the ALJ did “not
find this testimony to be persuasive,” A7835, dismissing it as “self-
interested testimony of the inventors.” A7834. In short, regardless of
8 Among other things, the ALJ also rejected Motiva’s “claim that thetechnology is currently ready to be commercialized.” A7820 (“theMotiva prototype was not close to being production-ready”).
Case: 12-1252 Document: 46 Page: 59 Filed: 08/29/2012
50
whether the Wii System does not or does infringe Motiva’s patents, the
ALJ found that Motiva still had no prospects of marketplace success.
No one was interested before the Wii System arrived, and no one would
be interested if the Wii System suddenly disappeared from the market.
For all these reasons, the ALJ was correct to find that the Wii
System’s success had nothing to do with the failure of Motiva’s
technology. But in any event, the Court need only find that substantial
evidence supports that conclusion. This Court affirms agency findings
where substantial evidence supports several competing conclusions.
Nippon Steel Corp. v. United States, 458 F.3d 1345, 1352
(Fed. Cir. 2006) (“‘[E]ven if it is possible to draw two inconsistent
conclusions from evidence in the record, such a possibility does not
prevent [the Commission’s] determination from being supported by
substantial evidence.’” (quoting Am. Silicon Techs. v. United States, 261
F.3d 1371, 1376 (Fed. Cir. 2001))).
Case: 12-1252 Document: 46 Page: 60 Filed: 08/29/2012
51
II. SUBSTANTIAL EVIDENCE SUPPORTS THE ALJ’SFINDING THAT THE WII SYSTEM DOES NOT INFRINGEMOTIVA’S PATENTS
In light of how the Wii System actually operates, substantial
evidence supports the ITC’s conclusion that the Wii System does not
infringe Motiva’s patents and must be affirmed. First, the Wii System
does not “track” movement of the user—a fact which, by itself, is a basis
on which to affirm. Second, and independently, the Wii System does
not determine user “movement.” It does not determine the position or
orientation of the user (under the ALJ’s construction). Nor does it
determine the position and orientation of the user (under the proper
construction Motiva incorrectly says the ALJ actually applied).
Motiva’s scattershot effort to challenge these rationales is unpersuasive.
A. Substantial Evidence Shows That The Wii SystemDoes Not “Track” Movement Of A User
Substantial evidence supports the Commission’s determination
that the Wii System does not “track” movement of a user. A7777-82.
There was no dispute as to what one of skill in the art understood
“tracking” movement to mean. The only witness to testify on this topic,
Professor Colgate, explained that
Case: 12-1252 Document: 46 Page: 61 Filed: 08/29/2012
52
A7777
(citing A30,437-38); see also A10,320-21. This is also how the patents
and others of skill in the art use the term “tracking.” A10,321-30.
The ITC’s holding—that the Wii lacks each of the three elements
that define tracking—is supported by the documents and the testimony.
First,
A7777; see A30,430-33.
A7777-78.
A30,430.
Second, the ALJ found that the Wii does not measure user
movement with A7777.
A7777.
Case: 12-1252 Document: 46 Page: 62 Filed: 08/29/2012
53
Third, the ALJ found that the Wii System did not keep any
Id. In an effort
to contradict this finding, Motiva invokes its expert, Dr. Jaswinder Pal
Singh,
A9920; see also A7779. But the ALJ found that Dr. Singh was
not qualified as an expert on motion-sensing devices (A20,154); he was
an expert in computer systems and electrical engineering. A20,153.
The ALJ was justified in finding that Nintendo manager
A7779.
Id. (citing A30,566, 30,592, 30,593-94, 30,596).
A7779.
Id. (emphasis in original).
Case: 12-1252 Document: 46 Page: 63 Filed: 08/29/2012
54
Simply put, tracking a fictional position in a three-dimensional virtual
world is not the same as tracking the human playing the game.
Motiva argues that the ALJ’s construction of “tracking” “directly
contradicts the Commission’s apparent construction of ‘tracking
movement of a user.’” Br. 24-25 (citing A7674 at ¶¶ 3-4). Nonsense.
The cited paragraphs do not construe the term “tracking.” The
paragraphs construe “tracking movement of a user” to mean “tracking
changes of position and/or orientation of a user.” A7674 (emphasis
added). So as relevant here, all that one can glean from this claim
construction is that the Commission said “tracking” means “tracking.”
The bottom line is that the Wii System does not “track”
movement, and nothing in Motiva’s brief casts doubt on that fact. This
alone requires affirmance.
B. Substantial Evidence Shows That The Wii SystemDoes Not Determine User “Movement”
In light of the above, this Court need not proceed to the next step
and address whether the Wii System determines user “movement” as
required by Motiva’s patents. But should the Court reach the issue, the
answer is “No”: The ALJ correctly found that the Wii sensors do not
determine user “movement,” i.e., “changes of position and/or orientation
Case: 12-1252 Document: 46 Page: 64 Filed: 08/29/2012
55
of a user” (or, consistent with Nintendo’s construction, changes of
position and orientation of a user). A7772. Motiva’s attacks on that
finding are meritless.
1. Three Credible Witnesses Demonstrated ThatThe Wii Sensors Cannot And Do Not DeterminePosition Or Orientation
To understand why the Wii System does not determine the
position or orientation of the user, it is useful to understand how the
Wii System actually works. As anyone who has played it or watched
the videos submitted with this brief might ask, isn’t the whole fun of the
Wii System that it responds to the player’s movement? See A7752
(“90% of consumer purchase decisions are based upon the Wii’s ability
to track motion”). In fact, the Wii System “trick[s]” the user into
thinking it knows the user’s movement. A20,439. The ALJ found that
the Wii “create[s] a convincing display of character movement on the
screen through animations based on real time computations that merely
give the appearance of tracking player movements.” A7780 (emphasis
added).
The ALJ found Nintendo’s witnesses credible:
Case: 12-1252 Document: 46 Page: 65 Filed: 08/29/2012
56
A7780. Mr. Ohta participated in the development of the Wii prototype
and designed many of the Wii’s most popular games, including Wii
Sports. A21,395, 21,401. He explained, in testimony endorsed by the
ALJ, how the illusion of tracking user movement was implemented in
the popular Wii Tennis game.
Case: 12-1252 Document: 46 Page: 66 Filed: 08/29/2012
57
Thus, the Wii System determines which tennis animation to
display without determining either the position or orientation.
A7762 (citing A21,479-80, 21,482, 30,575). See also SVA
at RDX-156.
The record includes various other examples that confirm the Wii
System works without tracking the user’s position or orientation.
Case: 12-1252 Document: 46 Page: 67 Filed: 08/29/2012
58
Dr. Colgate explains:
A30,452.
So too with boxing. As demonstrated in the Wii Boxing video,
SVA at RDX-154, the
Case: 12-1252 Document: 46 Page: 68 Filed: 08/29/2012
59
In sum, the Wii System does not determine either the position or
orientation of the user.
2. Motiva’s Hodgepodge Of Movement And PositionInformation Arguments Are All Meritless
Regarding the alleged infringement of the movement claim term,
Motiva raises a host of scattered arguments that are all meritless.
1. In an attempt to circumvent the ALJ’s findings, Motiva now
emphasizes that the gyroscope measures the “changes” in orientation by
measuring angular velocity. Motiva asserts that the ALJ erred by
evaluating “infringement based on whether the Wii determines the
‘position or orientation’ of the user instead of changes in the position
and/or orientation of a user.” Br. 22 (Motiva’s emphasis). Motiva
argues that the Wii’s gyroscope detects “changes in orientation over
time.” Id. at 23 (Motiva’s emphasis).
Case: 12-1252 Document: 46 Page: 69 Filed: 08/29/2012
60
But it is Motiva’s arguments that have changed. Before trial and
at the hearing, Motiva argued that the Wii System “determine[s]”
(A30,129) or “describes” (A9797) the orientation of the transponder.
Only later, in the petition to the ITC, did Motiva advance its newfound
theory that what matters is that the Wii System determines orientation
“changes.” See, e.g., A8602. The law is clear that arguments not raised
before the ALJ are waived. Finnigan Corp. v. ITC, 180 F.3d 1354, 1363
(Fed. Cir. 1999) (“The argument at the trial and appellate level should
be consistent, thereby ensuring a clear presentation of the issue to be
resolved.”).
In any event, the Wii System does not measure changes in the
orientation or position of the user.
It is undisputed that gyroscopes detect only “angular velocity.”
Br. 13. Angular velocity is the speed of rotation around an axis. A7755;
see A20,160; 20,174.
A20,160, 20,174; see also A30,423.
Case: 12-1252 Document: 46 Page: 70 Filed: 08/29/2012
61
A7775 (citing A20,412-15). Motiva agrees that the
gyroscope “cannot detect where an object is facing at a particular
moment in time.” Br. 13. The ALJ also embraced testimony from
A7776-77; see also A30,423-24, 30,430-31.
A7776-77.
Motiva argues that the accelerometers determine “changes in
position.” Br. 14 (emphasis added). It is undisputed that
accelerometers do nothing but measure acceleration. As with the
orientation argument, Motiva’s “changes” position argument is new and
Case: 12-1252 Document: 46 Page: 71 Filed: 08/29/2012
62
waived. See, e.g., A20,194
Moreover, the ALJ cited
substantial evidence in support of the finding that the accelerometers
do not determine position or any changes in position.
A7775.
A20,285
The ALJ also again credited Dr. Colgate’s testimony.
A7774.
A7775.
A7774.
Gravity provides a constant acceleration on every object; all objects
Case: 12-1252 Document: 46 Page: 72 Filed: 08/29/2012
63
register acceleration even when not moving. Id. (citing A30,417-18).
Therefore, one cannot simply calculate out the gravitational
acceleration because, for example,
A30,553.
A7775 (citing A20,234).
To the extent Motiva argues that the accelerometers determine
orientation,
A7776. Motiva’s patents concede that accelerometers can
only be used to determine pitch and roll when the device is “static.” A30
at 33:60-61; see also A30,418.
Furthermore, the ALJ found that Mr. Rabin
A7775. The ALJ
noted that Mr. Rabin had explained his presentation to Wii developers
Case: 12-1252 Document: 46 Page: 73 Filed: 08/29/2012
64
A7775-76. But
the ALJ noted that Mr. Rabin had
A7776. In other words,
Id.
And the ALJ found that
Id.
Id. Of course, measuring only
pitch and roll of a stationary transponder cannot determine user
movement.
In short, Motiva’s last-minute change in argument would be
unpersuasive even if it had not been waived.
Moreover, Motiva’s new infringement argument still requires
“tracking”—something the Wii System does not do. See above at II.A.
2. Next, Motiva argues that “movement” and “position” do not
necessarily involve movement or position in all three spatial
Case: 12-1252 Document: 46 Page: 74 Filed: 08/29/2012
65
dimensions. Br. 15-18, 26. It is enough, Motiva argues, if a device can
figure out “that an object is ‘above’ or ‘to the left of’ another object.” Id.
at 26.
The ALJ’s opinion demonstrates why Motiva is wrong to insist
that two-dimensional detection is enough under the claims. The ALJ
noted that the patent “specification explains that the invention may be
used ‘for the purposes of functional movement assessment for exercise,
and physical medicine and rehabilitation.’” A7678. “Such tracking,”
the ALJ held, “requires knowledge of the user’s location in 3D space.”
Id. A “user” is a human being, not a cartoon, which means that the user
resides, and moves, in three-dimensional space. Tracking in fewer than
three dimensions is not tracking “the user” as the claims require.
“Motiva cites no evidence from the ‘151 patent that supports a finding
that the claimed invention may track position in only one or two
dimensions.” A7679. Thus, the ALJ found “it appropriate to include
express reference to 3D space in the proposed construction.” A7678.
Motiva contends that the plain and ordinary meaning of position
and movement information does not require three-dimensional
tracking. In support of this view, Motiva invokes (at 26) Retractable
Case: 12-1252 Document: 46 Page: 75 Filed: 08/29/2012
66
Technologies v. Becton, Dickinson & Co., 653 F.3d 1296, 1306
(Fed. Cir. 2011), and (at 19) Northrop Grumman Corp. v. Intel Corp.,
325 F.3d 1346, 1355-56 (Fed. Cir. 2003). Those cases stand for the rule
that “statements from the description of the preferred embodiment . . .
are just that—descriptions of a preferred embodiment” and that such
statements do not disavow claim scope absent a clear disclaimer.
Northrop Grumman, 325 F.3d at 1356; see also Retractable, 653 F.3d at
1306 (specification disclosed an embodiment that operates by “cutting”
and mere statements about difficulties with prior art “cutting” type
devices did not disclaim claim scope).
This rule, however, has no bearing here because the ALJ did not
find that the asserted patents disclosed and then disclaimed
embodiments that track in one or two dimensions. Rather, the ALJ
found that every embodiment of “the invention” tracks the user in three
dimensions, and Motiva cited “no evidence . . . that the claimed
invention may track position in only one or two dimensions.” A7679.
This finding is confirmed by the specification’s statement that “[t]he
present invention described can be used . . . to measure frequency and
amplitude of body sway in three dimension (3D) space.” A21 at 16:8-13.
Case: 12-1252 Document: 46 Page: 76 Filed: 08/29/2012
67
And it is further confirmed by the statement that “[t]he present
invention” can be used to determine whether an “individual’s movement
trajectory varied from the intended two dimensional (2D) reference
movement trajectory by deviation from the planar path into the
uninvolved spatial dimension,” which would require tracking in three
dimensions. A22 at 18:3-16. Thus, where the specification recites that
“the invention” measures in three dimensions and only discloses
embodiments that track the user in three dimensions, tracking
movement and position must be in three dimensions. That is the only
construction that properly “tether[s] the claims to what the
specifications indicate the inventor actually invented.” Retractable, 653
F.3d at 1305.
None of Motiva’s arguments come close to overcoming the
conclusion compelled by the specification. Motiva cites (at 15-16, 20) to
a “dual axis inertial sensor” in a preferred embodiment. But the
specification explains that this sensor is optional; the sensor is used to
provide supplemental pitch and roll information. A28 at 30:43-31:50;
see also A30,538. Moreover, the asserted patents acknowledge that
such dual axis inertial sensors can only measure pitch and roll when
Case: 12-1252 Document: 46 Page: 77 Filed: 08/29/2012
68
“static,” making them useless for tracking movement. A30 at 33:60-61.
Motiva mischaracterizes a “Limits of Stability” test by asserting
that it measures body sway in just two dimensions. Br. 16. The
specification states twice that “body sway [is measured] in three
dimension [sic] (3D) space.” A21 at 16:12-13, 22 at 7:23-24.
See A30,539-40
Motiva also mischaracterizes the specification by stating that it
“describes tracking using ‘mechanical, inertial, acoustical or
electromagnetic radiation sensors.’” Br. 19. The passage that Motiva
cites, however, concerns the prior art—not the claimed invention:
“Known are commercial tracking and display systems that employ . . .
mechanical, inertial, acoustical or electromagnetic radiation sensors to
determine a mobile object’s position and orientation, referred to
collectively as pose.” A14 at 1:18-21.
Case: 12-1252 Document: 46 Page: 78 Filed: 08/29/2012
69
Motiva also cites to three prior art references that allegedly
disclose tracking humans in fewer than three dimensions. Br. 16-18.
As the ALJ recognized, however, Motiva’s citations to the prior art
merely shift the focus away from the central inquiry: what one skilled
in the art would understand “tracking” a “user” to mean in the asserted
patents. A7678-79.
As explained above, supra at 60-64 neither the gyroscopes nor the
accelerometers determine, much less track, position or orientation.
Motiva also emphasizes the Wii System’s camera, the DPD. Br. 37-39.
But, again, the ALJ’s fact findings are dispositive.
The ALJ had substantial evidence to support the finding that the
DPD does not determine position or orientation in any dimension. Dr.
Colgate
A7773.
A7774.
Thus, as demonstrated at the hearing,
Case: 12-1252 Document: 46 Page: 79 Filed: 08/29/2012
70
A7773. Indeed,
Id.
3. Motiva’s brief includes an attempt to circumvent its burden
under the substantial evidence standard by claiming that the ALJ first
construed “movement information” correctly but then “applied a
construction for ‘movement information’ that was inconsistent with its
initial construction.” Br. 32.
Motiva is wrong on the law, wrong on what the ALJ did here, and
wrong on the correct claim construction.
Motiva asserts that when an ALJ construes terms in one fashion
and then applies them in an arguably different fashion, the de novo
standard of review applies. Id. at 32. In support of its argument,
Motiva incorrectly describes the holding of Vita-Mix Corp. v. Basic
Holding, 581 F.3d 1317 (Fed. Cir. 2009). Motiva contends that Vita-Mix
stands for the proposition that “a lower court’s application of a
construction that is ‘inconsistent with its earlier claim construction’ is
Case: 12-1252 Document: 46 Page: 80 Filed: 08/29/2012
71
[reviewed] de novo to determine correct construction.” Br. 32. Vita-Mix,
however, says no such thing. Vita-Mix was an appeal from a summary
judgment of non-infringement, and thus the Court merely applied the
settled rule that grants of summary judgment are reviewed de novo.
Vita-Mix, 581 F.3d at 1323.
More instructive here is Tessera, Inc. v. ITC, 646 F.3d 1357 (Fed.
Cir. 2011), which Motiva asserts, without explanation, does not apply.
Br. 32-33. Like Motiva, Tessera won its claim construction, but the ITC
nevertheless found no infringement. Tessera, 646 F.3d at 1363-64.
Like Motiva, Tessera appealed to this Court and argued that the ITC
“initially adopted a correct claim construction, but ‘halfway through its
infringement analysis, the ITC inexplicably switched to an incorrect
claim construction’” and, thus, the ITC’s finding of noninfringement
should be reviewed “de novo, as an error in claim construction.” Id. at
1364. The Court, however, found that Tessera could not argue that the
ITC applied the wrong construction because it adopted Tessera’s
proposed construction and thus its “contention at best [was] a
disagreement over the Commission’s application of Tessera’s
construction to the accused . . . devices.” Id. Thus, the Court held that
Case: 12-1252 Document: 46 Page: 81 Filed: 08/29/2012
72
Tessera’s appeal was a challenge of the ITC’s “infringement
determination,” and should be “review[ed] for substantial evidence.” Id.
For the same reasons, the substantial evidence burden applies here.
In any event, the ALJ in fact applied the construction of “tracking
movement of a user” as “tracking changes of position and/or orientation
of a user.” The ALJ adopted Motiva’s construction of “movement
information,” construing it to mean “information about changes in
position and/or orientation.” A7674, 7676. The ALJ referred to this
construction at the outset of the infringement discussion: “The question
becomes whether or not the accused products ‘track changes of position
and/or orientation of a user.’ I find that they do not.” A7772. The ALJ
then described Motiva’s arguments regarding the gyroscope,
accelerometer, and DPD, and held they “do not, in fact, track the
movement of the user or provide information regarding the position or
orientation of the user.” A7773 (emphasis added).
A7777. And, indeed,
that is Nintendo’s view. But the ALJ ruled differently. A7674.
Case: 12-1252 Document: 46 Page: 82 Filed: 08/29/2012
73
Nothing in this accurate recitation of Dr. Colgate’s testimony suggests
that the ALJ had a change of heart.
A7777.
The ALJ was citing Dr. Colgate for the point that the Wii System
determines neither position nor orientation.
Nintendo continues to disagree with the ALJ’s construction of
movement. But the Court can and should resolve this appeal without
deciding whether the claims require determining both position and
orientation. As the ALJ held, the Wii System does not track either a
user’s position or a user’s orientation. A7773. There is no need to
decide whether a device that tracks only position would infringe the
claims.
4. Motiva repeatedly invokes the reexamination process of its
patents. Br. 14, 16, 18, 19-20, 27-31. But the reexamination is ongoing
and in any event not meaningfully relevant here.
Motiva emphasizes its “own actions” during reexamination. Br.
27-30. According to Motiva, its decision to press on with rejected claims
rather than canceling or amending the claims suggests that it did not
disavow the broader claim scope. Br. 28. But nothing in the ALJ’s
Case: 12-1252 Document: 46 Page: 83 Filed: 08/29/2012
74
claim construction turns on whether or not Motiva disavowed any claim
scope.
Motiva complains (at 28) that the ALJ should not have
disregarded the (still incomplete) reexamination record. Motiva cites
cases that stand for the proposition that a patent owner’s statements
during an incomplete prosecution can limit the claims.9 The cases also
suggest that any points raised by an examiner in a prosecution can be
evaluated for substantive persuasiveness.10
Under this authority, the ALJ’s decision to give little weight to the
incomplete examination was proper. Motiva seeks to use incomplete
proceedings to expand rather than narrow its claims. And, as the ALJ
9 See, e.g., In re Katz Interactive Call Processing Patent Litig., 07-ml-01816-BRGK (FFMx), 2008 WL 4952454, at *5 (C.D. Cal. Feb. 21, 2008)(“[T]he prosecution history can often inform the meaning of the claimlanguage . . . making the claim scope narrower than it would otherwisebe.”) (emphasis added), aff’d 639 F.3d 1303 (Fed. Cir. 2011); BeneficialInnovations, Inc. v. Blockdot, Inc., Nos. 2:07-CV-263-TJW-CE, 2:07-CV-555-TJW-CE, 2010 U.S. Dist. LEXIS 54151, at *8 (E.D. Tex. June 3,2010) (“[S]tatements made by a patentee during reexamination todistinguish a claim from the prior art may serve to limit the scope of theclaim.”) (emphasis added).
10 See, e.g., St. Clair Intellectual Prop. Consultants, Inc. v. Canon Inc.,412 F. App’x 270, 276-77 (Fed. Cir. 2011); SRAM Corp. v. AD-II Eng’g,Inc., 465 F.3d 1351, 1357, 1359 (Fed. Cir. 2006) (rejecting constructionadopted by examiner after three reexaminations).
Case: 12-1252 Document: 46 Page: 84 Filed: 08/29/2012
75
found, “the statements made by the Examiner regarding claim
construction are not persuasive.” A7679; see also supra II.A. Motiva
repeatedly notes (at 14, 16, 18, 19-20) that the examiner adopted
Nintendo’s definition of “movement.” But the statements that Motiva
cites were made under the express qualification that these are not
Nintendo’s arguments for the purposes of litigation because federal
courts use a narrower standard than the PTO in reexamination. The
ALJ’s treatment of the reexamination proceedings provides no basis for
reversal.
CONCLUSION
For the foregoing reasons, the ITC’s decision should be affirmed.
Case: 12-1252 Document: 46 Page: 85 Filed: 08/29/2012
76
Date: August 29, 2012 Respectfully submitted,
/s/ Mark S. Davies
Mark S. DaviesKatherine M. KoppORRICK, HERRINGTON & SUTCLIFFE LLP1152 15th Street, N.W.Washington, D.C. 20005(202) [email protected]
E. Joshua RosenkranzPeter A. BicksAlex V. ChachkesORRICK, HERRINGTON & SUTCLIFFE LLP51 West 52nd StreetNew York, NY 10019(212) 506-5000
Joseph S. PrestaRobert W. FarisNIXON & VANDERHYE P.C.901 North Glebe Road11th FloorArlington, VA 22203(703) 816-4000
Attorneys for Nintendo Co., Ltd. andNintendo of America Inc./Intervenors
Case: 12-1252 Document: 46 Page: 86 Filed: 08/29/2012
77
CERTIFICATE OF COMPLIANCEUNDER FEDERAL RULES OF APPELLATE PROCEDURE
32(A)(7) AND FEDERAL CIRCUIT RULE 32
Counsel for Intervenors Nintendo Co., Ltd. and Nintendo of
America Inc. certify that the brief contained herein has a proportionally
spaced 14-point typeface, and contains 13,791 words, based on the
“Word Count” feature of Word 2007, including footnotes and endnotes.
Pursuant to Federal Rule of Appellate Procedure 32(a)(7)(B)(iii) and
Federal Circuit Rule 32(b), this word count does not include the words
contained in the Certificate of Interest, Table of Contents, Table of
Authorities, Abbreviations, and Statement of Related Cases.
Dated: August 29, 2012 Respectfully submitted,
/s/ Mark S. DaviesMark S. Davies
Case: 12-1252 Document: 46 Page: 87 Filed: 08/29/2012
CERTIFICATE OF SERVICE
I certify that on the 29th day of August, 2012, the foregoing Non-
Confidential Answering Brief of Intervenors Nintendo Co., Ltd. and
Nintendo of America Inc. was electronically filed using the Court’s
CM/ECF System, which will automatically serve all counsel of record.
I further certify that two copies of the foregoing Non-Confidential
Answering Brief of Intervenors Nintendo Co., Ltd. and Nintendo of
America Inc. will be served at the time the paper copies of the brief are
submitted to the Court:
Clark S. CheneyWayne W. HerringtonDominic L. BianchiU.S. International Trade Commission500 E Street, S.W., Suite 707Washington, D.C. [email protected]
Counsel for Appellee International Trade Commission
Christopher D. BanysLanier Law Firm, P.C.2200 Geng Rd.Suite 200Palo Alto, CA [email protected]
Counsel for Appellant Motiva, LLC
/s/ Mark S. DaviesMark S. Davies
Case: 12-1252 Document: 46 Page: 88 Filed: 08/29/2012
Addendum
Case: 12-1252 Document: 46 Page: 89 Filed: 08/29/2012
Motiva, LLC v. International Trade CommissionFed. Cir. Appeal No. 2012-1252
Index to Addendum
DOCUMENT PAGE NO.
Patent No. 7,292,151, Dated November 6, 2007....................... A1-35
Patent No. 7,492,268, Dated February 17, 2009 .................... A36-67
Case: 12-1252 Document: 46 Page: 90 Filed: 08/29/2012
,-,
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
June 08, 20lO
THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COpy FROM
THE RECORDS OF THIS OFFICE OF:
U.S. PATENT: 7,292,151
ISSUE DATE: November 06,2007
By Autbority of tbe
Under Secretary of Commerce for Intellectual Property and Director of tbe United States Patent and Trademark Office
,-,
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
June 08, 20lO
THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COpy FROM
THE RECORDS OF THIS OFFICE OF:
U.S. PATENT: 7,292,151
ISSUE DATE: November 06,2007
By Autbority of tbe
Under Secretary of Commerce for Intellectual Property and Director of tbe United States Patent and Trademark Office
JX-001
A000001
Case: 12-1252 Document: 46 Page: 91 Filed: 08/29/2012
(12) United States Patent Ferguson et al.
(54) HUMAN MOVEMENT MEASUREMENT SYSTEM
(76) Inventors: Kevin Ferguson, 8156 Camp den Lakes Blvd., Dublin, OH (US) 43016; Donald Gronachan, 4 Spiral Rd., Holtsville, NY (US) 11742
( *) Notice: Subject to any disclaimer, the tenn of this patent is extended or adjusted under 35 U.S.c. 154(b) by 47 days.
(21) Appl. No.: 111187,373
(22) Filed: Jul. 22, 2005
(65) Prior Publication Data
US 2006/0022833 Al Feb. 2, 2006
Related U.S. Application Data
(60) Provisional application No. 60/592,092, filed on luI. 29,2004.
(51) Int. CI. G08B 23/00 (2006.01)
(52) U.S. CI. ............................... 340/573.1; 340/407.1; 434/114
(58) Field of Classification Search ............. 340/573.1,
(56)
340/573.4,539.12,539.13,539.22,407.1, 340/825.36; 3811315; 4341112,114
See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
4,337,049 A 611982 Connelly .................... 434/247 4,375,674 A 311983 Thornton .................... 364/559 4,627,620 A 1211986 Yang ....................... 273/1 GC 4,631,676 A 1211986 Pugh .......................... 364/413 4,645,458 A 211987 Williams .................... 434/251
4,695,953 A 911987 Blair et al. ................. 364/410 4,702,475 A 1011987 Elstein et aI ............. 273/1 GC 4,751,642 A 611988 Silva et aI . ................. 364/413 4,817,950 A 411989 Goo ....................... 273/148 B
111111 1111111111111111111111111111111111111111111111111111111111111 US007292151B2
(10) Patent No.: US 7,292,151 B2 Nov. 6,2007 (45) Date of Patent:
4,912,638 A 3/1990 Pratt .......................... 600/595
4,925,189 A 5/1990 Braeunig ................ 273/148 B 5,148,154 A 9/1992 Mackay et al. ............. 3401712 5,184,295 A 2/1993 Mann ......................... 364/410
5,214,615 A 5/1993 Baur .......................... 367/128 5,227,985 A 7/1993 DeMenthon ................ 364/559
5,229,756 A 7/1993 Kosugi et al. .............. 340/706 5,239,463 A 8/1993 Blair ............................. 463/3
5,255,211 A 10/1993 Redmond ................... 365/401
5,288,078 A 2/1994 Capper et al. .......... 273/148 B 5,320,538 A 6/1994 Baum ......................... 434/307
5,347,306 A 9/1994 Nitta .......................... 348/578 5,372,365 A 12/1994 McTeigue et al. ....... 273/182.2
5,375,610 A * 12/1994 LaCourse et al. ........... 600/595
(Continued)
FOREIGN PATENT DOCUMENTS
WO PCTlUS96117580 5/1997
OTHER PUBLICATIONS
Reality built for two: a virtual reality tool, Symposium on Interactive 3D Graphics, ACM Press webpages from http://portal.acm.org/ citation.cfm?id~91385.91409&dl+ACM&type~series&i (Jun. 10, 2004) 1-4.
(Continued)
Primary Examiner-Toan N. Pham (74) Attorney, Agent, or Firm-Standley Law Group LLP
(57) ABSTRACT
A system for measuring the position of transponders for testing and training a user to manipulate the position of the transponders while being guided by interactive and sensory feedback through a bidirectional communication link to a processing system for the purpose of functional movement assessment for exercise and physical rehabilitation.
91 Claims, 10 Drawing Sheets
(12) United States Patent Ferguson et al.
(54) HUMAN MOVEMENT MEASUREMENT SYSTEM
(76) Inventors: Kevin Ferguson, 8156 Camp den Lakes Blvd., Dublin, OH (US) 43016; Donald Gronachan, 4 Spiral Rd., Holtsville, NY (US) 11742
( *) Notice: Subject to any disclaimer, the tenn of this patent is extended or adjusted under 35 U.S.c. 154(b) by 47 days.
(21) Appl. No.: 111187,373
(22) Filed: Jul. 22, 2005
(65) Prior Publication Data
US 2006/0022833 Al Feb. 2, 2006
Related U.S. Application Data
(60) Provisional application No. 60/592,092, filed on luI. 29,2004.
(51) Int. CI. G08B 23/00 (2006.01)
(52) U.S. CI. ............................... 340/573.1; 340/407.1; 434/114
(58) Field of Classification Search ............. 340/573.1,
(56)
340/573.4,539.12,539.13,539.22,407.1, 340/825.36; 3811315; 4341112,114
See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
4,337,049 A 611982 Connelly .................... 434/247 4,375,674 A 311983 Thornton .................... 364/559 4,627,620 A 1211986 Yang ....................... 273/1 GC 4,631,676 A 1211986 Pugh .......................... 364/413
4,645,458 A 211987 Williams .................... 434/251
4,695,953 A 911987 Blair et al. ................. 364/410
4,702,475 A 1011987 Elstein et aI ............. 273/1 GC 4,751,642 A 611988 Silva et aI . ................. 364/413
4,817,950 A 411989 Goo ....................... 273/148 B
111111111111111111111111111111111111111111111111111111111111111111111111111 US007292151B2
(10) Patent No.: US 7,292,151 B2 Nov. 6,2007 (45) Date of Patent:
4,912,638 A 3/1990 Pratt .......................... 600/595
4,925,189 A 5/1990 Braeunig ................ 273/148 B 5,148,154 A 9/1992 Mackay et al. ............. 3401712 5,184,295 A 2/1993 Mann ......................... 364/410
5,214,615 A 5/1993 Baur .......................... 367/128
5,227,985 A 7/1993 DeMenthon ................ 364/559
5,229,756 A 7/1993 Kosugi et al. .............. 340/706 5,239,463 A 8/1993 Blair ............................. 463/3
5,255,211 A 10/1993 Redmond ................... 365/401
5,288,078 A 2/1994 Capper et al. .......... 273/148 B 5,320,538 A 6/1994 Baum ......................... 434/307
5,347,306 A 9/1994 Nitta .......................... 348/578
5,372,365 A 12/1994 McTeigue et al. ....... 273/182.2
5,375,610 A * 12/1994 LaCourse et al. ........... 600/595
(Continued)
FOREIGN PATENT DOCUMENTS
WO PCTlUS96117580 5/1997
OTHER PUBLICATIONS
Reality built for two: a virtual reality tool, Symposium on Interactive 3D Graphics, ACM Press webpages from http://portal.acm.org/ citation.cfm?id~91385.91409&dl+ACM&type~series&i (Jun. 10, 2004) 1-4.
(Continued)
Primary Examiner-Toan N. Pham (74) Attorney, Agent, or Firm-Standley Law Group LLP
(57) ABSTRACT
A system for measuring the position of transponders for testing and training a user to manipulate the position of the transponders while being guided by interactive and sensory feedback through a bidirectional communication link to a processing system for the purpose of functional movement assessment for exercise and physical rehabilitation.
91 Claims, 10 Drawing Sheets
JX-001.0002
A000002
Case: 12-1252 Document: 46 Page: 92 Filed: 08/29/2012
US 7,292,151 B2 Page 2
u.s. PATENT DOCUMENTS 6,400,452 Bl 6,430,997 Bl 6,487,906 Bl * 6,515,593 Bl * 6,720,876 Bl 6,749,432 B2 6,765,726 B2 6,774,885 Bl 6,834,436 B2 * 6,876,496 B2
6/2002 Maynard ................. 356/141.1
5,385,519 A 5,405,152 A 5,423,554 A 5,429,140 A 5,466,200 A 5,469,740 A 5,474,083 A * 5,485,402 A * 5,495,576 A 5,516,105 A 5,524,637 A 5,577,981 A 5,580,249 A 5,584,700 A 5,587,937 A 5,591,104 A 5,597,309 A 5,616,078 A 5,638,300 A 5,641,288 A 5,645,077 A 5,656,904 A 5,659,691 A 5,702,323 A 5,703,623 A 5,704,837 A 5,711,304 A * 5,715,834 A 5,720,619 A 5,759,044 A 5,785,630 A 5,785,631 A 5,790,076 A 5,790,124 A 5,792,031 A 5,812,257 A 5,838,816 A 5,846,086 A 5,850,201 A 5,872,438 A 5,888,172 A 5,890,995 A 5,913,727 A 5,929,782 A * 5,963,891 A 5,989,157 A 6,004,243 A 6,028,593 A 6,043,873 A 6,050,822 A 6,050,963 A 6,054,951 A 6,066,075 A 6,073,489 A 6,077,201 A 6,088,091 A 6,098,458 A 6,100,896 A 6,119,516 A * 6,132,337 A 6,152,856 A 6,162,191 A 6,164,973 A 6,181,343 Bl 6,183,259 Bl 6,198,528 Bl 6,244,987 Bl 6,308,565 Bl 6,346,045 B2 6,361,507 Bl 6,366,272 Bl
111995 Hsu et al. ..................... 482/57 411995 Katanics et al ............. 273/438 611995 Davis ......................... 273/437 711995 Burdea ....................... 600/587
1111995 Uerich .......................... 482/4 1111995 French et al ............. 73/379.04 1211995 Church et al. .............. 600/546
111996 Smith et al. ................ 702/160 211996 Ritchey ...................... 395/125 511996 Eisenbrey et al. ...... 273/148 B 611996 Erickson ..................... 1281779
1111996 Jarvik ........................... 482/4 1211996 Jacobsen et al. .............. 434/11 1211996 Feldnian et al ............. 434/247 1211996 Massie ....................... 364/578
111997 Andrus .......................... 48217 111997 Riess ......................... 434/258 411997 Oh ................................ 463/8 611997 Johnson ................. 364/551.01 611997 Zaenglein, Jr. ... ... ... ..... 434/21 711997 Foxlin ........................ 1281774 811997 Lander .................. 318/568.12 811997 Durward et al. ............ 395/329
1211997 Poulton ......................... 482/8 1211997 Hall ........................... 345/158
111998 Iwasaki et al ................ 463/38 111998 Dower ....................... 600/523 211998 Bergamasco et al. ....... 1281782 211998 Fisslinger ................... 434/336 611998 Redmond ................... 434/307 711998 Bobick .......................... 482/4 711998 Heidecke ....................... 482/5 811998 Sypniewski ................. 342/365 811998 Fischer et al. .............. 345/435 811998 Alton ... ... ..... ... ... ... ...... 482178 911998 Teitel et al .............. 356/141.4
1111998 Holmberg ................... 382/157 1211998 Bizzi et al. ................. 434/247 1211998 Lasko-Harvill ................ 345/8 211999 Roston .................. 318/568.11 311999 Andras et al. .. ... ... ... ... ... 48217 411999 Bobick .......................... 482/4 611999 Ahdoot. ... ..... ... ... ... ...... 463/38 711999 Stark et al. ............ 340/870.01
1011999 Walker et al. .............. 702/150 1111999 Walton .......................... 482/4 1211999 Ewart ............................ 482/8 212000 Rosenberg ................... 434/11 3/2000 Ramer.. ... ..... ... ... ... 356/139.03 4/2000 Faughnn ...................... 434/11 4/2000 Johnson et al. ............. 600/595 4/2000 Sypniewski ................. 342/465 5/2000 Poulton ......................... 482/8 6/2000 French et al ............. 73/379.04 612000 Cheng ......................... 482/57 7/2000 Ramer ..................... 356/141.5 8/2000 French et al ............. 73/379.04 812000 Strohecker et al. ......... 345/427 912000 Hock ...................... 73/379.01
1012000 Krupka .......................... 482/8 1112000 Studor ........................... 482/8 1212000 Foxlin ........................ 600/595 1212000 Macri ......................... 434/247
112001 Lyons ........................ 345/358 212001 Macri ......................... 434/247 3/2001 Maynard ................. 356/141.1 6/2001 Ohsuga ......................... 482/4
10/2001 French et al ............. 73/379.04 212002 Rider ... ... ..... ... ... ... ...... 463/31 3/2002 Foxlin ........................ 600/595 4/2002 Rosenberg .................. 345/156
8/2002 French et al. ............ 73/379.04 12/2002 Hock ...................... 73/379.01 212003 Stark et al. ............ 340/870.07 4/2004 Burgess ................... 340/568.1 6/2004 French et al. ............... 434/247 7/2004 French et al. ............... 359/630 8/2004 Even-Zohar ................ 345/156
1212004 Townsend et al. ............ 33/512 412005 French et al. ............... 359/630
200210183961 Al 12/2002 French et al. ............... 702/150
OTHER PUBLICATIONS
Europe is Bursting with Virtual Reality Ideas, But Developers Are Critically Strapped for Cash, webpages from https:llwww/lexis. comlresearch/retrieve? _m~66dI7057c 1 b77fl97 aledb9f5 fadb87 d &_browseType~Text, (Jan. 1993) 1-2. Allard, P., et al, Three-Dimensional Analysis of Human Movement, Human Kinetics (1995) 3, 8-14. Brownstein, B., et al, Functional Movement in Orthopedic and Sports Physical Therapy, Churchill Livingstone (1997), 15. Brugger, W., et ai, Computer-aided tracking of body motions using a c.c.d.-image sensor, Med. BioI. Eng. & Comput, (Mar. 1978), 207-210. Codella, c., et ai, Interactive Simulation in a Multi-Person Virtual World ACM (May 3-7, 1992), 329-334. DeLoura, M., et ai, Game Programming Gems, Charles River Media, (2000) 200-204. Greenleaf, W.J., DataGlove, DataSuit, and virtual reality Advanced technology for people with disabilities, Proceedings of the Seventh Annual Conference 'Technology and Persons with Disabilities,' (Mar. 1992) 211-214. Kasvand, T., et al, Computers and the Kinesiology of Gait, Comput. BioI. Med. Pergamon Press (1976) vol. 6 111-120. Kenmochi, A., et al, A network virtual reality skiing system-system overview and skiing movement estimation, Symbiosis of Human and Artifact, (Jul. 1995) 423-428. Kraus, A., Matrices for Engineers, Hemisphere Publishing Corp. (1987) 118-120, 124-126. Lengyel, E., Mathematics for 3D Game Programming & Computer Graphics, Charles River Media (2004) 76-78, 467-468. Medved, v., Towards a virtual reality-assisted movement diagnostics-an outline, Robotica (Jan.-Feb. 1994) vol. 12, 55-57. Mulder, A., Human movement tracking technology, School of Kinesiology, Simon Fraser University (Jul. 1994) 1-14. Ruby, D., Biomechanics-how computers extend athletic performance to the body's far limits, Popular Science (Jan. 1982) 58-60. Sandweiss, J., et al, Biofeedback and Sports Science, Plenum Press New York (1985) 1-201. Scarborough, E.L., Enhancement of Audio Localization Cue Synthesis by Adding Environmental and Visual Cues, Air Force Inst. Of Tech., Wright-Patterson AFB, OH School of Engineering (Dec. 1992) 1-4. Smith, J., et ai, Virtual Batting Cage and Human Model, Virtual Human http://www.cs.berkeley.edu/rcdavis/classes/cs294/, (Jun. 17, 2004)1-5. Zetu, D., et al, Extended range tracking for remote virtual realityaided facility management, Department of Mechanical Engineering The University of Illinois at Chicago, http://alpha.me.uic.eduldan/ NsfPaper/nsfl.htrnl, (Apr. 19, 2004)1-9. Codamotion: The science of real-time motion capture and analysis, webpages from http://www.charndyn.comlindex.htrnl. (Apr. 17, 2004) 1. Irex, Virtual Reality Technologies, webpages from http://www. irexonline.comlhow_it_works.htrn, (Apr. 19, 2004) 1-2. Polhemus, PATRIOT: The Fast and Affordable Digital Tracker, www.polhemus.com. (Feb. 2004) 1-2. Polhemus, LIBERTY: The Forerunner in Electromagnetic Tracking Technology, www.polhemus.com. (May 2003) 1-2. Success Story Profile: Innovative Sports Training, Motion Monitor, (2002) 1-2.
* cited by examiner
JX-001.0003
A000003
Case: 12-1252 Document: 46 Page: 93 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 1 of 10 US 7,292,151 B2
10
---------) y FIG-1A
y FIG-1B
JX-001.0004
A000004
Case: 12-1252 Document: 46 Page: 94 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 2 of 10 US 7,292,151 B2
(-l \----- ) 1----..... 1 1 1 1 1 1 1 1 1 1 1 ~ ) ..... _-",
FIG-2A ~ I
FIG-2B
1 kg I , ! I \
FIG-2C
FIG-2D
JX-001.0005
A000005
Case: 12-1252 Document: 46 Page: 95 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 3 of 10 US 7,292,151 B2
T •
Evaluate msplay PIN Entry I Security & Navigation
Requirements Controls
+[reqUiredl " ",
Request ---) CI) Authentication rn 0 .c a...
[not required] User Session b [identification] 'c :::J
Authenticatef _____ J 0 CI)
(/) User
+ User Session
[loaded]
.. Configure Display D Session UI Graphical
• Incons of Name & ID
1 Descriptor
-,,' ,-"
Display Graphical ~ Request ,,-"
Incons of Workload Program ---" Intensity & Limits, Selection ..
Measurement User Session Criteria, Stimulus [setup] rh Properties, ect.
I Process i I I
Program Display D CI) / ...---_./ rn I Request Graphical 0
Request I .c
a... Incons of Session ---"" " Parameters 0.. " Setup ~ " :::J
Request " & Instruction ...- " CI) (/) Program Text
User Session ---" Setup ..
[setup] rh User Session
Process I [setup] rh Session +-----"
J
Parameters Process l
l Program ...---_./
Parameters
J + +
[A or B]
( Deployment Phase ) FIG-3A
JX-001.0006
A000006
Case: 12-1252 Document: 46 Page: 96 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 4 of 10 US 7,292,151 B2
CI) CI) o
.J:: 0... ..... c ..... CI) 0 E S ~
"'i5.. CI)
Cl
( Deployment Phase) i
User Session
[save] rh
Process 1M Deployment
Requirements
--------, Program
[load]
-------,
Evaluate Secondary 1M Requirement Display Graphical
Incon for Detail of Modular Length,
Weighted, or Augmentative Force
Resistance Attachments
[not required]
.. / _.c._-. l 1
/ i / i
/ i r-~~----~I i
Fasten Primary 1M
i i
+ Program
[Deployment] I i i i i
, i \ i \.. J --r
\ Display
Graphical Incon of
User Body
[ d] Location & rea y Strap Holster
~echan;sm
[required]
Fasten Secondary 1M
[ready]
--- ...... i ..
Program
[Deployment]
i i i i i i i
\. i \ .. I --T-'
'\.
Display Graphical Incon of
User Body Location &
Strap Holster Mechanism
Calibration Phase FIG-3B
JX-001.0007
A000007
Case: 12-1252 Document: 46 Page: 97 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 5 of 10 US 7,292,151 B2
Q) ff) c
..c c..
Request
( Calibration Phase)
Process Requirements of
1M Calibration
Display GraphicalD. Incon of
Primary 1M Calibration Status
Determine Secondary 1M Requirement
~[ not requ;red]
[required] Display GraphicalD.
.---. Primary 1M Pose :B Modification
---" I I ? se~noc~;a;f 1M
",----+< Calibration Status
c E o c:
'--' Process
Primary 1M Pose
Modulate Aural/Visual/Tactile
Stimuli rh
Evaluate Primary 1M
1 Program
[Calibration]
1 1 1 J
----'
.---. ..c u ..... c E o c:
'--'
.-;"
r----=-......... --:--"'" ;/;'" Request
Secondary 1M Pose Modification
Process Secondary 1M
Pose
Modulate Aural/Visual/Tactile
Stimuli rh
---"" 1 I
t Program
[Calibration]
I I 1
---)
Pose tatch .... (_ ... _"'_'"_,, ........ __ ,..
'-----<0 Match Pose to D. Intial Reference
Movement Trajectory
Evaluate Secondary 1M
pose~MatCh P--,..."_"_"_"..;" ____ _
'----__ --< Match Pose to D. [match] Intial Field
'--__ ~~ ..... ___ .....,~---[m-t.:.t-c-h] Position
( Execution Phase) FIG-3C
JX-001.0008
A000008
Case: 12-1252 Document: 46 Page: 98 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 6 of 10 US 7,292,151 B2
,,---[Iearn]------..
Query Leam
Modality
[normal]
---, , Program [Record]
User Session ---'" [Setup] I
I Q) I I In I • c Process .s::. I c.. Primary 1M Program I:: Pose [Execution] 0
:;:::; :::l I 0 Process I Q)
I x Learn w I
Terminus I I I I
[not done] I J ---
[done]
Process Program Terminus
Program [not done]
[Save] [done] • )
FIG-3D Data Analysis Phase
JX-001.0009
A000009
Case: 12-1252 Document: 46 Page: 99 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 7 of 10
---------L-~ _~---
--=---J-~) ---=--~==---] ,-----------~
/ /
/ .,-..---,..'"
I /
/ I
I
I
I I
I I
FIG-4B
US 7,292,151 B2
JX-001.0010
A000010
Case: 12-1252 Document: 46 Page: 100 Filed: 08/29/2012
u.s. Patent Nov. 6,2007
Remote Position Processor
Analog Signal Processor Interface
Receiver Constellation
Sensor &: «access»
Preamplifier
I J
r---~
r.t, Amplifer &:
BW Filter (I I
«a..:c:~~_....l
Comparator
Absolute Value
«access»
~ tI.I tI.I CD o o c !II. r---'"'I
Peak Detect &:
Sample-Hold ....
I
I I I I I I I
«access» I I~-
«access»
~-------------------
L..-U_S_B_2_.0_J ..... ----- --
FIG-5
Communication Interface
L...-___ ---I
Sheet 8 of 10
Digital Signal Processor Interface
Processor
AID
US 7,292,151 B2
TImebase
Conversion
r.t, ~--I 2 I .s I
<II I L..----r--..,---:-r
r.t, Phase-Locked
Loop
TIming Capture &: PWM
I ..... _- .....
J I
\ I I I
------./ I
~ <II <II CD o o c ~
I I I
r.t, Software
Calculation Engine
I I I
r----
(
I I I
------'"
r.t,
Digital Filter
r------------I «1M status» I I
_./
20
Radio Link
30
Parallel/Serial
I/o
I --_./
I J
---
JX-001.0011
A000011
Case: 12-1252 Document: 46 Page: 101 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 9 of 10 US 7,292,151 B2
14
FIG-6A
FIG-6B
FIG-6e
JX-001.0012
A000012
Case: 12-1252 Document: 46 Page: 102 Filed: 08/29/2012
u.s. Patent Nov. 6,2007 Sheet 10 of 10 US 7,292,151 B2
Transponder
Stimuli Interface Processor I r===n.
I "k ~ ~ r.L, 28MHz
r-l White LED ~ & Driver e .... D Timebase
r.L, ..... Vibrator 8 .c::::::;L «clock»,t Motor ~ ~ r.L, & Driver l) )t. PWM
r r-l
r.L, «access>~ Audio «configure»
Annuciator f----'
r===n.l ~ "" & Driver r===il T
r.L, II> r.L, CD u AID u Software
,.-- c Calculation Conversion ~"
Sensor Interface - Engine
=jJ ~""f;9""'J, r..
r===il <)g. r.L, E
Ultrasonic :::;; ------- ~ Transmitter r.L,
J Parallel/Serial
~ess»T «access» I/o "
r-l T r.L, ~ ':-r.L, Accelerometer ~( :::l
/ Heart Rate 01 t;::
Receiver ,.. <:
"', 0 u ,.... , ¥
T I Dual Axis I
low g I I I I
I I
Communication Interface I Polar
compatible ~ uS~»T ~ r----' ? «uLr I/O» ----,
r.L, r.L, Graphic LCD
~ r- Switch I/O & Touch Screen n ( )
r-l «1M msg»
20-...." r.L,
~'\
FIG-7 Radio Link
30 (1)-----------_ _ 900MHz ISM
JX-001.0013
A000013
Case: 12-1252 Document: 46 Page: 103 Filed: 08/29/2012
US 7,292,151 B2 1
HUMAN MOVEMENT MEASUREMENT SYSTEM
This application claims the benefit of u.s. Provisional Application No. 60/592,092, filed luI. 29, 2004, which is hereby incorporated by reference in its entirety.
BACKGROUND AND SUMMER OF THE INVENTION
This invention relates to a system and methods for setup and measuring the position and orientation (pose) of transponders. More specifically, for training the user to manipulate the pose of the transponders through a movement trajectory, while guided by interactive and sensory feedback means, for the purposes of functional movement assessment for exercise, and physical medicine and rehabilitation.
Known are commercial tracking and display systems that employ either singularly, or a hybrid fusion thereof, mechanical, inertial, acoustical or electromagnetic radiation sensors to determine a mobile object's position and orientation, referred to collectively as pose.
The various commercial tracking systems are broadly classified by their relative or absolute position tracking capability, in which system the pose of a mobile object is measured relative to a fixed coordinate system associated with either combination of receiver(s) or passive or active transmitter( s) housing mounted on the user. The tracking system's components may be tethered with obvious inherent movement restrictions, or use wireless communication means to remotely transmit and process the information and allow for greater mobility and range of movement.
Typically these tracking systems are utilized for biomechanics and gait analysis, motion capture, or performance animation and require the sensors to be precisely mounted on the joints. Various means of presenting the tracking information in a visual display are employed, such as Heads-Up Display (HUD) , that provide occluded or seethrough visibility of the physical world, or Fixed-Surface Display (FSD), such as computer desktop monitors, depending upon the simulation and immersive quality required for the application. The application may require various degrees of aural, visual, and tactile simulation fidelity and construct direct or composite camera views of the augmented or three dimensional (3D) virtual reality environment to elicit interactive user locomotion and/or object manipulation to enhance the user's performance and perception therein. The tracked object may be represented in the virtual environment in various forms, i.e., as a fully articulated anthropoid or depicted as a less complex graphical primitive. The rendering strategy employed depends upon the degree of photo realism required with consideration to its computational cost and the application's proprioception requirements.
Tracking technologies possess certain inherent strengths and limitations dependent upon technology, human factors, and environment that need consideration when discussing their performance metrics. Regardless of differentiating resolution and accuracy performance benchmarks, many implementations suffer from varying degrees of static and dynamic errors, including spatial distortion, jitter, stability, latency, or overshoot from prediction algorithms. Some human factors include perceptual stability and task performance transparency, which are more subjective in nature. And environmental issues such as line-of-sight, sensor attachment, range, and multiple-object recognition, need to be considered when selecting the optimal technology for the most robust application development. Irrespective of the
2 intrinsic strengths and weaknesses of the tracking technology employed, ultimately the user's satisfaction with the system's utilization and efficacy, including the production of reliable, easily understood, measurable outcomes, will dictate the overall success of the device.
This invention's system and methods facilitates biomechanical tracking and analysis of functional movement. In the preferred embodiment, this invention is low cost, robust,
10 easy to deploy, noninvasive, unobtrusive, and conveys intuitive and succinct information to the user to execute movement properly and provides performance indicators of said movement for feedback purposes. One feature of the present invention provides for an interactive tracking system
15 because the sensor functionality, or referred to herein as active transponders or transponders, is integrated with local user input control, and real-time sensory interfaces on the same device. The transponder is a wireless communication and monitoring device that receives a specific signal and
20 automatically responds with a specific reply. In one embodiment, the invention provides functional movement assessment based upon the relative measures of limb pose with respect to two positions defined by the transponders. The transponders can operate independently or work in unison to
25 process and share computational tasks and information between the local databases. This decentralized, distributed processing scheme allows the configuration and coordination of the training session, and processing and analysis of the measurements to occur without requiring expensive
30 auxiliary computer and display systems to manage the same, and without relying on costly software development of complex synthetic environments for visualization purposes. Also, the user can manage the applications and performance databases off-line on a remote computer system with Inter-
35 net connectivity to customize and configure the system parameters in advance of their session.
The present invention is designed to provide such system and methods for high-fidelity tracking or registration of the poses of active transponders and engage the user to pur-
40 posely manipulate the transponders' pose along a prescribed or choreographed movement trajectory in order to train and assess functional movement capability. In the preferred embodiment, the system is comprised of two subsystems: (1) a subsystem comprised of one or more active transpon-
45 ders, which, in its most sophisticated implementation, responds to periodic requests from another component of the system to radiate or transmit a signal for purposes of absolute position tracking; processes an embedded inertial sensor for relative orientation tracking and absolute tracking
50 refinement; and provides an essentially real-time aural, visual, and tactile sensory interfaces to the user, and (2) a subsystem comprised of a centralized position processor system or unit and receiver constellation unit, collectively referred to as the processor unit, which is essentially a signal
55 processor that synchronizes the transponders' periodicity of radiating signal and other operational states; collectively receives and processes the radiated signal; iteratively calculates the transponders instantaneous pose and convolution, thereof; and continually exchanges this information,
60 and its analysis thereof, with the transponders and/or auxiliary host computer system in essentially real-time via a combined wireless and tethered communication means. This real-time bidirectional exchange of information allows for proper transponder identification, coordination, and the
65 accurate measurement of pose, thereof, and timely actuation of the sensory interfaces for optimal user regulated closedloop control.
JX-001.0014
A000014
Case: 12-1252 Document: 46 Page: 104 Filed: 08/29/2012
US 7,292,151 B2 3
The transponder is broadly classified by its level of hardware and software configuration that define its scope of intelligence, sensory support, and configuration. The degree
4 of movement control or smoothness. In summary, one embodiment of the present invention is comprised of:
of intelligence is detennined by its capability to locally access, process, and modifY the database. Further, either transponder classification can be sub-classified by its manipulative requirements. In one embodiment, where multiple transponders are used, a principle transponder is consciously and deliberately moved along the reference movement trajectory, while a subordinate transponder serves as an 10
anchor or secondary reference point elsewhere on the locomotion system whose kinematics are not necessarily controlled by the user's volition.
1) a means to create a single movement vector whose endpoints are defined by the locations of at least two transponders, wherein, the expansion and contraction of the vector's length is calculated, analyzed, and reported in essentially real-time;
2) a means to create a single movement vector whose endpoints are defined by the locations of two transponders, wherein, a representative point along the vector length is referenced and its higher-order derivatives are computed by mathematical numerical processes, wherein the result is calculated, analyzed, and reported in essentially real-time; and,
An interactive transponder, preferably, has significant intelligence; supports relative and absolute tracking capabilities; provides complete sensory stimuli support; provides for functional enhancement through attachment of modular, extension pieces; and provides a user display and input system to control the training session. In the preferred embodiment, the interactive transponder is primarily held in the hand to facilitate more complex user input and greater sensory intimacy. Conversely, in another embodiment, the fixed transponder has limited intelligence; supports only the absolute pose tracking capability; provides no sensory stimuli support; and is usually mounted to a fixed site on the limb or trunk.
A combination of transponder deployment strategies may be required depending on the training session's objectives, such as two interactive transponders grasped by each hand; or alternatively, an interactive transponder, and a fixed transponder attached to the limb or trunk; or lastly, two fixed transponders attached to the limb(s) and/or trunk.
In one embodiment, this invention proposes to elicit movement strategies based on the deployment of at least two transponders that define the endpoints of a movement vector whose relative translation and rotation is measured and evaluated for the assessment of functional movement capability, including but not limited to, limb range of motion and its control thereof, limb strength conditioning, and overall proprioception and hand-eye coordination skills, and overall body movement. This registration system measures a single movement vector whose endpoints are comprised of an anchor point, i.e. one that is located in a less dynamic frame
15 3) a means to correlate said vector's length and at least one other measure consisting of a higher-order derivative, to the reference movement trajectory, wherein the result is calculated, analyzed, and reported in essentially real-time. A registration system for practical functional movement
20 applications should clearly convey information to the user regarding his movement quality while he perfonns the task, without compromising or distracting from said execution by uunecessary head movements or change in eye gaze and normal focus. Poor visualization strategies that distract the
25 user are ineffectual for promoting heads-up, immersive interaction, and the alphanumerical information it imparts often can not be consciously processed fast enough to elicit corrective action. This system provides for both a local, standalone sensory interface as a primary feedback aid, or
30 alternatively, an interface to a remote fixed-surface display for greater visualization and simulation capabilities. The visual stimulus could be modulated to warn of range violations, or provide signals for purposes of movement cadence and directional cueing. A principle interactive transponder is
35 typically hand-held, which is naturally in close proximity to the user's aural and visual sensory field during most upper extremity movements, or, conversely, the visual stimulus may be viewed through a mirrored or reflective means if not in optimal line-of-sight. A remote fixed-surface display
40 might augment the immersive quality of the user's experience by providing control of a view camera of a simulated computer environment, and display of the transponders and/or interactive objects' static or dynamic poses within the computer display's skewed through-the-window perspective
45 projection. In summary, one embodiment of the present invention is comprised of:
of reference, e.g., such as the trunk or abdomen, and another more distal location fixed on or held by a limb or extremity, e.g., the hand, ann, or leg. As this movement vector is translated and rotated through space by the act of the user modifYing the pose of the principle transponder in concert with the reference movement trajectory, the vector's length 50
will expand and contract relative to the proximity of principle transponder with respect to the subordinate transponder. The vector's length conveys unique and explicit information regarding the user's movement efficiency and biomechanical leverage. For example, by attaching a fixed subordinate transponder at the hips and a fixed principle transponder on the upper ann, the biomechanics of the act of lifting a box or similar object can be elegantly qualified. If the user assumes a poor lifting technique, i.e. legs locked with the trunk severely flexed with head down and the arms 60
stretched out beyond the basis of support, the vector's length would consistently be measured longer than compared to a good lifting technique, i.e., legs bent at knees with the back straight, head gaze up, and arms close to body. Also, the measurement(s) of higher-order derivatives derived from 65
numerical mathematical processes of a reference point described by the vector would provide additional indication
1) a means for modulating an embedded luminescent display organized and oriented into a directional-aiding pattern, by varying its degree of intensity and color, or other physical characteristics, to provide a visual display stimulus. This sensory interface is excited at a rate, repetition, or pattern proportional to the pose error of the transponders' movement trajectory compared to the reference movement trajectory;
55 2) a means to view said visual display stimulus with the aid of a mirror(s) or other reflective means;
3) a means for the real-time projection of sound or speech commands through an audio device to provide warning, alann, instructional, and motivational aid, and/or additional cueing upon encroachment of static and dynamic limitlboundary conditions defined by the reference move-ment trajectory;
4) a means for real-time tactile feedback including, but not limited to, modulation of the rotational properties of a vibrator motor proportional to the pose error of the transponders' movement vector compared to the refer-ence movement trajectory;
JX-001.0015
A000015
Case: 12-1252 Document: 46 Page: 105 Filed: 08/29/2012
US 7,292,151 B2 5
5) a means for combining the excitation of said stimuli proportional to the pose error of the transponders' movement vector compared to the reference movement trajectory; and,
6) a means to coordinate the real-time, periodic parametric update and modulation of the stimuli imparted by the sensory interfaces within the transponders from a processing unit by means of a wireless communication linle This invention addresses the need for an intuitive, inter-
active method to instruct, create, and deliver a movement 10
trajectory command without necessarily relying on preprogrammed, regimented movement trajectories. The registration system can be configured via remote setup at the principle transponder to pre-record and choreograph a freeform movement trajectory of the principle transponder with 15
the intent of the user mimicking the same said path. This impromptu learning modality can expedite the session down time between different users and movement scenarios, and accommodate users' high anthropometric variability in range of movement. In summary, one embodiment of the 20
present invention is comprised of: 1) a means is to provide a movement trajectory learning
modality that allows the user to calibrate and create the desired endpoints, midpoints, and/or total reference movement trajectory through user progranliller entry of an 25
input device resident on the transponder; 2) a means to process and save a movement trajectory using
a computationally efficient Catmull-Rom spline algorithm or other similar path optimizing algorithms to create control points along key points of the movement trajec- 30
tory that define the optimally smoothest path intersecting the control points;
3) a means to provide database management by a processing unit via a wireless communication link or, alternatively, through user data entry of an input device resident on the 35
interactive transponder; and, 4) a means to access, edit, and store the program and/or
databases to nonvolatile memory operably coupled to the principle transponders for the purpose of automating the creation, delivery, storage, and processing of movement 40
trajectories. Customized user programs and databases would be downloaded from a central repository or relevant website in advance of the training session to the transponder from the user's home location via the Internet or other convenient locales having networked Internet 45
access, and transported to the systems remote physical location, and uploaded into the system's memory, and executed as the application program. This a priori process of remote selection, download, and transfer of programmatic content and database would minimize the user's 50
decision making and input during product utilization by offering only relevant and customized progranlilling material of their choosing targeted for their specific exercise, fitness, or rehabilitation goals. Performance data could be saved indefinitely in the database's nonvolatile 55
memory, until an upload process was performed through the said network so the database could be transferred to another location for purposes of, but not limited to, registration, processing, archival, and normative perfor-mance evaluation, etc. 60
An exemplary list of specific data structures contributing to or affecting the means for automating the creation, delivery, storage, and processing of movement trajectories described below may be stored within the non-volatile memory of the 65
transponder or position processor which may use highdensity serial FLASH, although other types of memory may
6 be used such as SmartMedia, Compact Flash, etc. Additionally, the memory device interface should not be limited to internal, but may include external media devices, such as USB FLASH Key or other portable media means, that may have inter-operability with other computerized devices. The data structures may include: Modulation & Feedback Thresholds/Triggers Properties
the aural, visual, tactile interfaces require threshold settings which determine their excitation or stimulation characteristics. These settings can be derived from previous performance data or defaults determined from normative data, or modified in real-time, by algorithmic methods including moving averages, standard deviations, interpolation based upon goal-oriented objectives, etc.
Normative Performance-performance data collected over a large population of users through controlled studies, that is distilled down into specific user categories based upon certain demographics that the user may compare and rank hislher results. This data may be initially embedded within the transponders or position processor non-volatile memory and may be augmented or modified auto mati -cally or by user volition when connected to the Internet.
Competitive Ranking-applications which have a predominate point goal-oriented purpose would allow access to a global ranking file archive accessed through the Internet or automatically via updated executive files. This ranking file would be created through an analysis of user participation and publishing of his/her results through Internet Web-based services.
Downloadable Executive Programs & Configurations-new software programs, including new features, enl13ncements, bug fixes, adjustments, etc., could be downloaded to the transponder through an Internet connection. Graphics images would be stored in compressed or uncompressed binary forms, i.e., bitmap, gif, jpeg, etc. This new programs could be transferred to any suitable computerized position processor unit located at a remote facility via the transponder's wireless link. Therefore, the user's transponder is the node that establishes the portable network capabilities of the system, not necessarily the computerized position processor.
Custom Menu Interfaces-specialized activities may require more advanced (or simplified) interfaces dependent upon the users' cognitive abilities and interactive specificity. This menu may include interactive queries or solicit information regarding the user's daily goals, subjective opinions or overall impression of the activity and ones performance which could be incorporated in the Motivation Index described below.
Report Generation Tools and Templates-XML, HTML or other authoring language used to create documents on the Web that would provide an interactive browser-based user interface to access additional performance data analysis and report generation tools and templates that may not be available or offered with the standard product.
Custom Performance Algorithms---certain application-specific performance analysis may require dynamically linked algorithms that process and calculate non-standard or specialized information, values, units, physical measurements, statistical results, predictive behaviors, filtering, numerical analysis including differentiation and integration, convolution and correlation, linear algebraic matrices operations to compute data pose and scaling transformation, and proprietary types. One example of a proprietary type is Motivation Index, a composite numerical value derived from a weighted average of statistical performance indicators and subjective user input includ-
JX-001.0016
A000016
Case: 12-1252 Document: 46 Page: 106 Filed: 08/29/2012
US 7,292,151 B2 7
ing relative scoring improvements, conformity to ROM pattern, lengthy activity access duration, high access rate, relative skill level improvement, daily goal achievement, etc., that could represent the overall level of enthusiasm and satisfaction, the user has for a particular activity.
Range of Motion (ROM) Pattern Generator-the ROM pattern requires some key control points to be captured along the desired trajectory and stored in order that the algorithm can calculate an optimally smooth path, in real-time, during the comparative analysis phase.
8 optimal tracking location based upon collectively maximizing the ultrasonic source's energy received at the transducer interface. This invention addresses the practicality and robustness of
the registration system when used in either indoor or outdoor environments, and especially when the tracking volnme likely contains potentially occluding objects, i.e., an uninvolved limbs or clothing, that become potential sources of competing, reflected paths. The preferred embodiment of the
10 registration system utilizes the time of flight (TOF) measurement of ultrasonic acoustic waves due to its immunity from interference from the visible and near-visible electromagnetic spectrum and its superior ability to overcome most
ROM Pattern Capture & Replay-the ROM pattern can be can saved to memory in real-time by discrete position samples versus time depending upon the resolution desired and memory limitations and later played back on 15
the transponder or remote display for analysis.
multi-path reflections problems by simple gated timing of the initial wave front. Upon command from the processor unit, the transponders produce a few cycles burst of ultra-
Activity Specific Attributes-includes Reps/Sets, Duration, Pause, Heart Rate Limits, intra-activity delay, level, point scalars, energy expenditure, task-oriented triggers, etc., and other parametric data that controls intensity, execution rate and scoring criteria for the activity.
Instructional Information-textual, graphical, or animationbased instruction, advice, coaching, activity description, diagramed transponder deployment and intra-device connectivity, etc. that facilitates the intuitiveness, understanding, and usage of the system. The form of instruction may include music files saved in various formats, including Wave, MP3 or other current or future audio data compression formats, and video files saved in MPEG or other current or future video data compression formats.
Real-time Data Management-proprietary data management protocols that reside above the communication driver layer that manage the real-time, synchronous and asynchronous exchange of data between transponder(s) and position processor. This would provide an essential real-time sharing of activity data, analysis, and feedback stimulus thresholds, or coordination of multiple transponder configurations, or for a collaboration of same or different user requirements to complete a similar activity objective. This invention addresses the need for adaptability of the
registration system to different movement measurement scenarios. In one embodiment, it utilizes a versatile, modular configuration and mounting of the transponders onto the user. The efficient deployment of the transducers between different users' and from task to task requires a universal mounting scheme to provide consistent localization and pose of the transponders at the desired measurement sites on user's body. Also, to compensate for the receivers' finite tracking volume when stationary, the receiver constellation unit may be mechanically modified to optimize its tracking properties by conveniently repositioning it in closer proximity to the expected transponders movement trajectories and line-of-sight, thereof. In summary, one embodiment of the present invention is comprised of:
1) a means to quickly and effIciently alter the location of the transponders using a fastening system designed to quickly attach and dispose various forms of transponder assemblies;
2) a means to augment the physical properties, i.e., weight and length, of the principle transponder with adjunct electromechanical components that provide variations in biomechanical leverage for isotonic and isometric utilization; and,
3) a means to allow the user to manually alter the geometry and pose of the receiver constellation unit to facilitate an
sonic energy and the transducers of the receiver constellation unit are stimulated and mechanically resonate accordingly, upon the wave front arrival. The processor unit's
20 analog signal processing circuits transform the mechanical energy into electrical signals that resemble tapered sinusoidal waveforms, which another electronic circuit triggers upon using an adaptive threshold technique which, in turn, the processor unit detects and calculates TOF timestamps
25 indicating the wave front arrival. In the preferred embodiment, the system overcomes the ultrasonic technology's intrinsic challenge of precisely triggering on same the waveform location and provides consistent unambiguous trigger detection by complementing the adaptive threshold tech-
30 nique with a software timestamp correction algorithm, which includes in part, a digital over-sampling and averaging timestamp algorithm, a relative timestamp correction scheme utilizing a predictive algorithm of higher-order Taylor series based derivatives, and an absolute timestamp
35 correction scheme that minimizes the range error based upon discrete biasing of timestamps.
Further, in the preferred embodiment, the processor unit utilizes the absolute and relative trigger timestamps in a multi-modal trilateration algorithm for the measurement of
40 three-dimensional (3D) translations and rotations of the transponders. The primary trilateration calculation is derived by an application of Pythagoream theorem involving a point position solution based-upon range measurements from at least three (3) points, versus the well-known triangulation
45 method which uses bearing angles of two cameras of known pose. Additionally, the system's main accuracy limitation is mostly affected by the temperature variability of outdoor environments and its influence on the speed of sound in air value. This algorithm mitigates this problem by mathemati-
50 cally computing the speed of sound every analysis period provided at least five (5) receivers and a transponder synchronizing means are utilized. If the integrity of the synchronizing signal is temporarily compromised, the system automatically employs a variation of the trilateration algo-
55 rithm that uses the last known speed of sound value. In the preferred embodiment, the maximum update rate,
and hence the major contributor to the latency of the position calculation, is determined by the typical acoustical reverberation, typically between 20 to 100 ms, encountered in an
60 indoor environment. Since the transponders are held or fixed on the user's body and, therefore, are mobile, the TOF measurements will experience an additional latency effect. A Kalman filter is used as a prediction/estimation strategy to minimize and compensate for the latency effect. The pre-
65 diction algorithm uses a higher-order Taylor series based derivatives and augmentative inertial sensor data. Its predictive refinement is dependent upon predefined models of
JX-001.0017
A000017
Case: 12-1252 Document: 46 Page: 107 Filed: 08/29/2012
US 7,292,151 B2 9
expected movement conditions. Because functional movement is episodic, having periods of stillness interspersed with bursts of motion activity, a multi-modal filtering strategy is preferably employed to handle the unpredictable jerkiness at the start of motion and relatively predictable, smooth motion afterwards. In summary, the preferred embodiment of the present invention is comprised of: 1) a means to detect the same carrier wave cycle of ultra
sonic energy using a software correction algorithm requiring multiple, consecutive TOF acquisitions as input for 10
the digital over-sampling and averaging algorithm, the calculation of a higher-order numerical differentiation of the past and current TOF information as input for the predictive algorithm of higher-order Taylor series based derivatives used for the relative TOF correction, and a 15
measurement of the intra-pulse time intervals of consecutive TOF acquisitions as input for the absolute TOF correction scheme that minimizes the range error based upon selective biasing of the TOFs;
2) a means to utilize a dual matrix formulation of the 20
trilateration algorithm, and a calculation strategy thereof, which decision is dependent upon the integrity of the system's communication link, synchronization condition, and the desired measurement accuracy; and,
3) a means to coordinate the information transfer between 25
transponders and the processor unit so that their contribution to the resultant movement vector calculation can be measured without intra-signal interference. These goals will be attained by such system and methods
that are comprised of the user's interaction described by the 30
following steps as set forth as the preferred embodiment: 1) Authenticate user access and open user session from a
local or remote database;
10 FIGS. 2A-2D illustrates example extension pieces for the
present invention; FIGS. 3A-3D illustrate one example of process flows for
the present invention; FIGS. 4A and 4B illustrate a sample application of the
present invention; FIG. 5 illustrates a block diagram of the remote process
ing system of the present invention; FIGS. 6A-6C illustrate example receiver configurations of
the present invention; and FIG. 7 illustrates a block diagram of the components of
one embodiment of the transponder of the present invention.
DETAILED DESCRIPTION OF INVENTION
The present invention provides a practical, versatile measurement tool for the assessment of the user's manipulation strategy of the transponder 10 or transponders along a reference movement trajectory. Moreover, the system and methods measure and analyze the kinematics of the relative translations and rotations of the limbs or extremities with respect to each other or to a more inertial reference location on or off body as the transponders are manipulated. This information provides useful insight on biomechanical demands and anthropometric factors that influence human movement efficiency and control. Although measurement performance metrics are important design criteria, it's equally important to provide intuitive and motivating program instruction and administration, and to provide comprehensive analysis and integration of the motion data in a form that is objective and easily interpreted. This system improves upon the practicality and user interactive aspects of setup, deployment, calibration, execution, feedback, and data interpretation of a tracking system designed for func-2) Setup user training session, i.e., workload limitations,
measurement criteria, and audio/visual/tactile stimuli; 35 tion human movement. 3) Select training program and configure its options; 4) Deploy the transponders as instructed to predefined
locations of users locomotion system to create at least one transponder movement vector;
5) Calibrate the transponder movement vector to establish its 40
reference pose; 6) Create a movement trajectory using learn mode, if
required; 7) Initiate the start of session; 8) Determine the instantaneous pose of transponder move- 45
ment vector relative to its reference pose from a periodic temporal iteration of this step;
9) Perform qualitative and quantitative statistical analysis of accumulated measured poses of the transponder movement vector relative to the pattern of instantaneous poses 50
defined by the reference movement trajectory; 10) Update the major transponders sensory interfaces to
modulate said system parameters in a periodic temporal iteration of this step;
11) End the session once program objectives have been 55
obtained; 12) Analyze the results by interacting with local and/or
remote databases;
Human movement is a response to external environmental forces which requires the accurate coordination of the distal segment( s) to compensate for these forces. Skillful coordination of human movement is dependent upon the cohesive interaction of multiple sensory systems, including visual, vestibular, with the musculoskeletal system. More specifi-cally, the challenges and goals of cognitive spatial mapping, (2) minimization of energy expenditure, (3) maintaining stability, (4) steering and accommodation strategies for various environments, (5) dynamic equilibrinm, (6) active propulsion and weight support, and (7) core locomotion pattern should be relationally considered to properly assess hnman movement. Therefore, it is preferable to engage the interaction of these sensory systems during a training session to promote the desired functional movement outcome. Because many movements persist for 400-500 ms, enough time is allowed for the initiation of the movement and for user correction based upon visual and kinesthetic information acquired during the time of the movement. However, the implemented means of visual feedback should be not be distracting or interfering with the task at hand. In the preferred embodiment, this system engages the sensory systems with non-distracting, intuitive, embedded aural, visual, and tactile stimuli which provide real-time indication 13) Provide numerical, graphical, and/or animated informa
tion indicating desired performance measurements. 60 of the principle transponder pose error with respect to the reference movement trajectory.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. lA illustrates one example of a deployment apparatus of the present invention;
FIG. IB illustrates one example of hand-held form for the transponder of the present invention;
In order to conduct a time efficient training session, this registration system attempts to minimize the encumbering experimental setup and calibration procedures characteristic
65 of more complex and higher cost motion analysis technology. These complementary systems serve important academic or clinical oriented research needs or for motion
JX-001.0018
A000018
Case: 12-1252 Document: 46 Page: 108 Filed: 08/29/2012
US 7,292,151 B2 11
capture for computer animation purposes and strive for highly accurate measurement of joint motion data in terms of angular displacement. Therefore, the integrity and reliability of their motion data is dependent upon proper sensor setup and calibration.
For instance, single axis goniometer-based systems usually require specially designed harnesses to hold the monitor and are firmly strapped or taped over the joint to avoid relative motion artifacts. Usually these devices are tethered and their fit, weight, and constraining mechanical linkages can impose limitations on the joint motion and cause discomfort for the user. Most optical or video-based systems require the placement of numerous active or passive markers over landmarks, such as the joints' center of rotation. These systems should guarantee sufficient environmental illumination and contrast between markers and background to function optimally. Also, these systems are severely affected by occluded markers that may disappear for long periods of time due to rotations and line-of-sight limitations. Other video-based systems do not use markers but require the assignment of the body's joints manually or through computerized automation during data analysis, making real-time analysis arduous and real-time feedback virtually impossible.
In the preferred embodiment, the system doesn't require complicated, time consuming sensor setup and calibration by virtue of it minimalist sensor requirements and uncomplicated sensor mounting. Instead, it requires only the deployment of a sensor on the body (in one embodiment a dual sensor group on a combination oflimb(s) and or trunk) and doesn't enforce stringent movement protocol, but encourages free-form, unrestrictive movement of the transponders.
The transponder's preferred deployment means, include either insertion into a universal strap and holster apparatus (FIG. lA) that secures on the user's limb, extremity, or trunk, including, but not limited to, the hip, ankle, knee, wrist, upper arm, neck, waist or an augmentative mechanical attachment to one or a combination of modular extension pieces shaped into a hand-held form (FIG. IB). A strap or torx-like clip and holster design provides a firm, yet light weight and comfortable mounting location away from areas that clothing and or uninvolved limbs may occlude.
The modular extension piece is either an instrumented sensory type designed to support alternative tactile stimulus device or alternative configurations of aural, visual, and tactile feedback types, or non-instrumented, weighted extension pieces as shown in FIGS. 2A-2D. All modular extension pieces may be of various physical dimensions and intrinsic weight, with a captive handle design that preferably requires zero grip strength to grasp. Alternatively, the modular extension piece may be coupled to the transponder through a fixed or flexible, segmented, articulated coupling to accommodate attachment of additional transponders and/ or other modular extension pieces. These components would quickly assemble to each other using a spherical snap joint or twist snap latch, or similar mechanism, to provide quick alteration of form and function when used for different movement trajectory scenarios.
In one embodiment, the weighted extension attachments (FIG. 2A) are offered in fixed gradations of one (1) kilogram increments or other convenient unit of measure and either be indicated as such with a numerical label, quantitative mark, or color-code feature, or combination thereof. For upper extremity evaluation, the weighted extension piece integrated into a zero-grip handle would enhance the improve-
12 ment of musculature strength of the limb, while not compromising the user's endurance with a potentially fatiguing hand grasp requirement.
In one embodiment (FIG. 2B), the tactile type provides force feedback functionality by controlling the rotational speed of an embedded vibrator motor in the shaft. Alternatively, the visual type (FIG. 2C) may be comprised of a series of light emitting diodes that could be uniformly
10 embedded along the length of the handle or transponder and their intensities variably controlled therein. It should be appreciated that a simple, economical mirrored or reflective surface placed in front of the user's visual field could provide sufficient real-time indication of the user's subjec-
15 tive conformity to the said movement trajectory while allowing non-distracting viewing of this visual stimulus. For example, a program that requires the user to reposition the principle interactive transponder through an arc-like movement trajectory in the midsagittal plane through out a range
20 of motion begiuning from the waist upwards until parallel to shoulder height. As the user performs the movement, the visual sensory interface could be proportionally excited if the user moves too quickly, or hesitates too long, or produces shaky or erratic episodic motions, or is beyond the pre-
25 scribed bounds of the movement arc. The light stimulus is easily viewed in the mirror and would indicate corrective action in his or her movement strategy, while appropriate aural commands may be issued simultaneously to encourage the same correction. Regardless of the sensory interface
30 type, its control and excitation properties will be determined by some statistical aspect of the user's conformity to and progression through the movement trajectory.
The hand-held transponder may include a modular extension piece with an embedded graphic display device and
35 associated input means to allow the user to setup, operate, provide visual feedback, and view performance results of the device usage without additional remote display means. More specifically, a software-controlled user interface could provide certain visual prompts in a menu oriented presentation,
40 to instruct the user on (1) device setup, i.e., aural, visual, and tactile feedback parameters, types of program start and termination cues, program intensity based on ratio of amount of repetitions, sets, and rest periods or categorical gradation of challenge, learn mode behavior, etc., (2) scrol-
45 lable program selection with brief descriptions including objective, desired measurement, i.e., range of motion, energy, accuracy, speed, etc., and instructive information, and (3) alphanumeric and/or graphical display of measured performance data and other biophysical data and its analysis
50 thereof, displayed in standard plotted forms including line, bar, and pie charts, etc. It is important to note that the user input process is intuitive and streamlined so as not to detract from the practicality and user friendliness of the system. Only relevant applications and its control thereof will be
55 sequestered from the database and presented to the user. In one embodiment, two or more transponders and exten
sion pieces, or combinations thereof, may be assembled at their endpoints with a universal spring coupling. The assembled device could be grasped in both hands and bent
60 in various rotational angles about the spring coupling's axis. Isotonic strength conditioning programs can be developed due to the force resistance feedback supplied by the spring. A multi-transponder assembly in the form of a flexible rod or staff could provide an indication of balance of upper
65 extremity strength and proprioceptive function dependent upon the angular closure rate and rotational imbalance and orientation deviation from initial starting position.
JX-001.0019
A000019
Case: 12-1252 Document: 46 Page: 109 Filed: 08/29/2012
US 7,292,151 B2 13
Additionally, in the preferred embodiment, the modular extension pieces have provisions for other attachable apparatus (FIG. 2D) that can augment the program's intensity or difficulty. For example, an eyelet is embedded in the end of the extension piece and is designed to attach an elastomeric band, such as the type manufactured by Theraband®. By securing the other open stirrup end of the band to the user's foot, isotonic strength conditioning programs can be developed due to the force resistance feedback supplied by the elastomeric band. Moving the transponder through a movement trajectory is now made more restrictive and challenging.
APPLICATION EXAMPLES
An example training session deploying a dual transponder group is now described that may be designed to improve the range of motion, strength, and coordination of shoulder abduction in a user. The training session would primarily serve as an exercise aid that provides essential feedback to the user so that he/she learns to progressively improve the manipulation of the transponder through the reference movement trajectory, while benefiting from increased shoulder range of motion and strength improvement.
In advance of the training session, a software application is operated from a host computer that provides a utility for baseline configuration and management of the system's and transponder's local databases, and/or access to other remote databases, and for the real-time interface to the data flow between the system's components. The application's navigation and selections are presented to the user through a typical graphical user interface like Microsoft Windows® XP operating system. A generalized step-wise procedure requires the administrator or user to (1) select the desired program and features from a menu screen list, and (2) to initiate a communication process that causes the program parameters to be transferred to the processor unit through a standard computer communication protocol, i.e. serial, USB, ethernet, etc., whereupon, (3) the information is subsequently processed and transferred into the transponders local memory via a wireless communication link, and, finally, (4) the transponder's software program accesses this database to manage the device utilization and configuration of the local display means. Alternatively, a Compact FLASH-based memory card, embedded serial FLASH, or a similar nonvolatile memory device provides the user an additional specialized database supporting remote data collection capabilities. This database would be preprogrammed in advance and the resultant performance data retained, even if the device's power is lost, or for extended unsupervised exercise sessions conducted remotely from the host computer system or when the host computer system is unattached or unavailable. After the session is completed, the user would be queried if the results are to be saved for later analysis or would automatically be saved, dependent upon device setup. This data could be retrieved at a later time when the system is once again attached to a host computer system, and the software utility could be commanded to upload the database.
Henceforth, the following procedural description refers to the activity dependencies diagrammed in FIGS. 3A-3D that the user would encounter while operating the system.
During the Security Phase (FIG. 3A), the user may be requested to provide a security authentication code for validation, which opens access to his/hers custom programs
14 limited to, workload intensity, measurement criteria, sensory interface properties, and reporting features. A program menu list would indicate name, ID, and a brief description, or alternatively, be represented by a detailed graphical icon. When the program is selected, other program-specific options can be setup.
During the Deployment Phase (FIG. 3B), and dependent upon the program's objectives, a suitable combination of transponder types will be mounted on the user's body as
10 instructed by the program. This example requires the assembly of a hand-held interactive transponder with graphical display, and a weighted extension piece coupled therein to be grasped by the hand on the same side as the affected shoulder. Another subordinate transponder 12 is placed into
15 a holster assembly strapped around the lower quadriceps on the same side. This setup is shown in FIG. 4.
During the Calibration Phase (FIG. 3C), a simple calibration procedure may be requested to evaluate transponder function and specific user range of motion constraints.
20 Typically, this information is determined beforehand and saved in the system's database. Also, practicality of this system is claimed for lack of extensive calibration requirements.
Dependent upon the program's options, a user-defined 25 movement trajectory may be created prior to program start
in lieu of executing the predefined version. The learn mode could be utilized to quickly choreograph free-form movement trajectories and save them into the transponder's non-volatile memory for later execution. The learn mode
30 would be accessed through the user interface and instruct the management of the control point assignment by pressing the push button switch at the appropriate junctures of movement discontinuity or, preferably, allowing automated assignment by the software. In the preferred embodiment, a computa-
35 tionally efficient Catmull-Rom spline is used to define a three dimensional (3D) curve that passes through all the control points along the movement trajectory path. Ifmanually interceding, the user is instructed to press the push button once at each major juncture in the movement trajec-
40 tory, but, preferably, for no more than a few locations, until the desired end of range of motion is reached as shown in FIG. 4B. Similarly, the return path may be similarly defined or he/she may elect to use the same forward path in reverse. These control points are registered by the processor unit and
45 transferred and saved to the transponders' memory to serve as the control points for the real-time calculation of a Catmull-Rom spline. The Catmull-Rom spline is calculated in real-time from the desired initial starting point to provide a continuous set of position points representing the
50 "learned" reference movement trajectory. After the program is selected or the learn mode complete,
the user may be instructed to alter the pose of the transponders to satisfY the initial starting conditions of the program. Either one or a combination of sensory interfaces could be
55 excited by the principle transponder to cause the user to direct or steer it towards the desired start point. For instance, the visual sensory interface could sequentially extinguish or dim its peripheral light sources to converge to a central light source as the principle transponder is positioned closer to the
60 desired start point. Alternatively, the aural sensory interface could change its tonality and loudness as the start point is approached. Or alternatively, the tactile sensory interface could be modulated to provide less force feedback as the start point is approached.
in the training session. Next, during the Setup Phase (FIG. 65
3A), the user can configure global options or select the desired program. The global options may include, but are not
During the Execution Phase (FIG. 3D), the transponders are continually manipulated along the reference movement trajectory to the best of the user's skill and fidelity, within
JX-001.0020
A000020
Case: 12-1252 Document: 46 Page: 110 Filed: 08/29/2012
US 7,292,151 B2 15
the bounds of the user's physical limitation, until an aural, visual, or tactile response is given that indicates the activity volume has been successfully completed or a sufficient number of confonnity violations or failures have been registered. The processor unit calculates the instantaneous pose coordinates of the transponders every analysis interval and periodically communicates this information to the transponders via the wireless communication linle As the principle transponder is moved in mimicry to the reference movement trajectory the confonnity error between the actual 10
and reference movement trajectory is calculated periodically
16 increased risk for falls, balance disorders can shorten attention span, disrupt normal sleep patterns, cause excessive fatigue, increase dependence on others and reduce quality of life. It is not uncommon for individuals with a history of balance problems to regain their balance control through accurate diagnosis followed by specific medical treatment and/or rehabilitation exercises.
The present invention described can be used as a testing and training device for balance improvement under both static and dynamic conditions.
One testing and training scenario for postural stability would be to measure frequency and amplitude of body sway in three dimension (3D) space while feet remain in a fixed position. This task can be performed in both a double or
in real-time to determine the characteristics of feedback quality to be elicited by the sensory interfaces for the user's closed-loop control to correct his/her manipulation strategy. For example, the conformity error may be calculated from statistical processes based upon the standard deviation of the least mean squared (LSM) principle transponder's position error compared to the reference movement trajectory, or based upon, or combination thereof, a threshold magnitude
15 single leg stance to test for bilateral symmetry relating to balance. Another modification of the test would be to
of some multi-order numerical differentiation of said move- 20
ment to indicate a "smoothness" quality of translation and rotation along the movement trajectory path.
perfonn each test with eyes both open and closed to help determine the contribution of the visual component to overall balance ability. Tracking body sway while creating the illusion of motion through proper visual cueing on a display means would be another test to help detennine the reliance on specific sensory components of balance. Delivering repetition of protocols with increasing difficult oscillation thresholds with biofeedback of successes and failures of
25 such predetermined goals is one way to train to improve balance.
Alternatively, a host computer system could provide an auxiliary processing and display means to allow another software program to access the transponder's calculated positional data through an application programmer's interface and use this data to alter the pose of a graphical primitive in proportion to the motions of the transponders within the context of computer generated virtual environment. The dynamic control of objects in the computer 30
generated virtual environment could be used to augment the local sensory interfaces of the transponders through an interactive, goal-oriented video game modality. The video game challenges could be increased over time based upon some scoring criteria of successful manipulation of the 35
principally controlled on-screen graphical object with respect to cueing derived from other secondary static or dynamically moving objects. It is important to note that only primitive fonns of video game challenges would be considered, to take into account the user's cognitive awareness and 40
physical limitations, and the economics of software development for photo realistic virtual environments and animation. Also, this auxiliary computer display means would offer an alternative visualization method of interactive and immersive video feedback aid to enhance the application 45
presentation. Additional examples of how the present invention may be
applied are described as follows:
Balance The body has the ability to maintain balance under both
static and dynamic conditions. In static conditions, such as
The transponder can deliver aural, visual, and tactile stimuli to queue the individual to the degree of frequency and amplitude of body oscillations. The aural and tactile components provide the only means of feedback when the testing and training are perfonned with eyes closed or the visual field is compromised. Examples include, but are not limited to, (1) an audio signal increasing and decreasing in volnme proportional to the amplitude of body sway, (2) a vibration action proportional to frequency of body oscillations, and (3) a light source illuminated when both frequency and amplitude goals are achieved. Multiple transponders can be used to evaluate and reinforce proper balance posture by communicating position information of certain body segments in relationship to others. An example would be the comparison of position of the head with respect to the hips while generating a vibration action if an excessive forward lean of the head as compared to the hips is recognized.
Another test for balance would be to test ones Limits of Stability (LOS). This test refers to ones ability to effectively operate within their sway envelope. The sway envelope or LOS is defined as the maximal angle a person's body can achieve from vertical without losing balance. An individual with healthy balance is capable ofleaning (swaying) within
50 a known sway envelope and recover back to a centered position without the need for a secondary maneuver such as a step, excessive bend at the torso or ann swinging. LOS for bilateral stance in nonnal adults is 8 degrees anterior, 4 in standing, the body strives to efficiently maintain posture
(often referred to as postural stability) with minimal movement. In dynamic conditions such as in walking or sports 55
play, the body strives to maintain balance while perfonning
degrees posterior and 8 degrees laterally in both directions. The present invention described can be used as a testing
and training device for balance control during movement or perturbations within a desired sway envelope. Through proper visual queuing represented on the display means that defines a normal sway envelope, the amount of body dis-
in an ever changing environment. The ability to maintain balance is a complex process that depends on three major sensory components. The sensory systems include visual, vestibular and proprioception. For example, we rely on our visual system (eyes) to tell us if the environment around us is moving or still; we rely on our vestibular system (inner ears) to tell us if we are upright or leaning, standing still or moving; and we rely on our proprioceptive system (feet and joints) to tell us if the surface we are standing on is uneven or moving. If balance problems develop, they can cause profound disruptions in your daily life. In addition to
60 placement can be measured from vertical stance. The transponder can deliver aural, visual, and tactile
stimuli to queue the individual as to when he or she has achieved the desired range of their sway envelope, then assess the individual's ability to return back to a vertical
65 stance. Examples include, but are not limited to, (1) a vibration action when the user varies (meanders) from the desired movement path, (2) an array of lights change inten-
JX-001.0021
A000021
Case: 12-1252 Document: 46 Page: 111 Filed: 08/29/2012
US 7,292,151 B2 17
sity and pattern as the individual successfully approaches the intended target, (3) an audio signal is generated when the individual has maintained a stable position with respect to proper visual queuing represented on the display means for a selected period of time. Multiple transponders can be used to evaluate and reinforce proper balance posture by communicating position information of certain body segments in relationship to others. An example would be the comparison of position of the head with respect to the hips while generating a vibration action if an excessive forward lean of the head as compared to the hips is recognized.
Dynamic balance can be evaluated while having the individual perform coordinated movements which specifically challenge the various components of balance in a dynamic nature. Such movements include, but are not limited to jumping, hopping, and walking. These movements can be performed with eyes both open and closed, during interaction with static or dynamic visual queuing on the display means. The ability to perform these dynamic balance tasks with comparisons to others of similar sex, age or disability can be assessed. Example measurements may include, but are not limited to, (1) amount of body sway in three dimension (3D) space, (2) time to complete specific task, and (3) effects of fatigue on balance ability.
Balance training in both static and dynamic conditions can be easily achieved by providing specific visual queuing
18 the display means to instruct and motivate the individual through the proper testing procedure.
The present invention described can be used as a testing and training device for individuals involved in physical rehabilitation or general fitness to improve ROM. Proper visual queuing can be represented on a display means to motivate individuals to extend their range of motion beyond their current capabilities.
The transponder can deliver aural, visual, and tactile 10 feedback that alerts the individual to successes or failures in
proper execution of each repetition. An example of tactile feedback would be the transponders are vibrated if the individual's movement trajectory varied from the intended two dimensional (2D) reference movement trajectory by
15 deviation from the planar path into the uninvolved spatial dimension. An array of light sources could increase illumination in intensity and repetition as the ROM goal was approached and an audio tone could signal the individual they have achieved the desired pause time at the proper
20 ROM. Multiple transponders can be deployed to determine the
contribution of each joint or anatomical structure where more then one joint is involved in the ROM movement (example; shoulder and scapular in overhead reaching). The
25 vector sum of each transponder movement in a specific axis can be added together to determine the total ROM. The ROM of one joint in a two joint motion can be subtracted from the total ROM to determine the contribution of a single on the display means, which challenge the individual to
perform repetitive and progressively more difficult balance drills. Performance reports can be generated to establish a 30
baseline, isolate specific strengths and weaknesses within the specific sensory and motor control aspects of balance, and document progression and improvements.
joint in a two joint movement.
Human Performance Testing and Training There are many devices that test the strength and speed of
isolated joint movements, for example, the leg extension and bicep curl. This information has value in testing both healthy The transponder can deliver aural, visual, and tactile
stimuli to queue the individual as to when he or she has achieved the desired balance task. By example, a vibration action is produced proportional to the frequency of a body segment oscillation after the user lands from a hop test and attempts to stabilize and maintain proper postural balance. When the individual finally stabilizes and achieves correct postural balance, an audio signal indicates the task has successfully completed. Multiple transponders can be used to evaluate and reinforce proper balance posture by communicating position information of certain body segments in relationship to others. An example would be the comparison of position of the head with respect to the hips while generating a vibration action if an excessive forward lean of the head as compared to the hips is recognized.
Range of Motion (ROM)
The present invention described can be used as a testing and training device to determine the range of motion within
35 individuals, athletes and individuals whose strength and speed capabilities may be compromised by injury, disease, poor conditioning or simply age. Recently in the field of human performance, it has been recognized that the analysis of the mobility of the isolated joint, although providing some
40 value, does not offer enough information to determine how the body will perform during functional movements. Functional movements are defined as movements equal to those encountered on the athletic field, in the work environment or while performing activities of daily living. Functional move-
45 ments involve the movement and coordination of multiple joints and muscle groups acting together to perform a more complex task then a single, isolated joint movement.
The present invention described can be used as a testing and training device for functional movement improvement.
50 By tracking various registration points on the body with respect to each other or to an off-body registration point, performance measurements of functional movements can be assessed, such as jumping, cutting, turning, bounding, hop-a joint. Range of Motion is the normal distance and direction
through which a joint can move. Limited ROM is a relative term indicating that a specific joint or body part cannot move 55
through its normal and full ROM. Motion may be limited by
ping, shuttling, etc. The present invention described can be used as a testing
and training device for individuals involved in physical rehabilitation, general fitness or sports performance enhancement to improve their functional movement abilities. Proper visual queuing can be represented on the display
a mechanical problem within the joint that prevents it from moving beyond a certain point, by swelling of tissue around the joint, by spasticity of the muscles, or by pain. Diseases that prevent a joint from fully extending, over time may produce contracture deformities, causing permanent inability to extend the joint beyond a certain fixed position.
The present invention described can be used to test the starting point and end point which an individual is capable of moving a body part, typically a limb and associated joint(s). Comparisons to age and sex based normative data can be made. Proper visual queuing can be represented on
60 means to instruct and motivate individuals to perform specific functional movements.
The transponder can deliver aural, visual, and tactile feedback of proper movement execution. Examples include, but are not limited to, (1) an audio signal alerting the user
65 that the desired performance stance is incorrect, (2) the light sources illuminate when the desired speed is achieved in a first step quickness drill, (3) a vibration action to indicate the
JX-001.0022
A000022
Case: 12-1252 Document: 46 Page: 112 Filed: 08/29/2012
US 7,292,151 B2 19
limits of tracking range, (4) a vibration action proportional to the magnitude of a biophysical measurement during the interaction with visual queues represented on the display means, (5) a vibration action when the body or limb position does not correlate well to the desired body or limb position of the visual queuing represented on the display means, (6)
20
an audio signal indicating start, stop and pause periods or other controlling events, (7) an audio signal indicating proper body alignment or posture has been compromised, and (8) an audio signal indicating the relationship of desired 10
target heart rate to a desired threshold.
oriented perpendicular to the horizontal plane. Further, as indicated in the preceding figures, the transducers vertical axes are oriented 90° with respect to the typical vertical axis orientation of the transponder's transmitter to improve acoustic coupling in the vertical plane, a consideration for overhead, upper extremity tracking. Although this causes some reduction in the lateral registration bounds, the compromise provides a more symmetric field about the middle or primary location of tracking interest.
In the preferred embodiment, the overall size of the receiver constellation unit is predicated on a phenomenon referred to as Geometric Dilution of Precision (GDOP). The solution of a unique three-dimensional location based upon trilateration requires the precise resolution of the common
Hardware Description In the preferred embodiment, the processor unit is com
prised principally of a constellation of five (5) ultrasonic transducers and signal processing circuitry, thereof, and a signal processor that interfaces to this receiver group, performs the pose calculations, and interfaces to the transponders and host computer databases. The following interface descriptions for the processor unit are based upon the dependency flow represented by FIG. 5.
The sensors 14 preferably used for the receiver constellation unit are cylindrically-shaped ultrasonic transducers, for example, the model US40KR-0l 40 kHz PVDF ultrasonic receivers manufactured by Measurement Specialties Inc., which provide adequate acoustic pressure sensitivity and exhibit 360 degree onmidirectional broad beam response along the horizontal plane. The onmidirectional characteristic, albeit in one plane only, is very desirable to minimize line-of-sight occlusion. Because of its low resonance Q value, the rising and decay times are much faster than conventional ceramic transmitters. This reduces its power requirements since less burst drive duration is needed to achieve sufficient triggering thresholds at the receiver. This transducer type is also utilized similarly in the transponders to provide the potential for the most optimal acoustic coupling.
The receiver constellation unit is preferably mounted on a fixed support base, and has a pivoting and/or swiveling mechanical linkage which provides an adjustable mechanism for configuration of the receiver constellation unit's inertial frame of reference relative to the tracking field. In the preferred embodiment, it is strategically positioned and oriented in proximity to the tracking field in order (1) to minimize line-of-sight degradation with respect to the expected transponder orientation, (2) to optimize registration resolution with respect to field volume size, and (3) to satisfY the mathematical restrictions of performing trilateration calculations based upon the solution of simultaneous linear equations. It should be noted that the trilateration matrices may be solved if the matrices have a rank of five, and are non-singular, i.e., the matrix detenninant is nonzero. In the preferred embodiment, the geometric parameters and their coordinate location of the receiver constellation must insure linear independence of the columns of the matrices and to avoid the matrices from becoming singular.
One example geometrical pennutation of the receiver constellation unit that satisfies these rules is shown in FIG. 6A. It occupies a volnme of approximately 8 cu. ft. and essentially fixes the transducers in a way that defines two primary orthogonal, bisecting planes defined by three noncollinear points each. Another preferred implementation that occupies nearly the same volume is shown in FIG. 6B and is characterized by its S-shaped curve and tilted with respect to the horizontal plane. Another preferred implementation that occupies nearly the same volume is shown in FIG. 6C and is characterized by its helical or logarithmic spiral shape
15 intersection of multiple spheres circumscribed by the distance between each transmitter and receiver transducer. Each sphere has an inexact radius due to system noise and measurement resolution. Therefore, the intersection becomes a volnme instead of a point and the size of the
20 volnme is dependent upon the radii of the intersecting spheres as well as the distance between the spheres' centers. As the radii get larger with respect to the distance between the centers, i.e., the transmitter is farther down range, the spheres begin to appear more and more tangential to one
25 another and the intersection volnme increases, although not necessarily symmetrical in all dimensions. Therefore, to minimize position uncertainty, the receiver transducers should be separated from each other as much as practical proportions allow with respect to the confines of the tracking
30 field volume as the above said geometric examples provide. This receiver constellation unit can be repositioned with
respect to the tracking field by a simple mechanical adjustment as shown in the preceding figures. The mechanical adjustment raises and lowers and changes the length and
35 pivot axis of the cantilever ann which is fixed to a ground base support.
Because the receiver constellation unit operates a distance from the processor unit, each receiver preferably has an associated pre-amplifier circuit to convert the high-input
40 impedance piezoelectric signal into a low-level voltage proportional to the acoustic signal energy impinging the transducers sufficient in order to accurately transmit the signals to the processor unit. In one embodiment, a highinput impedance AC amplifier design with 30 dB gain can be
45 utilized. The preferred operational amplifier is the OPA373 manufactured by Texas Instruments. It was chosen for its low 1 pA input bias current, high 6 MHz GBW, and low-voltage single supply operation. The amplifier is configured as a non-inverting type with the high-pass cutoff
50 frequency set at 1 kHz. The overall circuitry is preferably enclosed in a metal shield to minimize electromagnetic noise coupling into the highly sensitive amplifier inputs. In addition, a local, regulated power supply is included to allow for a wide range of input voltage supply and provide sufficient
55 power supply rejection to compensate for the noise susceptibility of remote power distribution. All the pre-amplifier circuits' power and signal counections preferably originate from the processor unit.
The processor unit subsystem preferably consists of an 60 analog signal processing interface that provides (1) addi
tional voltage amplification and filtering of base band signal from the preamplifiers, (2) absolute value function, (3) peak detection function, and (4) analog-to-digital comparator function to provide support for an adaptive threshold means.
65 The adaptive threshold technique provides robust triggering of the most proximal ultrasonic source at a precise temporal point along the traversing sinusoidal waveform of the elec-
JX-001.0023
A000023
Case: 12-1252 Document: 46 Page: 113 Filed: 08/29/2012
US 7,292,151 B2 21
trical signal. Essentially, a new threshold signal is recalculated each analysis period based upon a small percentage reduction of the last peak wavefonn detected. Therefore, the tracking range is not necessarily restricted due to an arbitrarily high threshold setting and the noise immunity is improved as the threshold tracks the wavefonn envelope and not transient disturbances. An alternative automatic gain control strategy for the amplification function is unnecessary since the trigger threshold will adjust to the signal level instead. In the preferred embodiment, the threshold faithfully tracks the peak to minimize integer period phase errors, so the amplifier's gain is set to prevent signal saturation from occurring when the receiver constellation unit and transponders are in closest proximity during normal use.
In one embodiment, an amplifier and BW (band width) filter circuit receives the output from the sensor and preamplifier circuit and provides additional amplification and low-pass filtering to condition it for reliable threshold triggering and input to other analog signal processing circuitry. A dual amplifier configuration may be used to provide an additional gain of 40 dB, AC coupling to remove DC offsets of the preamplifier outputs and long cable losses, and low-pass filter to reject noise beyond the interest signal's bandwidth. The first stage amplifier may be configured as a non-inverting type with a gain of20 dB. The low-impedance DC input signal is effectively blocked by the coupling capacitor in series at its non-inverting input with a high-pass frequency cutoff set at 20 kHz. This gain stage feeds a second amplifier configured as low-pass, 2nd order Butterworth MFB filter. This filter type provides smooth pass band response and reduced sensitivity to component tolerances. The second stage low-pass frequency cutoff is set at 80 kHz with a pass band gain of 20 dB.
An absolute value circuit receives the output of the amplifier and BW filter circuit and converts the bipolar signal into a unipolar form for magnitude detection. A dual amplifier configuration may be used to provide highly accurate full wave rectification of the millivolt-level signal. The first stage amplifier feedback switches to control the distribution of input current between the two signal paths dependent upon the input signal polarity. For a positive input voltage the input current will be positive which forward biases Dl and reverse biases D2. This configures the 1st
stage as an inverter driving the inverting input resistor of the 2nd stage, which is also configured as an inverter because its non-inverting input is held at virtual ground due to the non-conducting path of D2. This effectively creates a combined circuit of two cascaded inverters for an overall gain of +1. For a negative input signal its input current is negative which forward biases D2 and reverse biases Dl. This configures the 1st stage as an inverter driving the noninverting input of the 2nd stage which changes the sign of the circuit gain. In this mode, the input current is shared between two paths to the input of the 2nd stage, where _2/3 of the input current flows around the 1 st feedback stage and -l!3 flows in the opposite path around the 2nd stage feedback path for a net gain of -1.
In the preferred embodiment, a peak detect and samplehold circuit receives the output of the absolute value circuit and registers a peak value that is required to set a magnitude threshold precisely at some percentage of full-scale of the peak. A dual amplifier configuration may be used to provide the highest ratio of high output slew rate to low droop. The first stage is typically in negative saturation until the input voltage rises and exceeds the peak previously stored on the sample capacitor at the inverting input. Now the amplifier acts as a unity gain buffer and the input voltage charges the
22 sample capacitor which faithfully tracks the rising voltage. Once the input voltage diminishes in magnitude, the first blocking diode reverse biases and the sample capacitor holds an accurate replica of the highest voltage attained with minimal droop because of the low input bias current of the amplifier and elimination ofleakage altogether in the second blocking diode by bootstrapping its cathode at the same potential provided by the low-impedance buffer of the second output stage. An electronic switch and bleed resistor
10 allow the voltage across the sample capacitor to be reset by the processor during power up and after the triggering event is recorded so the adaptive threshold value can be refreshed each cycle. A 1 st order Butterworth filter may be used at the input to smooth false in-band transients that could disrupt
15 the peak accuracy detection. In the preferred embodiment, a comparator circuit
receives the output from the peak detect and sample-hold circuit to convert the analog signal to digital form for high-speed triggering operation of the processor. The pre-
20 ferred device is the MAX941 which is manufactured by Maxim. A percentage of the peak threshold is used to set the inverting input. When the non-inverting voltage exceeds the inverting voltage, the comparator's output will trip and produce a high-true logic pulse that triggers the processor. A
25 latch control input allows the processor to disable the comparator action to prevent urmecessary triggering during the reverberation phase and to prevent potentially disruptive noisy output chattering near threshold crossover beyond its hysteretic immunity. The percentage of threshold level is
30 predetermined through the scaling resistors to be set low enough to trigger on the rising edge of the signal's first crest at the furthest range of transponder operation, but high enough above the intrinsic system noise level and external noise caused by reverberation and other ultrasonic sources.
35 Once the first crest is registered, subsequent crests may be triggered at their zero-crossing representing the most precise timing registration by momentarily disabling the samplehold circuit. Because of the longer duration trigger receptivity window, early multiple reflections are mitigated by
40 transducer placement at least 3.5 em away from adjacent planar surfaces, so the reflected acoustic energy doesn't produce a canceling effect of the direct acoustic energy of the later crests. Once a sufficient number of crests have been registered, then the triggering window is blanked for the
45 remainder of the analysis period by latching the comparator's value.
In the preferred embodiment, a digital signal processing interface is connected to the analog signal processing interface to transfonn the analog trigger processing into digital
50 position information. The digital filter circuit receives output from the com
parator circuit and preferably consists of a digital low-pass filter implemented in a complex programmable logic device (CPLD) that serves to precondition the comparator circuit's
55 digital outputs. The preferred device is an AT1504ASVL CPLD which is manufactured by Atmel. Base band system noise or other glitches potentially occurring in the analog signal processor interface, but prior to the actually arrival of the ultrasonic signal, could cause a threshold disruption that
60 registers a "runt" pulse as a false trigger condition. The "runt" pulse would be misinterpreted as the actual TOF trigger and cause serious error in the position calculation. An ANDINOR one-hot state machine design may be used to ignore level transitions that are not stable for at least Ih
65 system clock frequencyx8 states, so only transitions of 4 flS or greater are passed through. The system clock delays introduced by the digital filter's synchronous state machine
JX-001.0024
A000024
Case: 12-1252 Document: 46 Page: 114 Filed: 08/29/2012
US 7,292,151 B2 23
affect all channels the same and are, therefore, effectively eliminated by the inherent dependency on relative measurement.
In the preferred embodiment, the processor and digital filter circuits receive the output from the analog processor and provide controlling signals therein. The preferred processor circuit is the MC9S08GB60 which is manufactured by Motorola Inc. It is a low-cost, high-performance 8-bit microcontroller device that provides all the aforementioned hardware circuits integrated into one convenient device. The calculation circuit is abstracted from embedded 60 KB FLASH for program memory with in-circuit programmable capability and 4 KB RAM for data memory. The time base circuit is preferably comprised of an external, high-noise immnnity, 4.0 MHz system clock, which multiplies this by the internal frequency-locked loop for a bus clock of 40.0 MHz and single instruction execution time of 25 llS. This clock also provides all the capture and control timing fnnctionality for the other specified circuits. Multiple parallel I/O ports and dedicated asynchronous serial communication signals provide for the digital control of the analog signal processing and commnnication interfaces, respectively.
The timing capture-control circuit receives the output from the digital filter circuit representing the arrival of the TOF triggers to detennine the relative TOF propagation of the ultrasonic acoustic wave as it passes through the receiver constellation nnit. More specifically, it is comprised of a five channel 16-bit timer input capture module with programmable interrupt control that provides edge detection and 50 llS timing precision to automatically register the TOF triggers timestamps asynchronously without using inefficient and less accurate software polling means.
24 ber and Product Description strings. A host computer may enumerate and access this device utilizing the manufacturer's virtual COM port device drivers using a USB channel.
In the preferred embodiment, the radio link circuit is comprised of a wireless bidirectional communication interface to preferably (1) broadcast a synchronization signal to control the transponders interoperability, (2) to receive other transponder sensor data, including, but not limited to, accelerometer, heart rate, battery, user I/O status, (3) to provide
10 control messages for the transponders' sensory interfaces, and (4) to provide means to configure transponders' local databases. The preferred wireless communication link is based upon the AT86RF211, a highly integrated, low-power FSK transceiver optimized for license-free ISM band opera-
15 tions from 400 MHz to 950 MHz. and manufactured by Atmel. It supports data rates up to 64 kbps with data clock recovery and no Manchester Encoding required. The device has a three wire microprocessor interface that allows access of read/write registers to setup the frequency selection,
20 transmission mode, power output, etc. or get infonnation about parameters such as battery, PLL lock state, etc. In normal mode, any data entering its input channel is immediately radiated or any desired signal collected by the aerial is demodulated and transferred to the microprocessor as
25 reshaped register bit information. In wake-up mode, the device periodically scans for an expected message sequence and broadcasts an interrupt if a correct message is detected.
In the preferred embodiment, at least three (3) consecutive TOF timestamps are registered for each receiver during the
30 acquisition phase. Preferably, the transponder's transducer emits a multi-cycle ultrasonic acoustic burst of at least ten cycles in duration so that sufficient energization of the receiver transducer is realized and at least three crests of the
The phase-locked loop circuit receives the output from the timing capture-control circuit and is preferably comprised of 35
a three channel, 16-bit timer compare module is implemented as an all-digital phase locked loop (ADPLL), which synchronizes the capture window and blanking functions with respect to the reference input channel. It is comprised primarily of a free-running 16-bit timer configured to peri- 40
odically interrupt the processor dependent upon a precise convergence of its period and phase to the reference trigger source, by means of an over/under count matching and correction technique.
waveform can be properly registered. At low signal levels when ultrasonic acoustic coupling is poor, this requirement may fail and an invalid tracking status is asserted. Prefer-ably, the reference receiver transducer of the receiver constellation unit is positioned in closest proximity to the acoustic signal source so that it is the first transducer to be affected by the initial wave front. This reference receiver provides the overall system timing and state machine control for the phase-locked loop circuit, so that the processing, calculation, and commnnication tasks are executed in a deterministic and efficient fashion.
The AID conversion circuit receives the output from the 45
amplifier and BW filter circuit and consists of an eight channel lO-bit analog-to-digital converter used to monitor channel offsets and magnitudes for range and polarity errors and correction. This information is utilized by the calculation circuit as input to the TOF software correction algo- 50
rithm to detennine the slope of the wavefonn crest.
It should be appreciated that a high-resolution ultrasonic acoustic tracking system that depends upon threshold detection means has an inherent nncertain trigger dilemma. This uncertainty arises because of the multi-cycle nature of the transmitted signal's waveform and the associated difficulty detecting the exact temporal location for consecutive analysis periods when the signal's magnitude may vary greatly
In the preferred embodiment, the serial communication circuit is comprised of two asynchronous serial communication interfaces that are connected between the calculation circuit and host link and radio link circuits of the communication interface. The host link provides a 115K bit per second (baud) bi-directional communication link to an auxiliary host computer system through a Serial-to-Universal Serial Bus bridge. The preferred device is the CP2101 which is manufactured by Silicon Laboratories. It supports the conversion of a fully asynchronous serial data bus protocol, with buffering and handshaking support, to an integrated Universal Serial Bus (USB) Function Controller and Transceiver and internal clock providing USB 2.0 full-speed compliancy. An integrated 512 bit EEPROM stores the required USB device descriptors, including the Vendor ID, Product ID, Serial Number, Power Descriptor, Release num-
depending upon the efficiency of the acoustic coupling, the distance between transmitter and receiver, and signal-tonoise ratio of the signal processing techniques. If a threshold
55 is set near one of the minor crests of the wavefonn during the last analysis period, then it is conceivable a slight reduction of magnitude of the waveform during the next analysis period may fall slightly below the set threshold and actually not be triggered nntil the next larger excursion of
60 the wavefonn occurs. This would create a TOF error proportional to the period of the acoustic waveform or its intra-pulse interval and have a detrimental affect on the measurement accuracy. This analog processing described above establishes trigger thresholds that allow no more than
65 a single intra-pulse interval of nncertainty, but that is still inadequate for high-resolution measurements. Although a technique is known that controls the largest peak profile of
JX-001.0025
A000025
Case: 12-1252 Document: 46 Page: 115 Filed: 08/29/2012
US 7,292,151 B2 25
the transmitter acoustic signal and claims to provide an absolute trigger condition, this procedure is difficult to reliably tune and control among different transducer types.
26
In the preferred embodiment of the invention, no modulation of the acoustic signal is required. Rather, the adaptive threshold method is augmented with a TOF software correction algorithm that unambiguously detennines the correct TOF based upon a means to detect the same carrier wave cycle of ultrasonic energy every period. The software correction algorithm requires multiple, consecutive TOF acquisitions as input for the digital over-sampling and averaging algorithm, the calculation of a higher-order numerical differentiation of the past and current TOF information as input for the predictive algorithm of higher-order Taylor series based derivatives used for the relative TOF correction, and a measurement of the intra-pulse time intervals of consecutive TOF acquisitions as input for the absolute TOF correction scheme that minimizes the range error based upon selective biasing of the TOFs.
correction algorithm. The initial condition that precedes the start of the relative compensation algorithm may be due to the resumption of a stable, locked tracking state after recovery from a fault condition and, therefore, requires computation of a set of reference TOFs producing minimum range error as a starting basis. The algorithm utilizes a wireless synchronization means to determine a reference TOF calculation between the transponder and reference sensor of the receiver constellation. By computing the reference range
10 distance by the product of the reference TOF and speed of sound in air, this reference range may be compared to the range calculated from the matrices solutions described below. By iteratively and sequential increasing and decreas-
15 ing the TOFs by a single intra-pulse time interval and applying the input to matrices formulations described below, all possible combinations of compensation are permutated and tested, which produces a unique set of TOFs that minimize the error between the calculated range distance
The calculation circuit preferably processes multiple, consecutive TOF acquisitions to effectively improve the timing resolution that proportionally affects position accuracy and precision. The digital filter discussed above introduces quantization errors because of its discrete operation. And minor fluctuations in the acoustical coupling produces timing jitter or uncertainty in the triggered zero-crossings of the acoustic sinusoidal. A Gaussian average or mean value
20 with respect to the reference range distance. This unique set of initial TOFs serves as the starting basis for the relative compensation algorithm. In the preferred, embodiment, this absolute compensation algorithm works most effectively when (1) the wireless synchronization means is tightly
of multiple TOF is a simple and effective filter strategy. Due
25 coupled to the excitation of the acoustic source, (2) the synchronizing signal's arrival is timed by the same mechanism that times the arrival of the reference transducer's acoustic signal, and (3) the coordinate locations of the
to the possibility of poor acoustic coupling or misalignment, and distant transponder location from the processor unit, the 30
number of detectable triggered zero-crossings may vary for a fixed duration of multi -cycle ultrasonic acoustic burst. The averaging algorithm automatically adjusts to this condition by only including TOFs whose delta changes fall within the expected range of the nominal intra-pulse interval defined by 35
the transmission properties of the acoustic source. The nominal intra-pulse interval is detennined and utilized by the following compensation schemes.
The calculation circuit preferably processes a relative TOF correction algorithm based upon a predictive tuned 40
algorithm that requires higher-order numerical differentiation calculation of the past and current TOFs. This compensates the TOFs that may have registered one intra-pulse interval earlier or later than the nominally expected time due to the trigger dilemma described above. By fonnulating 45
these derivatives into a truncated 2nd order Taylor series expansion and weighting the terms contribution, an estimate of expected TOF is calculated and compared to the actual TOF through an iterative error minimization calculation. A minimized error that results in a delta time change indicative 50
of a discrete intra-pulse interval increase or decrease due to an early or late TOF, respectively, produces a characteristic value that directs the algorithm to compensate the actual TOF by the intra-pulse interval and restore it to its correct value. In the preferred embodiment, this relative compen- 55
sation algorithm works most effectively when, (1) the maximally expected inter-period TOF change is less than the discrete intra-pulse interval, (2) the TOF inter-period processing is contiguous, (3) the TOF increase or decrease is no more than a single intra-pulse interval, and (3) the Taylor 60
series tenns are suitably weighted in the prediction algorithm.
The calculation circuit preferably processes an absolute TOF correction algorithm at least once initially, when the phase-locked loop is stable, but may be perfonned every 65
analysis period depending on computational resources, that determines the initial set of TOF values for the relative
sensors of the receiver constellation are established to a high degree of accuracy.
The calculation circuit preferably employs two software methods of trilateration calculation to estimate transponder position, wherein the particular method used depends upon the availability of a synchronizing signal and the accuracy desired. The first method is based on a relative TOF calculation and the speed of sound is treated as a constant estimated at ambient indoor room temperature. The second method requires calculation of an additional TOF timestamp between the transponder and reference receiver, but calculates the speed of sound as an unknown every analysis period, and thus improves measurement accuracy. The first method eliminates the global system timing variances and delays due to the multiplicity of signal conditioning circuitry and eliminates the need for a controlling signal means synchronized at the generation of the transmission of the ultrasonic acoustic wave. The second method also employs relative TOF calculation but requires an additional synchronization signal from the processor unit to determine the absolute TOF between transponder and reference receiver. Since the absolute TOF is based upon a single chaunel only, its timing latencies can be readily accounted for and easily corrected. This method computes the speed of sound every analysis period, provided the synchronization signal is detected, without need for additional hardware temperature processing or requiring more then five (5) receivers, and automatically accounts for the system's main accuracy limitation of speed of sound in air as defined by Eq.1.1, if uncorrected, yields a 1.6 mmlm ranging error for every 10 C. temperature shift. If the synchronization signal is not detected and, therefore, the second method is not resolvable, the last calculated speed of sound can be utilized within the first method's calculation to minimize error.
c=34.6 rn/s+0.5813 rn/s(Tc--25c C.) (1.1)
The TOF timestamps and speed of sound values are input into linear independent algebraic equations in a matrix
JX-001.0026
A000026
Case: 12-1252 Document: 46 Page: 116 Filed: 08/29/2012
US 7,292,151 B2 27
formulation to solve for the unknown transponder(s) position, in a form as shown in Eq.2.1,
A·X =B (2.1) 5
all al2 al3 a14
A= a2l a22 a23 a24
a3l a32 a33 a34
a4l a42 a43 a44 10
Xl
X= X2
X3
X4 15
bl
b2 B=
28 Since each receiver is fixed at a distance Di from the
transponder as determined by the receiver constellation geometry and because the acoustic waves propagate spherically, by using Pythagorean's theorem the following set of range equations are defined in Eqs.5.1-5,
(x l-U )2+(y l-V)2+(Z 1-W)2 =D12 (5.1)
(x2-u )2+(Y2-V)2+(Z2-W)2 =D22 (5.2)
(xru)2+(yrv)2+(zrw)2=D32 (5.3)
(X4-U)2+(y4-V)2+(Z4-W)2=Dl (5.4)
(XS-U)2+(yS-V)2+(ZS-W)2=Ds2 (5.5)
Equivocally, the four (4) non-reference receivers are preferably located at an incremental distance relative to the reference receiver, so by substitution of the incremental distance defined by Eq.6.1, the following set ofrelativistic
b3
b4 20 range equations are defined by Eqs.6.2-5,
To solve for the unknowns X, Eq.2.1 is rearranged as shown in Eq.3.1, whereas the inverse of A requires computation of the cofactor matrix AC for the adjoint and determi- 25
nant calculations for Eq.3.2 and Eq.3.3, respectively,
X=A-l.B=(A'l.B IAI
(3.1)
30
(6.1)
(6.2)
(6.3)
(6.4)
(XS-U)2+(yS-V)2+(ZS-W)2~(Dl+CLl.TlS)2 (6.5)
By expanding and rearranging the terms of Eqs.6.2-5, a set All A2l
(A'l = Al2 A22
Al3 A23
A3l
A32
A33
A4l
A42
A43
(3.2) of four linear algebraic equations and four unknowns for the first method algorithm, depicted in the matrix form of Eq.2.1, is defined by Eq.7.1,
A14 A24 A34 A44 35
(3.3)
To setup the coefficient matrix A, the utilization of five (5) 40
receivers produces the following set of relative TOF equations defined by EqsA.1-4,
(4.1)
(4.2) 45
(4.3)
(4.4)
The receiver locations are fixed within the system's inertial reference frame, while the transponder(s) are mobile with respect to the same reference frame and are defined as follows,
S(xo,yo,zo)==S(u,v,w)~unknown transponder location
50
55
Xl -X2 Yl - Y2 Zl - Z2
2 Xl -X3 Yl - Y3 Zl -Z3
Xl -X4 Yl - Y4 Zl - Z4
Xl -Xs Yl - Ys Zl -Zs
-cLl.Tl2
-cLl.Tl3
-cLl.T14
-cLl.TlS
U (cLl.Tl2)2 + Ri - R~
(cLl.Tl3)2 + Ri - R~
W (cLl.T14)2 + Ri - R~
where Rf = xf + yf + zf for 5 ;:: i ;:: 1
(7.1)
(7.2)
Alternatively, if the second method algorithm is used, the unknown range of the reference receiver Dl can be substituted by Eq.8.1,
Dl ~cToj, Ll.Tol~time of flight (TOF) from S(u,v,w) to S(xjo Yl' Zl) (8.1)
And, by rearranging terms, it is depicted in the matrix form defined by Eq.9.1,
Xl -X2 Yl - Y2 Zl -Z2 -(Ll.TOlLl.Tl2 + o.SLl.TI2) u Ri - R~ (9.1)
Xl -X3 Yl - Y3 Zl -Z3 -(Ll.TOlLl.Tl3 + o.SLl.TI3) Ri - R~ 2
Xl -X4 Yl - Y4 Zl -Z4 -(Ll.TOlLl.T14 + o.SLl.Ti4) W Ri - R~
Xl -Xs Yl - Ys Zl -Zs -(Ll.TOlLl.TlS + o.SLl.Tls) c2 Ri - R~
JX-001.0027
A000027
Case: 12-1252 Document: 46 Page: 117 Filed: 08/29/2012
US 7,292,151 B2 29
Although similar results may be obtained by application of more computational efficient processes such as pivotal condensation or Crout's decomposition, the application of Cramer's rule was used to evaluate the first-order determinant in Eq.3.3 using second-order determinants from Laplace expansion. The final transponder(s) position equations are defined by Eqs.1O.1-S.
30 By examination of the matrices element equivalency of
Eqs.l1.2-3 and manipulation of terms so that the angles may be found using the inverse tangent function, the following rotation equations Eqs.12.1-3 are derived,
(12.1)
(10.1) 10 (12.2)
(10.2) 4' X3 sin8,sin8y ) Y3 -
8 cos8, - 1 z = at X3
(12.3)
(10.3) 15 cosBz_1 == cos()z from previous iteration
(10.4) 8, = atan(~) for 1st iteration
(12.4)
(12.5)
(10.5) 20 These calculations are performed through iterative step
processes which inherit angular approximations of the preceding steps until the final desired angular accuracy is achieved by assuming the conditions of Eqs.12A-S. There-
(10.6)
(10.7)
(10.8) 25 fore the rotation 8z ' roll, is first approximated by Eq.12.S; then the rotation 8x ' pitch, is approximated by Eq.12.1; and then the final rotation 8y , yaw or tum, is approximated by Eq.12.2. The next approximation of 8z utilizes the previous value of 8z in Eq.12.3 and the similar steps are preferably
If the first method is used, D, the range of the transponder to the reference receiver from Eq.lOA may be calculated as a redundant confirmation of the Eqs.1O.1-3 calculations, provided the frame of reference origin and location of the reference receiver are identical or their offsets accounted for. If the second method is used, C, the speed of sound in air, from Eq.lOA must be computed every analysis period if its value is anticipated to be used in the first method in the absence of a synchronization signal.
The orientation of the transponders can be derived from a similar utilization of the above algorithms for a transponder configured with a triad of ultrasonic transmitters. The transducers are preferably arranged in a triangular plane at the transponder of sufficient area for the desired angular resolution. The sequential excitation of each transducer and subsequent calculation of position by the aforementioned methods provides suffIcient information to determine orientation by the inverse kinematic calculations of Eqs.l1.1-4, where the analysis is simplified by assuming the origin of rotations occurs about Tl and Tl23 represents the initial relative position matrix from this origin and Tl23 is the transformed or forward kinematic position matrix.
[
0 X2COS()y + Z2smBy
Rx(8)Ry(8)T123" 0 s1ll8x(X2S1118y -Z2cos8y)
o -cOS8x(X2S1118y - Z2cos8y)
X3cosB
y [ X3 smBxsmBy
-x3cosBxsmBy
30 repeated until the desired accuracy is achieved. The transcendental functions may be evaluated through a conventional look-up table or by a power series expansion.
Preferably, the overall analysis period duration is effectively trebled until the three (3) transducers' positions are
35 calculated, which reduces the system's frequency response and imposes an increased latency effect. Typically, robust absolute orientation processing requires more stringent lineof-sight operation and is reserved for more sensitive, less dynamic, and reduced ROM movement trajectories, e.g.,
40 balance and sway. Therefore, the latency effect is less noticeable upon the real-time performance of the sensory interfaces.
In the preferred embodiment, the interactive hand-held transponders support a dual axis inertial sensor, which is
45 operably configured to provide tilt (pitch and roll) orientation in its horizontal mounting plane. The inertial sensor is mounted in the intended operational horizontal plane with respect to the systems inertial frame of reference. Once the sensors signals has been converted to an acceleration value
(11.1)
(11.2)
r
Xl COS()z + Yl sinBz X2 COS()z + Y2 sinBz X3 COS()z + Y3 sinBz r Rz({)r1 TI23 == -Xl sinBz + Yl cosBz -x2sinBz + Y2COS()z -x3sinBz + Y3cos Bz
~ ~ ~
(11.3)
.: Xl = YI = Zl = 0 A Y2 = 0 A Y3 = Z3 = 0 for initial orientation (11.4)
JX-001.0028
A000028
Case: 12-1252 Document: 46 Page: 118 Filed: 08/29/2012
US 7,292,151 B2 31
that varies between +/-1 g the tilt in degrees is calculated as shown in Eqs.13.1-2, for pitch and roll, respectively.
<I>~a sin(A)1 g) (13.1)
<I>~a sin(Ajl g) (13.2)
This outside-in ultrasonic tracking implementation, where the transponders are mounted on the mobile object, produces inherent temporal delays due to the finite TOF registration and calculation delays after the transponder has already moved into a different position before the measurement is complete. This overall latency period is compensated and minimized through use of a Kalman filter data processing algorithm to estimate the pose of the transponder by optimally and recursively combining past history, new measurements, and a priori models and information. Generally speaking, the Kalman filter is a digital filter with timevarying gains that are optimally determined through a stochastic dynamical model of the motion. The overall goal is to minimize filter lag while providing suffIcient smoothing of the motion data.
An adaptive, multi dynamic model is developed based upon the kinematic quality of the expected movement trajectory. The predictive kinematic model for the Kalman filter is depicted in matrix form utilizing a truncated 2nd order Taylor series expansion as below in Eqs.14.1-2,
[:L =[~ ~ttl +[:l (14.1)
[1 0 1: : o~:" 11:[.[:[ (14.2)
The Kalman filter is now described for a single dimension, although it is utilized for prediction and smoothing for all position dimensions. The predictor stages consist of the calculation of the state and the error covariance projection equations. The state projector equation, EqlS.l, utilizes a discrete time-sampled difference equation of r calculated from Eq.1S.2. In other words, the numerically derived velocity and acceleration components of motion are linearly combined with the previously a priori position to estimate the new position. The corrector stages consist of sequential computation of the gain, updated state estimate, and updated error covariance equations. The a posteriori state estimate, Eq.1S.4, is based on a linear combination of the weighted measurement residual and the last state estimate.
(15.1)
(15.2)
(15.3)
(15.4)
(15.5)
32 defined as a small constant and based upon the actual static timing variance empirically measured. The smaller this value the more confidence there exists in the systems' measurement capability.
In the preferred embodiment, the product of the numerically-derived 1st and 2nd order derivatives of the measured position scaled by a frequency dependent gain provides a computationally practical adaptive dynamic process noise estimate model. The derivative product term increases Qk
10 proportionally for higher velocity and acceleration components of motion, e.g., quick, abrupt directional changes, which effectively increases the gain and, therefore, means more confidence exists in the measurement rather than the estimate. This provides faithful, low-latency response to
15 high-frequency motions. Conversely, the frequency scaling term decreases the predictive "overshoot" characteristic of lower power, repetitive motion, e.g. slower, cyclic, ROM trajectories, which effectively decreases the gain and, therefore, means more confidence exists in the estimate rather
20 than the measurement. It should be appreciated this filter implementation provides superior tracking fidelity and comparable smoothing characteristics as compared to practical lengths of finite impulse response running-average filters and various low-orders infinite impulse response filters. It
25 achieves enough predictive response to compensate for the inherent TOF and computation latencies, while providing and comparable smoothing properties of other filter types.
30
QK==IKq[ (Zk_1-Zk_3)(Zk_1- 2Zk_2 +Z 1_3)sin(zk_1-zk_3)]1 (16.1)
(16.2)
In the preferred embodiment, a three dimensional (3D) piecewise cubic curve interpolates a movement trajectory for smoothing and reduced sample storage for greater
35 memory effIciency. Preferably, four (4) sequential discrete control points of the n-length set of control points, the sample resolution dependent upon the desired movement granularity, and corresponding timestamp are needed to calculate in real-time the interpolated position between any
40 pair of control points. A Catmull-Rom spline algorithm is the preferred method in that the path intersects the control points and would best approximate a movement that may have acute directional changes. The Catmull-Rom spline algorithm is defined by Eqs.17.1-3, where the geometry matrix
45 Gk represents the matrix of three dimensional (3D) control points.
50
55
60
0 -1 2 -1
1 2 0 -5 3 Ck(J.l) = G"i
0 4 -3
0 0 -1
-O.SJ.l + J.l2 - O.SJ.l3
1 - 2.SJ.l2 + l.SJ.l3 Ck(J.l)=Gk
O.SJ.l + 2J.l2 - l.SJ.l3
-O.SJ.l2 + O.SJ.l3
(17.1)
J.l
J.l2
J.l3
(17.2)
(17.3)
The new error covariance projector, Eq.1S.2, is it's previously computed value combined with the current process noise covariance, Qk' which is tuned by an example model 65
derived from the measured motion dynamics shown in Eq.16.1. The gain's measurement noise covariance, Rio is
The fl value is normalized and represents the % value between the 2nd 3rd control points. To calculate the interpolated value between the 1 st and 2nd or the n-l th and nth control points, the value of first control point of the pair and the value of the last control point pair are doubly entered into the
JX-001.0029
A000029
Case: 12-1252 Document: 46 Page: 119 Filed: 08/29/2012
US 7,292,151 B2 33 34
associated drivers. The preferred LED device is a CMD87 manufactured by Chicago Miniature Lamp. These LEDs' intensity is controlled by a white LED driver. The preferred white LED driver device is a MAX1570 manufactured by Maxim. The white LED driver provides a maximum 120 rnA constant current source to each LED for optimal uniform luminescence. The drive current can be proportionally regulated through external pulse width modulation (PWM) means from the processor circuit to modulate its brightness
geometry matrix, respectively. The appropriate dflldt is determined by the desired rate of playback of movement trajectory. To playback at the same rate as the recorded session, and assuming fairly constant velocity, a timestamp should also be saved at each control point registration so that the fl calculation is correctly scaled by the delta time interval. The n-Iength set of control points would be manually registered by the user pressing a switch or automatically post processed by a sorting method where a control point is registered at the tangents of the trajectory having sufficient magnitude and/or experience sign changes which indicates discontinuous or non-monotonic movement.
The major functional interfaces of the transponder unit preferably include the sensory interface, transducer interface, processor, and communication interface. The following descriptions of the transponder unit are based upon the dependence flow represented by FIG. 6.
10 level. Additionally, an electronic switch is connected in series to each LED drive to individually control its active state. By simultaneously controlling the PWM duty cycle and active state of each LED, the light strobe can appear to smoothly migrate along the linear array in spite of its
15 discontinuous operation.
The sensor interface refers to the collective support for the ultrasonic transmitter, heart rate receiver, and accelerometer circuits. The ultrasonic transmitter circuit is preferably gated by a pulse-width modulated (PWM) digital signal at nominally 0.8% duty cycle of the 40 kHz resonant frequency, e.g., a single 250 fls pulse every analysis period, by the processor circuit. The radiated ultrasonic signal strength is controlled by gating a MOSFET transistor switch at a duty cycle which optimally energizes the transducer's series resonant tank circuit for sufficient duration. The resonant circuit's reactive components include an impedance matching inductor, the transducer's intrinsic capacitance, and a small damping resistive load. At resonance, a electrical damped sinusoidal 30
with a potential up to -400 V pk_pk is developed across the transducer to sufficiently drive it at acoustical power levels practical for the system's intended range of operation. Enabling a lower duty cycle control through means of a software algorithm monitoring the transponders range 35
would effectively lower the transponders power consumption and radiate less ultrasonic acoustic energy for close range operation when signal saturation and clipping is undesirable. Conversely, a higher duty cycle control would radiate greater ultrasonic energy to compensate for less 40
efficient, non-optimal acoustical coupling orientations of the transponder with respect to the receiver constellation. Optionally, two additional transducers may be driven in unison or sequentially from a different transponder assembly
Preferably, the stimuli interface circuit provides the primary aural stimulus by means of a 4 kHz piezo buzzer. The preferred device is SMT-3303-G manufactured by Projects Unlimited. This electromechanical buzzer requires an exter-
20 nal transistor drive circuit and digital control signal gated at a rate near its resonant frequency. The buzzer inputs are connected to and controlled by PWM means from the processor circuit to provide a gross volume adjustment which is dependent upon the amplitude of the drive signal.
25
to support measurement of absolute rotation about a single 45
or multiple axes, or provide calculated positional redundancy for certain diffIcult line-of-sight applications.
The heart rate receiver circuit wirelessly receives a 5 kHz heart rate signal from a Polar® transmitter belt. The transmitter, worn around the chest, electrically detects the heart 50
beat and starts transmitting a pulse corresponding to each heart beat. The receiver captures the signal and generates a corresponding digital pulse which is received by the timing capture-control circuit of the processor interface. A software algorithm processes the signal with known time-based aver- 55
aging and an adaptive window filter techniques to remove any extraneous artifact or corruption caused by interfering sources.
Additionally, the stimuli interface circuit provides the primary tactile stimulus by means of a vibrator motor. The driver for the vibrator motor enables a 120 rnA DC current source to excite the motor armature. The preferred driver device is the MAX1748 manufactured by Maxim. The rotational speed of the motor's armature is controlled by PWM means from the processor circuit.
The processor circuit preferably receives input from the stimuli interface, sensor interface, and the communication interface and provides controlling signals therein. The preferred processor circuit is the MC9S08GB60 which is manufactured by Motorola Inc. It is a low-cost, highperformance 8-bit microcontroller device that integrates the specialized hardware circuits into one convenient device. The software calculation engine circuit operates from an embedded 60 KB FLASH for program memory with in-circuit progranJillable capability and 4 KB RAM for data memory. The time base circuit is preferably comprised of an external, high-noise immunity, 4.0 MHz system clock, which multiplies this value by the internal frequency-locked loop for a bus clock of 40.0 MHz and single instruction execution time of 25 llS. This clock also provides all the capture and control timing requirements for the other specified circuits. Multiple parallel I/O ports and dedicated asynchronous serial communication signals provide digital control for the circuits of the parallel/serial I/O circuit.
In the preferred embodiment, the graphic LCD and touch screen circuit is the primary user input device for database management for an interactive transponder configuration. For example, it may be a 128x64 graphical liquid crystal display system (LCD) and associated 4-pin touch screen input device. A preferred LCD device is the 51553 manufactured by Optrex and the preferred touch screen device is the TSG-51 manufactured by Apollo Displays. LCD display The accelerometer circuit consists of a low cost +/-1.5 g
dual axis accelerometer that can measure both dynamic, e.g. vibration, and static, e.g. gravity or tilt, acceleration. If the accelerometer is oriented so both its axes are parallel to the earth's surface it can be used as a two axis tilt sensor with a roll and pitch axis.
60 information, configuration commands, and bitmaps images can be loaded through the software calculation engine via a parallel memory interface to emulate a graphical user interface. A touch screen input device is connected to a controller circuit to decode soft key presses at areas over the graphical
The stimuli interface circuit provides the primary visual sensory interface preferably comprised of a linear array of five (5) bright, white light emitting diodes (LED) and
65 object. Preferably, the key presses are registered, filtered, decoded, and processed by the controller and then transferred to the software calculation engine via an interrupt
JX-001.0030
A000030
Case: 12-1252 Document: 46 Page: 120 Filed: 08/29/2012
US 7,292,151 B2 35
driven asynchronous serial communication channel of the I/O interface. A preferred LCD controller is the UR7HCTS manufactured by Semtech.
The timing capture-control circuit provides controlling means for the stimuli interface and portions of the sensor interface. The stimuli interface is preferably comprised of a five channel 16-bit timer PWM module with programmable interrupt control which provides 250 llS timing resolution to automatically modulate the circuits' drivers through variable duty cycle control. 10
36 device, which further comprises a second visual display for providing visual stimuli to a user in combination with the first visual display.
6. The system of claim 1, wherein: the output device comprises an array of light emitting
devices. 7. The system of claim 1, wherein: the output device provides audible stimuli to the user. 8. The system of claim 1, wherein:
In the preferred embodiment, the AID conversion circuit receives the output from the accelerometer circuit and consists of a two channel lO-bit analog-to-digital converter used determine the rotational angle of roll and pitch in the accelerometer deviates from its horizontal plane orientation. 15
This information is communicated to the signal processor via the radio link.
the output device provides tactile stimuli to the user. 9. A system according to claim 1, wherein the first
communication device sends ultrasonic signals received by the processing system for determining movement information for the first communication device.
10. A system according to claim 1, wherein said system comprises a registration system adapted to be configured remotely by said first communication device.
In the preferred embodiment, the radio link circuit is comprised of a wireless bi -directional communication interface (with a receiver and transmitter shown generally at 20 and 30) to (1) receive a synchronization signal for control of the transponders interoperability, (2) to transfer acquired local sensor data, including, but not limited to, accelerometer, heart rate, battery, user I/O status, to processor unit and
11. A system according to claim 10, wherein said regis-20 tration system allows the user to record a reference move
ment trajectory remotely using said first communication device.
12. A system according to claim 1, wherein said first communication device is further comprised of:
(3) to provide means to configure its local database from 25
command of processor unit. The preferred wireless communication link is based upon the AT86RF211, a highly integrated, low-power FSK transceiver optimized for licensefree ISM band operations from 400 MHz to 950 MHz. and manufactured by Atmel. Its key features are described 30
above.
a user input device and display and wherein said first communication device is configured with multiple training applications and wherein the user may choose one training application to activate, and wherein said user may download additional training applications to said communication device.
13. A system according to claim 12, wherein said user input device may be used to configure options customized for the user.
In the preferred embodiment, the switch I/O circuit uses a SPST push button switch for user input to control the system's operational states, start and stop program execution, and function as feedback input to the program. A preferred device is the KSS231 SPST pushbutton switch manufactured by ITT Industries.
What is claimed is: 1. A system for tracking movement of a user, comprising: a first communication device comprising a transmitter for
transmitting signals, a receiver for receiving signals and an output device, said first communication device adapted to be hand-held;
a processing system, remote from the first communication device, for wirelessly receiving said transmitted signals from said first communication device, said processing system adapted to determine movement information for said first communication device and sending data signals to said first communication device for providing feedback or control data; and
wherein said first communication device receives and processes said data signals from said processing system and wherein the output device provides sensory stimuli according to the received data signals.
14. A system according to claim 12, wherein said user 35 input device may be used to authenticate user access and
open a user session. 15. A system according to claim 12, wherein said user
input device may be used to calibrate said first communication device to establish a reference pose or reference
40 traj ectory. 16. A system according to claim 1, wherein said first
communication device is adapted to accept various mechanical extensions pieces depending on the application desired.
17. A system according to claim 1, wherein said first 45 communication device transmits accelerometer signals to
said processing system.
50
18. A system according to claim 1, wherein said first communication device transmits heart rate signals to said processing system.
19. A system according to claim 1, wherein said first communication device is further comprised of:
2. A system according to claim 1, wherein said first 55
communication device is a transponder.
an inertial sensor and wherein said first communication device transmits signals containing orientation information to said processing system.
20. A system according to claim 1, wherein said first communication device is further comprised of: 3. The system of claim 1, wherein the first communication
device further comprises: a first visual display for providing an interactive interface
for the user. 4. The system of claim 3, further comprising: a display device in communication with the processing
system for providing sensory stimuli for the user according to the transmitted signals from the first communication device.
5. The system of claim 4, wherein the display device indicates the movement direction of the first communication
a nonvolatile memory. 21. A system according to claim 20, wherein the first
60 communication device is adapted to download customized user programs from the Internet to be uploaded to a remote system as the application program.
65
22. A system according to claim 20, wherein performance algorithms are stored in said memory.
23. A system according to claim 22, where said performance algorithms calculate custom information personal to a user in real-time.
JX-001.0031
A000031
Case: 12-1252 Document: 46 Page: 121 Filed: 08/29/2012
US 7,292,151 B2 37
24. A system according to claim 22, wherein said performance algorithm produces a Motivation Index that represents the overall level of enthusiasm or enjoyment for a particular activity.
38 37. A system according to claim 36, wherein said receiver
array is in the form of an S-shaped curve. 38. A system according to claim 36, wherein position
information is calculated based on time of flight measurement of said ultrasonic signals.
39. A system according to claim 38, wherein position information can be calculated without interference from occluding objects.
25. A system according to claim 22, wherein said performance algorithm produces a composite numerical value derived from a weighted average of statistical performance indicators and subjective user input including at least one of the following items: relative scoring improvements, conformity to a range of motion pattern, duration of participation, high activity access rate, relative skill level improvement, daily goal achievement.
40. The system of claim 1, wherein the processing system 10 is adapted to determine position information.
41. The system of claim 40, wherein said processing system is adapted to determine the error between the actual movement information of said first communication device and a movement information defined by a reference move-
26. A system according to claim 20, wherein a user's session data can be saved to said memory for later retrieval.
27. A system according to claim 1, further comprising: a second communication device, adapted to be hand held,
in electrical communication with the first communication device, with the processing system adapted to determine movement information of the second com-
15 ment trajectory.
munication device relative to the first communication 20
device.
42. A system according to claim 41, wherein said processing system is adapted to send feedback signals to said first communication device based on said error.
43. The system according to claim 42, wherein the output device provides feedback stimuli to the user in response to the received feedback signals.
28. A system according to claim 27, wherein said processing system is adapted to determine movement information for both said first and second communication devices and to calculate a displacement vector from said movement information.
44. A system according to claim 43, wherein said feedback stimuli are aural instructions to the user for guiding the user's movements to conform to said reference movement
25 traj ectory. 45. A system according to claim 43, wherein said feed
back stimuli are aural cues informing the user of encroachments of threshold conditions.
29. A system according to claim 28, wherein said processing system is adapted to compare said calculated displacement vector to a reference vector position and to calculate a numerical result.
30. A system according to claim 29, wherein said processing system sends feedback signals to said first communication device based on said numerical result.
46. A system according to claim 43, wherein said output 30 device is an array of light emitting devices adapted to be
strobed at an intensity, rate or pattern proportional to said
31. A system according to claim 30, wherein a user's movement efficiency can be determined. 35
32. A system according to claim 27, wherein said processing system is adapted to determine movement information for both said first and second communication devices and wherein a vector is calculated and compared to a desired reference vector to calculate a numerical result and wherein 40
said processing system sends feedback signals to said first communication device based on said numerical result, said first communication device further comprised of an output device for providing feedback stimuli to the user in response to said received feedback signals. 45
33. A system according to claim 27, wherein said processing system is adapted to determine movement information for both said first and second communication devices
:~e~h:~~i:~e~::' :a~:t;::e:~d e!~~:?c~:n~~~~~~~ 50
devices are adapted to communicate with each other for synchronization purposes.
34. The system of claim 27, wherein the second communication device comprises:
an output device for providing sensory stimuli to said user according to said received data signals.
35. A system according to claim 1, wherein said signals transmitted from said first communication device are radio frequency signals.
36. A system according to claim 1, further comprising:
55
60
a receiver array in data communication with said processing system for receiving ultrasonic signals from said first communication device and wherein said receiver array sends data to said processing system for use in 65
calculating movement information for said first communication device.
error. 47. The system of claim 1, wherein the processing system
is adapted to determine acceleration information of the first communication device.
48. The system of claim 1, wherein: the first communication device further comprises a sensor
for determining tilt information of the first communication device; and the first communication device is adapted for transmitting the tilt information to the processing system.
49. The system of claim 1, wherein: the first communication device comprises an interactive
interface such that movement of the first communication device controls the movement of an object in a computer generated virtual environment.
50. An apparatus for use in tracking movement of a user, comprising:
a transmitter for transmitting signals; a receiver for receiving signals wirelessly from a remote
processing system; wherein the apparatus is hand-held; wherein the receiver is adapted to receive feedback or
control data signals from the processing system, the feedback or control data signals derived from processed information including movement information of the apparatus; and
wherein the receiver receives the data signals from the processing system and wherein the apparatus processes the received data signals to provide feedback or control information to the user.
51. An apparatus according to claim 50, wherein said apparatus is further comprised of:
a display for providing an interactive interface for the user.
52. An apparatus according to claim 50, wherein said apparatus is further comprised of:
JX-001.0032
A000032
Case: 12-1252 Document: 46 Page: 122 Filed: 08/29/2012
US 7,292,151 B2 39
an output device for providing sensory stimuli to said user according to said received data signals.
53. An apparatus according to claim 52, wherein said output device is an array of light emitting devices.
54. An apparatus according to claim 52, wherein the output device provides audible stimuli to the user.
55. An apparatus according to claim 50, wherein said processing system is adapted to determine the error between the actual movement infonnation of said apparatus and a reference movement trajectory.
40 71. An apparatus according to claim 50, further comprised
of: an inertial sensor and wherein apparatus transmits signals
containing orientation infonnation to the processing system.
72. An apparatus according to claim 50, further comprised of:
a nonvolatile memory. 73. An apparatus according to claim 72, wherein said
10 apparatus is adapted to download customized user programs from the Internet to be uploaded to the processing system as the application program.
56. An apparatus according to claim 55, wherein the processing system is adapted to send feedback signals to said apparatus based on said error and wherein said apparatus is further comprised of an output device for providing feedback stimuli to the user in response to said received 15
feedback signals.
74. An apparatus according to claim 72, wherein performance algorithms are stored in said memory.
75. An apparatus according to claim 74, wherein said perfonnance algorithm produces a Motivation Index that represents the overall level of enthusiasm or enjoyment for a particular activity.
57. An apparatus according to claim 56, wherein said feedback stimuli are aural instructions to the user for guiding the user's movements to conform to said reference movement trajectory.
58. An apparatus according to claim 56, wherein said feedback stimuli are aural cues infonning the user of encroachments of threshold conditions.
59. An apparatus according to claim 56, wherein said output device is an array oflight emitting devides adapted to be strobed at an intensity, rate or pattern proportional to said error between the movement of said apparatus compared to said reference movement trajectory.
60. An apparatus according to claim 50, wherein said transmitted signals by said transmitter are ultrasonic signals received by the processing system for determining movement infonnation for said apparatus.
61. An apparatus according to claim 50, wherein the processing system is adapted with a registration system adapted to be configured remotely by said apparatus.
62. An apparatus according to claim 61, wherein the registration system allows the user to record a reference movement trajectory remotely using apparatus.
63. An apparatus according to claim 50, wherein said apparatus is further comprised of:
a user input device and display and wherein said apparatus is configured with multiple training applications, each of which is selectively activated with the user input device.
64. An apparatus according to claim 63, wherein said user input device may be used to authenticate user access and open a user session.
65. An apparatus according to claim 63, wherein said user input device may be used to calibrate said apparatus to establish a reference pose or reference trajectory.
66. An apparatus according to claim 50, further comprising:
a remote visual display in communication with the processing system for providing visual stimuli for the user.
76. An apparatus according to claim 74, wherein said 20 perfonnance algorithm produces a composite numerical
value derived from a weighted average of statistical performance indicators and subjective user input including at least one of the following items: relative scoring improvements, conformity to a range of motion pattern, duration of partici-
25 pation, high activity access rate, relative skill level improvement, daily goal achievement.
77. An apparatus according to claim 72, wherein a user's session data can be saved to said memory for later retrieval.
78. An apparatus according to claim 50 adapted to operate 30 in conjunction with a receiver array in data communication
with the processing system for receiving ultrasonic signals from apparatus and wherein said receiver array sends data to the processing system for use in calculating movement information for said first apparatus.
35 79. An apparatus according to claim 78, wherein said
receiver array is in the form of an S-shaped curve. 80. An apparatus according to claim 79, wherein move
ment information is calculated based on time of flight 40 measurement of said ultrasonic signals.
81. An apparatus according to claim 50 further comprised of an output device and wherein said apparatus processes said feedback data from the processing system and provides stimulus from said output device to cue the user to move in
45 a predetennined direction to assess the user's ability to balance.
82. An apparatus according to claim 50 further comprised of an output device and wherein said apparatus processes said feedback data from the processing system and provides
50 stimulus from said output device to cue the user based on the movement information of said apparatus in reference to at least one of: a desired range of motion and a desired location.
67. An apparatus according to claim 50 wherein said 55
transmitted signals by said transmitter are radio frequency signals for transmitting information to the remote processing system.
83. An apparatus according to claim 82, wherein the desired range of motion be established by placing targets in the real or virtual world at predetennined locations.
84. An apparatus according to claim 50, wherein the apparatus is a first transponder adapted for communicating with a second transponder, also hand held by the user. 68. An apparatus according to claim 50, wherein said
apparatus is adapted to accept various mechanical exten- 60
sions pieces depending on the application desired. 69. An apparatus according to claim 50, wherein said
apparatus transmits accelerometer signals to the processing system.
70. An apparatus according to claim 50, wherein said 65
apparatus transmits heart rate signals to the processing system.
85. An apparatus for use in tracking movement of a user, said apparatus being hand-held, comprising:
a transmitter for transmitting signals wirelessly to a remote processing system;
a receiver for receiving signals from the processing system, wherein the received signals are feedback signals derived from comparing movement infonnation of the apparatus with a reference movement infonnation;
JX-001.0033
A000033
Case: 12-1252 Document: 46 Page: 123 Filed: 08/29/2012
US 7,292,151 B2 41
an output device for providing stimuli to the user, wherein the feedback signals are used to initiate aural stimuli to the user; and
a user-actuated button for providing input to the apparatus.
86. An apparatus according to claim 85, wherein said transmitted signals by said transmitter are ultrasonic signals for use in obtaining movement information for said apparatus.
87. An apparatus according to claim 85 wherein said aural stimuli are cues to guide the user's movements to conform to a desired reference movement information.
88. An apparatus according to claim 87, wherein said aural signals are cues to the user for proper movement
42 execution to increase range of motion of a predetermined body part of the user.
89. An apparatus according to claim 85, wherein said aural stimuli are cues to train the user for a predetermined physical activity.
90. An apparatus according to claim 85, wherein said aural stimuli provides the user with on-going performance information and where movement information of said apparatus is collected over a period to time to determine the
10 user's ability to perform a particular movement or activity. 91. An apparatus according to claim 85, wherein said
button is actuated by the user to signal an end of a user movement.
* * * * *
JX-001.0034
A000034
Case: 12-1252 Document: 46 Page: 124 Filed: 08/29/2012
UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
PATENT NO. : 7,292,151 B2 APPLICATION NO. : 111187373 DATED : November 6, 2007 INVENTOR(S) : Kevin Ferguson and Donald Gronachan
Page 1 of 1
It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:
In column 1, line 8, please delete "SUMMER" and insert -- SUMMARY --.
In column 7, lines 11-12, please delete "can be can saved" and insert -- can be saved --.
In column 39, line 25, please delete "devides" and insert -- devices --.
Signed and Sealed this
Twenty Second Day of April, 2008
JON W, DUDAS Director of the United States Patent and Trademark Office
JX-001.0035
A000035
Case: 12-1252 Document: 46 Page: 125 Filed: 08/29/2012
Case: 12-1252 Document: 46 Page: 126 Filed: 08/29/2012
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
June 08, 2010
THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COpy FROM
THE RECORDS OF THIS OFFICE OF:
U.S. PATENT: 7,492,268
ISSUE DATE: February 17,2009
By Authority of the
Under Secretary of Commerce for Intellectual Property and Director of the United States Patent and Trademark Office
N. WILLIAMS
Certifying Officer
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
June 08, 2010
THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COpy FROM
THE RECORDS OF THIS OFFICE OF:
U.S. PATENT: 7,492,268
ISSUE DATE: February 17,2009
By Authority of the
Under Secretary of Commerce for Intellectual Property and Director of the United States Patent and Trademark Office
N. WILLIAMS
Certifying Officer
JX-003
A000036
Case: 12-1252 Document: 46 Page: 127 Filed: 08/29/2012
(12) United States Patent Ferguson et al.
(54) HUMAN MOVEMENT MEASUREMENT SYSTEM
(75) Inventors: Kevin Ferguson, Dublin, OH (US); Donald Gronachan, Holtsville, NY (US)
(73) Assignee: Motiva LLC, Dublin, OH (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.c. 154(b) by 0 days.
This patent is subject to a terminal disclaimer.
(21) Appl. No.: 11/935,578
(22) Filed: Nov. 6, 2007
(65) Prior Publication Data
US 2008/0061949 Al Mar. 13,2008
Related U.S. Application Data
(63) Continuation of application No. 111187,373, filed on luI. 22, 2005, now Pat. No. 7,292,151.
(60) Provisional application No. 60/592,092, filed on luI. 29,2004.
(51) Int. Cl. G08B 23/00 (2006.01)
(52) U.S. Cl. .................. 340/573.1; 340/407.1; 4341114 (58) Field of Classification Search .............. 340/573.1,
(56)
340/573.4,539.12,539.13,539.22,407.1, 340/825.36; 3811315; 4341112, 114
See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
4,337,049 A 4,375,674 A 4,627,620 A 4,631,676 A
611982 Connelly 311983 Thorton
1211986 Yang 1211986 Pugh
111111 1111111111111111111111111111111111111111111111111111111111111 USOO7492268B2
(10) Patent No.: US 7,492,268 B2 (45) Date of Patent: *Feb.17,2009
4,645,458 A 2/1987 Williams 4,695,953 A 9/1987 Blair et al. 4,702,475 A 10/1987 Elstein et al. 4,751,642 A 6/1988 Silva et al. 4,817,950 A 4/1989 Goo 4,912,638 A 3/1990 Pratt 4,925,189 A 5/1990 Braeunig 5,148,154 A 9/1992 MacKay et al. 5,184,295 A 2/1993 Mann 5,214,615 A 5/1993 Baur 5,227,985 A 7/1993 DeMenthon 5,229,756 A 7/1993 Kosugi et al. 5,239,463 A 8/1993 Blair 5,255,211 A 10/1993 Redmond 5,288,078 A 2/1994 Capper et al.
(Continued)
FOREIGN PATENT DOCUMENTS
WO PCT /uS9617 580 5/1997
OTHER PUBLICATIONS
Allard, P., et ai, Three-Dimensional Analysis of Human Movement, Human Kinetics (1995) 3,8-14.
(Continued)
Primary Examiner-Toan N Pham (74) Attorney, Agent, or Firm-Standley Law Group LLP
(57) ABSTRACT
A system for measuring the position of transponders for testing and training a user to manipulate the position of the transponders while being guided by interactive and sensory feedback through a bidirectional communication link to a processing system for the purpose of functional movement assessment for exercise and physical rehabilitation.
15 Claims, 10 Drawing Sheets
(12) United States Patent Ferguson et al.
(54) HUMAN MOVEMENT MEASUREMENT SYSTEM
(75) Inventors: Kevin Ferguson, Dublin, OH (US); Donald Gronachan, Holtsville, NY (US)
(73) Assignee: Motiva LLC, Dublin, OH (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.c. 154(b) by 0 days.
This patent is subject to a terminal disclaimer.
(21) Appl. No.: 11/935,578
(22) Filed: Nov. 6, 2007
(65) Prior Publication Data
US 2008/0061949 Al Mar. 13,2008
Related U.S. Application Data
(63) Continuation of application No. 111187,373, filed on luI. 22, 2005, now Pat. No. 7,292,151.
(60) Provisional application No. 60/592,092, filed on luI. 29,2004.
(51) Int. Cl. G08B 23/00 (2006.01)
(52) U.S. Cl. .................. 340/573.1; 340/407.1; 4341114 (58) Field of Classification Search .............. 340/573.1,
(56)
340/573.4,539.12,539.13,539.22,407.1, 340/825.36; 3811315; 4341112, 114
See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
4,337,049 A 4,375,674 A 4,627,620 A 4,631,676 A
611982 Connelly 311983 Thorton
1211986 Yang 1211986 Pugh
111111111111111111111111111111111111111111111111111111111111111111111111111 US007492268B2
(10) Patent No.: US 7,492,268 B2 (45) Date of Patent: *Feb.17,2009
4,645,458 A 2/1987 Williams 4,695,953 A 9/1987 Blair et al. 4,702,475 A 10/1987 Elstein et al. 4,751,642 A 6/1988 Silva et al. 4,817,950 A 4/1989 Goo 4,912,638 A 3/1990 Pratt 4,925,189 A 5/1990 Braeunig 5,148,154 A 9/1992 MacKay et al. 5,184,295 A 2/1993 Mann 5,214,615 A 5/1993 Baur 5,227,985 A 7/1993 DeMenthon 5,229,756 A 7/1993 Kosugi et al. 5,239,463 A 8/1993 Blair 5,255,211 A 10/1993 Redmond 5,288,078 A 2/1994 Capper et al.
(Continued)
FOREIGN PATENT DOCUMENTS
WO PCTlUS9617580 5/1997
OTHER PUBLICATIONS
Allard, P., et ai, Three-Dimensional Analysis of Human Movement, Human Kinetics (1995) 3,8-14.
(Continued)
Primary Examiner-Toan N Pham (74) Attorney, Agent, or Firm-Standley Law Group LLP
(57) ABSTRACT
A system for measuring the position of transponders for testing and training a user to manipulate the position of the transponders while being guided by interactive and sensory feedback through a bidirectional communication link to a processing system for the purpose of functional movement assessment for exercise and physical rehabilitation.
15 Claims, 10 Drawing Sheets
JX-003.0002
A000037
Case: 12-1252 Document: 46 Page: 128 Filed: 08/29/2012
US 7,492,268 B2 Page 2
u.s. PATENT DOCUMENTS 6,308,565 Bl 6,315,673 Bl 6,346,045 B2 6,361,507 Bl 6,366,272 Bl 6,400,452 Bl 6,430,997 Bl 6,487,906 Bl 6,515,593 Bl 6,545,661 Bl 6,720,876 Bl 6,749,432 B2 6,765,726 B2 6,774,885 Bl 6,834,436 B2 6,876,496 B2 7,158,118 B2 7,236,156 B2 7,239,301 B2 7,262,760 B2 7,359,121 B2 * 7,414,611 B2
1012001 French et al. 1112001 Kopera et al. 212002 Rider 3/2002 Foxlin
5,320,538 A 5,347,306 A 5,372,365 A 5,375,610 A 5,385,519 A 5,405,152 A 5,423,554 A 5,429,140 A 5,466,200 A 5,469,740 A 5,474,083 A 5,485,402 A * 5,495,576 A 5,516,105 A 5,524,637 A 5,577,981 A 5,580,249 A 5,584,700 A 5,587,937 A 5,591,104 A 5,597,309 A 5,616,078 A 5,638,300 A 5,641,288 A 5,645,077 A 5,656,904 A 5,659,691 A 5,702,323 A 5,703,623 A 5,704,837 A 5,711,304 A 5,715,834 A 5,720,619 A 5,759,044 A 5,785,630 A 5,785,631 A 5,790,076 A 5,790,124 A 5,792,031 A 5,812,257 A 5,838,816 A 5,846,086 A 5,850,201 A 5,872,438 A 5,888,172 A 5,890,995 A 5,913,727 A 5,929,782 A * 5,963,891 A 5,989,157 A 6,004,243 A 6,028,593 A 6,043,873 A 6,050,822 A 6,050,963 A 6,054,951 A 6,066,075 A * 6,073,489 A 6,077,201 A 6,088,091 A 6,098,458 A 6,100,896 A 6,119,516 A 6,132,337 A 6,152,856 A 6,162,191 A 6,164,973 A 6,181,343 Bl 6,183,259 Bl 6,198,528 Bl 6,244,987 Bl
611994 911994
1211994 1211994
111995 411995 611995 711995
1111995 1111995 1211995
111996 211996 511996 611996
1111996 1211996 1211996 1211996
111997 111997 411997 611997 611997 711997 811997 811997
1211997 1211997
111998 111998 211998 211998 611998 711998 711998 811998 811998 811998 911998
1111998 1211998 1211998 211999 311999 411999 611999 711999
1011999 1111999 1211999 212000 3/2000 4/2000 4/2000 4/2000 5/2000 6/2000 6/2000 7/2000 8/2000 8/2000 912000
1012000 1112000 1212000 1212000
112001 2/2001 3/2001 6/2001
Baum Nitta McTeigue et al. LaCourse et al. Hsu et al. Katanics et al. Davis Burdea Uerich French et al. Church et al. Smith et al. ................. 702/160 Ritchey Eisenbrey et al. Erickson Jarvik Jacobsen et al. Feldman et al. Massie Andrus et al. Riess Oh Johnson Zaenglein, Jr. Foxlin Lander Durward et al. Poulton Hall Iwasaki et al. Dower Bergamasco et al. Fisslinger Redmond Bobick Heidecke Sypniewski Fischer et al. Alton Teitel et al. Holmberg Bizzi et al. Lasko-Harvill Roston Andras et al. Bobick Andoot Stark et al. Walker et al. Walton Ewart Ro senberg et al. Ramer et al. Faughnn Johnson et al. Sypniewski
340/870.01
Poulton ......................... 482/8 French et al. Cheng Ramer French et al. Strohecker et al. Hock Krupka et al. Studor Foxlin Macri Lyons Macri Maynard Ohsuga
200210183961 Al 2006/0028446 Al
4/2002 Rosenberg et al. 6/2002 Maynard 8/2002 French et al.
12/2002 Hock 212003 Stark et al. 4/2003 Goschyet al. 4/2004 Burgess 6/2004 French et al. 7/2004 French et al. 8/2004 Even-Zohar
12/2004 Townsend et al. 412005 French et al. 112007 Liberty 6/2007 Liberty et al. 7/2007 Liberty et al. 8/2007 Liberty 4/2008 French et al. ............... 359/630 8/2008 Liberty
12/2002 French et al. 212006 Liberty et al.
OTHER PUBLICATIONS
Brownstein, B., et al, Functional Movement in Orthopedic and Sports Physical Therapy, Churchill Livingstone (1997), 15. Brugger, W., et al, Computer-aided tracking of body motions using a c.c.d.-image sensor, Med. BioI. Eng. & Comput, (Mar. 1978),207-210. Codella, c., et ai, Interactive Simulation in a Multi-Person Virtual WorldACM (May 3-7,1992),329-334. DeLoura, M., et ai, Game Programming Gems, Charles River Media, (2000) 200-204. Greenleaf, W.J., DataGlove, DataSuit, and virtual reality Advanced technology for people with disabilities, Proceedings of the Seventh Annual Conference 'Technology and Persons with Disabilities,' (Mar. 1992).211-214 Kasvand, T., et ai, Computers and the Kinesiology of Gait, Comput. BioI. Med. Pergamon Press (1976) vol. 6 111-120. Kenmochi, A., et ai, A netwotk virtual reality skiing system-system overview and skiing movement estimation, Symbiosis of Human and Artifact, (Jul. 1995) 423-428. Kraus, A., Matrices for Engineers, Hemisphere Publishing Corp. (1987) 118-120, 124-126. Lengyel, E., Mathematics for 3D Game Programming & Computer Graphics, Charles River Media (2004) 76-78, 467-468. Medved, v., Towards a virtual reality-assisted movement diagnostics-an outline, Robotica (Jan.-Feb. 1994) vol. 12, 55-57. Mulder, A., Human movement tracking technology, School of Kinesiology, Simon Fraser University (Jul. 1994) 1-14. Ruby, D., Biomechanics-how computers extend athletic performance to the body's far limits, Popular Science (Jan. 1982) 58-60. Sandweiss, J., et al, Biofeedback and Sports Science, Plenum Press New York (1985) 1-201. Scarborough, E.L., Enhancement of Audio Localization Cue Synthesis by Adding Environmental and Visual Cues, Air Force Inst. Of Tech., Wright-Patterson AFB, OH. School of Engineering (Dec. 1992) 1-4. Smith, J., et ai, Virtual Batting Cage and Human Model, Virtual Human http://www.cs. berkeley.edu/'rcdavisl classes/cs294/, (Jun. 17,2004)1-5. Zetu, D., et al, Extended range tracking for remote virtual realityaided facility management, Department of Mechanical Engineering The University of Illinois at Chicago, http://alpha.me.uic.edu/dan/ NsfPaper/nsfl.htrnl, (Apr. 19,2004)1-9. Codamotion: The science of real-time motion capture and analysis, webpages from http://www.chamdyn.comlindex.htrnl. (Apr. 17, 2004) 1. IREX, Virtual Reality Technologies, webpages from http://www. irexonline.comlhow_it_works.htrn, (Apr. 19,2004) 1-2.
JX-003.0003
A000038
Case: 12-1252 Document: 46 Page: 129 Filed: 08/29/2012
US 7,492,268 B2 Page 3
Polhemus, Patriot: The Fast and Affordable Digital Tracker, www. polhemus. com, (Feb. 2004) 1-2. Polhemus, Liberty: The Forerunner in Electromagnetic Tracking Technology, www.polhemus.com. (May 2003) 1-2. Success Story Profile: Innovative Sports Training, Motion Monitor, (2002) 1-2. Reality built for two: a virtual reality tool, Symposium on Interactive 3D Graphics, ACM Press webpages from http://portal.acm.org/cita-
tion.cfm?id~91385.91409&dl+ACM&type~series&i (Jun. 10, 2004) 1-4. Europe is Bursting with Virtual Reality Ideas, But Developers Are Critically Strapped for Cash, webpages fromhttps:llwww/lexis.com! research/retrieve? _m~66dI7057cl b77fl97aledb9f5fadb 87d&_browseType~Text, (Jan. 1993) 1-2.
* cited by examiner
JX-003.0004
A000039
Case: 12-1252 Document: 46 Page: 130 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 1 of 10 US 7,492,268 B2
10
----------"') Y FIG-1A
-~-ooIO~O 1 ~~: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
'"--I ~ __
~-------------- --------) y FIG-1B
JX-003.0005
A000040
Case: 12-1252 Document: 46 Page: 131 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 2 of 10 US 7,492,268 B2
FIG-2A ~ I
FIG-2B
1 kg I I \
FIG-2C
FIG-2D
JX-003.0006
A000041
Case: 12-1252 Document: 46 Page: 132 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 3 of 10 US 7,492,268 B2
, t
Evaluate Display PIN Entry I Security & Navigation
Requirements Controls
~I ./ . Request Q) [requlred]--. AuthentiCation}"" - - -"") . II) c
-'= c.. [not required] User Session ~ [identification] °C :::l
Authenticatet ____ J u Q)
U) User
• User Session
[loaded]
t Configure Display ~ Session UI Graphical
.. Incons of Name & ID
1 Descriptor ,,/"
Display Graphical D Request ,,"
'" Incons of Workload Program ---" Intensity & Limits, Selection +
Measurement User Session Criteria, Stimulus [setup] rh Properties, ect.
I Process I I
Program Display ~ Q) I 1+---_ .... I
II) I C I Request Graphical
-'= Request c.. Incons of Session ---"'" ,,~
Parameters a.
~ ,,-
:::l Setup -" Request
~" & Instruction +' ,,~
Q) U) Program Text
User Session ---" Setup ~
[setup] rh User Session
I Process [setup] rh Session +-----"
J
Parameters Process J
l Program 1+---_ ....
Parameters
i ~ [A or B]
( Deployment Phase FIG-3A
JX-003.0007
A000042
Case: 12-1252 Document: 46 Page: 133 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 4 of 10
( Deployment Phase) i
User Session [save] rh
Process 1M Deployment
Requirements
--------,
Program
[load] - ______ 1
Evaluate Secondary 1M Requirement
US 7,492,268 B2
Display Graphical Incon for Detail
of Modular Length, Weighted, or
Augmentative Force Resistance
Attachments
[not required]
/ _c __ ,> 1
" i / i / i
,-~+.--~/ i i i
Fasten Primary 1M
• Program
[Deployment] I i i i i i
, i \.. J --t-,
Display Graphical Incon of
User Body
[ d] Location &: rea y Strap Holster
~eChan;sm
[required]
Fasten Secondary 1M
[ready]
--- ...... i ..
Program
[Deployment]
i i i i i i i
, i \ .. J --t--'
\ Display
Graphical Incon of
User Body Location &:
Strap Holster Mechanism
Calibration Phase FIG-3B
JX-003.0008
A000043
Case: 12-1252 Document: 46 Page: 134 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 5 of 10 US 7,492,268 B2
Request
( Calibration Phase
Process Requirements of
1M Calibration
Display GraphicalD. Ineon of
Primary 1M Calibration Status
Determine Secondary 1M Requirement
~!nOI requiredl~ [required] Display Graphical~
,......., Primary 1M Pose -'=
---, 1 se~n;~~a~f 1M
~---+<y calib~:~~n Status .B Modification o E o I::
I I
~ Request ' ---""
CI) L......J Process
Primary 1M Program ,......., -'= o +' o
Secondary 1M Pose Modification 1
I If) o
-'= c.. Pose
Modulate Au ra I/Visua I/Tacti Ie
Stimuli rh
Evaluate Primary 1M Pose Match (~,
[Calibration]
I I I J
----'
-----<! Mot~' Pose to" Intial Reference
Movement Trajectory
E o I::
L......J
Process Secondary 1M
Pose
Modulate Aural/Visual/Tactile
Stimuli rh
Evaluate Secondary 1M
+ Program
[Calibration]
: I I
__ ~J
'------< Match Pose to ~
pose~MatCh P-,.,_"_,,_,_,,"""--__ .,..
[match] Intial Field
____ ~~ ..... ___ ~ ____ [_t .... a~tc_h] Position
( Execution Phase FIG-3C
JX-003.0009
A000044
Case: 12-1252 Document: 46 Page: 135 Filed: 08/29/2012
u.s. Patent Feb. 17,2009
~--[Iearn]---......
Request Primary 1M Pose ---,
Modification • Program
[Record]
User Session
[Setup]
CI> I Cf) I 0 .s::. I
c.. I:: 0
:,J :::l 0
Process CI> x Learn I.LI
Terminus
[not done]
[done]
Program
[Save]
• )
FIG-3D
Sheet 6 of 10
Query Learn
Modality
[normal]
Process Primary 1M
Pose
Process Program Terminus
[done]
US 7,492,268 B2
---, I I I
+ Program
[Execution] I I I I I I I I I J
--~
[not done]
Data Analysis Phase
JX-003.0010
A000045
Case: 12-1252 Document: 46 Page: 136 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 7 of 10 US 7,492,268 B2
---------L-~ --r------=---I-~) -:~-:--~=:---]
"~------~
/ .-.-.-
--- -""" ......
10
I I
/
I I
I
I
I I
I I
FIG-4B
JX-003.0011
A000046
Case: 12-1252 Document: 46 Page: 137 Filed: 08/29/2012
u.s. Patent Feb. 17,2009
Remote Position Processor
Analog Signal Processor Interface
Receiver Constellation
Sensor & Preamplifier
«access»
r----I J
I-I I I I J r--------
r, Amplifer &
BW Filter
J
,I I
«a.:c:~~_...J
r, Comparator
I I I I I I I
«access»
Absolute Value
«access»
Peak Detect &
Sample-Hold
«access»
~-------------------
L..-U_S_B_2_'O_J ..... ----- --
FIG-5
Communication Interface
Link
L...-___ ---'
Sheet 8 of 10
Digital Signal Processor Interface
Processor
US 7,492,268 B2
r, ~-~, Timebase .s I AID
Conversion .s I en I M I
L..-____ .r ...... --.... 1
\ I I I
r, Phase-Locked
Loop
r-
I "" ) I I I I
L...-_...,._---' ~ I 15 I I "t) --l-
~-) (
Timing Capture & PWM
J
------./ I
~ en en CI) o o c ~
I I I
r, Software
Calculation Engine
I I I
r---- r,
Para lIel/Seria I
I/O (
I I I
_____ J
r, Digital Filter
r------------I «1M status» I I
-.I
20
Radio Link
30
I ---.I
~~..--=-=-.;--n ---
JX-003.0012
A000047
Case: 12-1252 Document: 46 Page: 138 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 9 of 10 US 7,492,268 B2
FIG-6A
FIG-6E
FIG-6e
JX-003.0013
A000048
Case: 12-1252 Document: 46 Page: 139 Filed: 08/29/2012
u.s. Patent Feb. 17,2009 Sheet 10 of 10 US 7,492,268 B2
Transponder
Stimuli Interface Processor I =jl
I 1-, Fit n r.L, 28MHz
White LED ~ -----, & Driver e ... ~ Tlmebase r.L, .....
Vibrator 8 .c:::::::n «clock»l Motor ~ ¥ r.L, & Driver >- IJ }I. PWM
f-, -----, r.L, «access>~
Audio «configure» Annuciator
-----'
& Driver r===n. ~ r===n. T (J)
r.L, (J) r.L, CI) 0
AID 0 Software ,- 0
Calculation Conversion ¥1"0._ Sensor Interface ~ Engine
Fit lioonngu .. J ~ =n ()~
r.L, E
Ultrasonic ~
Transmitter r.L, ¥
J Parallel/Serial
~ess»T «access» I/O " -----, T r.L,
~ -::-r.L, Accelerometer ~()
/ :::l
Heart Rate 0> t;::
Receiver " .:: 0
-""'''" 0
" ¥ i I Dual Axis I
low g I I I I
I I
Communication Interface I Polar
( I compatible ~ user I/O» ..... ( ,.----'
-----, T -----, «user I/O» r.L, r.L,
Graphic LCD I/o &
r---- ,- Switch Touch Screen n ( ) -----, «1M msg»
20, r.L, ~'\
FIG-7 Radio Link
30 ( )------------ - 900MHz ISM
JX-003.0014
A000049
Case: 12-1252 Document: 46 Page: 140 Filed: 08/29/2012
US 7,492,268 B2 1
HUMAN MOVEMENT MEASUREMENT SYSTEM
This application is a continuation of U.S. Ser. No. 111187, 373 filed 22 luI. 2005, now U.S. Pat. No. 7,292,151, issued 6 Nov. 2007, which claims the benefit of U.S. 60/592,092, filed 29 luI. 2004. Each of these applications is incorporated by reference as if fully recited.
BACKGROUND OF THE ART AND SUMMARY OF THE INVENTION
This invention relates to a system and methods for setup and measuring the position and orientation (pose) of transponders. More specifically, for training the user to manipulate the pose of the transponders through a movement trajectory, while guided by interactive and sensory feedback means, for the purposes of functional movement assessment for exercise, and physical medicine and rehabilitation.
Known are commercial tracking and display systems that employ either singularly, or a hybrid fusion thereof, mechanical, inertial, acoustical or electromagnetic radiation sensors to determine a mobile object's position and orientation, referred to collectively as pose.
The various commercial tracking systems are broadly classified by their relative or absolute position tracking capability, in which system the pose of a mobile object is measured relative to a fixed coordinate system associated with either combination of receiver(s) or passive or active transmitter(s) housing mounted on the user. The tracking system's components may be tethered with obvious inherent movement restrictions, or use wireless communication means to remotely transmit and process the information and allow for greater mobility and range of movement.
Typically these tracking systems are utilized for biomechanics and gait analysis, motion capture, or performance animation and require the sensors to be precisely mounted on the joints. Various means of presenting the tracking information in a visual display are employed, such as Heads-Up Display (HUD), that provide occluded or see-through visibility of the physical world, or Fixed-Surface Display (FSD), such as computer desktop monitors, depending upon the simulation and immersive quality required for the application. The application may require various degrees of aural, visual, and tactile simulation fidelity and construct direct or composite camera views of the augmented or three dimensional (3D) virtual reality environment to elicit interactive user locomotion and/or object manipulation to enhance the user's performance and perception therein. The tracked object may be represented in the virtual environment in various forms, i.e., as a fully articulated anthropoid or depicted as a less complex graphical primitive. The rendering strategy employed depends upon the degree of photo realism required with consideration to its computational cost and the application's proprioception requirements.
Tracking technologies possess certain inherent strengths and limitations dependent upon technology, human factors, and environment that need consideration when discussing their performance metrics. Regardless of differentiating resolution and accuracy performance benchmarks, many implementations suffer from varying degrees of static and dynamic errors, including spatial distortion, jitter, stability, latency, or overshoot from prediction algorithms. Some human factors include perceptual stability and task performance transparency, which are more subjective in nature. And environmental issues such as line-of-sight, sensor attachment, range, and multiple-object recognition, need to be considered when
2 selecting the optimal technology for the most robust application development. Irrespective of the intrinsic strengths and weaknesses of the tracking technology employed, ultimately the user's satisfaction with the system's utilization and efficacy, including the production of reliable, easily understood, measurable outcomes, will dictate the overall success of the device.
This invention's system and methods facilitates biomechanical tracking and analysis of functional movement. In the
10 preferred embodiment, this invention is low cost, robust, easy to deploy, noninvasive, unobtrusive, and conveys intuitive and succinct information to the user to execute movement properly and provides performance indicators of said movement for feedback purposes. One feature of the present inven-
15 tion provides for an interactive tracking system because the sensor functionality, or referred to herein as active transponders or transponders, is integrated with local user input control, and real-time sensory interfaces on the same device. The transponder is a wireless communication and monitoring
20 device that receives a specific signal and automatically responds with a specific reply. In one embodiment, the invention provides functional movement assessment based upon the relative measures of limb pose with respect to two positions defined by the transponders. The transponders can oper-
25 ate independently or work in unison to process and share computational tasks and information between the local databases. This decentralized, distributed processing scheme allows the configuration and coordination of the training session, and processing and analysis of the measurements to
30 occur without requiring expensive auxiliary computer and display systems to manage the same, and without relying on costly software development of complex synthetic environments for visualization purposes. Also, the user can manage the applications and performance databases off-line on a
35 remote computer system with Internet connectivity to customize and configure the system parameters in advance of their session.
The present invention is designed to provide such system and methods for high-fidelity tracking or registration of the
40 poses of active transponders and engage the user to purposely manipulate the transponders' pose along a prescribed or choreographed movement trajectory in order to train and assess functional movement capability. In the preferred embodiment, the system is comprised of two subsystems: (1) a sub-
45 system comprised of one or more active transponders, which, in its most sophisticated implementation, responds to periodic requests from another component of the system to radiate or transmit a signal for purposes of absolute position tracking; processes an embedded inertial sensor for relative
50 orientation tracking and absolute tracking refinement; and provides an essentially real-time aural, visual, and tactile sensory interfaces to the user, and (2) a subsystem comprised of a centralized position processor system or unit and receiver constellation unit, collectively referred to as the processor
55 unit, which is essentially a signal processor that synchronizes the transponders' periodicity of radiating signal and other operational states; collectively receives and processes the radiated signal; iteratively calculates the transponders instantaneous pose and convolution, thereof; and continually
60 exchanges this information, and its analysis thereof, with the transponders and/or auxiliary host computer system in essentially real-time via a combined wireless and tethered communication means. This real-time bi-directional exchange of information allows for proper transponder identification,
65 coordination, and the accurate measurement of pose, thereof, and timely actuation of the sensory interfaces for optimal user regulated closed-loop control.
JX-003.0015
A000050
Case: 12-1252 Document: 46 Page: 141 Filed: 08/29/2012
US 7,492,268 B2 3
The transponder is broadly classified by its level of hardware and software configuration that define its scope of intelligence, sensory support, and configuration. The degree of intelligence is detennined by its capability to locally access, process, and modify the database. Further, either transponder classification can be sub-classified by its manipulative requirements. In one embodiment, where multiple transponders are used, a principle transponder is consciously and deliberately moved along the reference movement trajectory, while a subordinate transponder serves as an anchor or sec- 10
ondary reference point elsewhere on the locomotion system whose kinematics are not necessarily controlled by the user's volition.
4 a means to create a single movement vector whose end
points are defined by the locations of at least two transponders, wherein, the expansion and contraction of the vector's length is calculated, analyzed, and reported in essentially real-time;
a means to create a single movement vector whose endpoints are defined by the locations of two transponders, wherein, a representative point along the vector length is referenced and its higher-order derivatives are computed by mathematical numerical processes, wherein the result is calculated, analyzed, and reported in essentially real-time; and,
a means to correlate said vector's length and at least one other measure consisting of a higher-order derivative, to the reference movement trajectory, wherein the result is calculated, analyzed, and reported in essentially real-time.
A registration system for practical functional movement applications should clearly convey information to the user regarding his movement quality while he performs the task, without compromising or distracting from said execution by uunecessary head movements or change in eye gaze and normal focus. Poor visualization strategies that distract the user are ineffectual for promoting heads-up, immersive inter-
An interactive transponder, preferably, has significant intelligence; supports relative and absolute tracking capabili- 15
ties; provides complete sensory stimuli support; provides for functional enhancement through attachment of modular, extension pieces; and provides a user display and input system to control the training session. In the preferred embodiment, the interactive transponder is primarily held in the hand 20
to facilitate more complex user input and greater sensory intimacy. Conversely, in another embodiment, the fixed transponder has limited intelligence; supports only the absolute pose tracking capability; provides no sensory stimuli support; and is usually mounted to a fixed site on the limb or trunk. 25 action, and the alphanumerical information it imparts often
can not be consciously processed fast enough to elicit corrective action. This system provides for both a local, standalone sensory interface as a primary feedback aid, or alternatively,
A combination of transponder deployment strategies may be required depending on the training session's objectives, such as two interactive transponders grasped by each hand; or alternatively, an interactive transponder, and a fixed transponder attached to the limb or trunk; or lastly, two fixed transpon- 30
ders attached to the limb(s) and/or trunk. In one embodiment, this invention proposes to elicit move
ment strategies based on the deployment of at least two transponders that define the endpoints of a movement vector whose relative translation and rotation is measured and evalu- 35
ated for the assessment of functional movement capability, including but not limited to, limb range of motion and its control thereof, limb strength conditioning, and overall proprioception and hand-eye coordination skills, and overall body movement. This registration system measures a single move- 40
ment vector whose endpoints are comprised of an anchor point, i.e. one that is located in a less dynamic frame of reference, e.g., such as the trunk or abdomen, and another more distal location fixed on or held by a limb or extremity, e.g., the hand, arm, or leg. As this movement vector is trans- 45
lated and rotated through space by the act of the user modifying the pose of the principle transponder in concert with the reference movement trajectory, the vector's length will expand and contract relative to the proximity of principle transponder with respect to the subordinate transponder. The 50
vector's length conveys unique and explicit infonnation regarding the user's movement efficiency and biomechanical leverage. For example, by attaching a fixed subordinate transponder at the hips and a fixed principle transponder on the upper arm, the biomechanics of the act of lifting a box or 55
similar object can be elegantly qualified. If the user assumes a poor lifting technique, i.e. legs locked with the trunk severely flexed with head down and the anns stretched out beyond the basis of support, the vector's length would consistently be measured longer than compared to a good lifting 60
technique, i.e., legs bent at knees with the back straight, head gaze up, and arms close to body. Also, the measurement( s) of higher-order derivatives derived from numerical mathemati-cal processes of a reference point described by the vector would provide additional indication of movement control or 65
smoothness. In summary, one embodiment of the present invention is comprised of:
an interface to a remote fixed-surface display for greater visualization and simulation capabilities. The visual stimulus could be modulated to warn of range violations, or provide signals for purposes of movement cadence and directional cueing. A principle interactive transponder is typically hand-held, which is naturally in close proximity to the user's aural and visual sensory field during most upper extremity movements, or, conversely, the visual stimulus may be viewed through a mirrored or reflective means if not in optimal lineof-sight. A remote fixed-surface display might augment the immersive quality of the user's experience by providing control of a view camera of a simulated computer environment, and display of the transponders and/or interactive objects' static or dynamic poses within the computer display's skewed through-the-window perspective projection.
In summary, one embodiment of the present invention is comprised of:
a means for modulating an embedded luminescent display organized and oriented into a directional-aiding pattern, by varying its degree of intensity and color, or other physical characteristics, to provide a visual display stimulus. This sensory interface is excited at a rate, repetition, or pattern proportional to the pose error of the transponders' movement traj ectory compared to the reference movement trajectory;
a means to view said visual display stimulus with the aid of a mirror(s) or other reflective means;
a means for the real-time projection of sound or speech commands through an audio device to provide warning, alarm, instructional, and motivational aid, and/or additional cueing upon encroachment of static and dynamic limit/boundary conditions defined by the reference movement trajectory;
a means for real-time tactile feedback including, but not limited to, modulation of the rotational properties of a vibrator motor proportional to the pose error of the transponders' movement vector compared to the reference movement trajectory;
JX-003.0016
A000051
Case: 12-1252 Document: 46 Page: 142 Filed: 08/29/2012
US 7,492,268 B2 5
a means for combining the excitation of said stimuli proportional to the pose error of the transponders' movement vector compared to the reference movement trajectory; and,
a means to coordinate the real-time, periodic parametric update and modulation of the stimuli imparted by the sensory interfaces within the transponders from a processing unit by means of a wireless communication linle
This invention addresses the need for an intuitive, interactive method to instruct, create, and deliver a movement tra- 10
jectory command without necessarily relying on pre-programmed, regimented movement trajectories. The registration system can be configured via remote setup at the principle transponder to pre-record and choreograph a freeform movement trajectory of the principle transponder with 15
the intent of the user mimicking the same said path. This impromptu learning modality can expedite the session down time between different users and movement scenarios, and accommodate users' high anthropometric variability in range of movement. In summary, one embodiment of the present 20
invention is comprised of: a means is to provide a movement trajectory learning
modality that allows the user to calibrate and create the desired endpoints, midpoints, and/or total reference movement trajectory through user programmer entry of 25
an input device resident on the transponder; a means to process and save a movement trajectory using a
computationally efficient Catmull-Rom spline algorithm or other similar path optimizing algorithms to create control points along key points of the movement 30
trajectory that define the optimally smoothest path intersecting the control points;
a means to provide database management by a processing unit via a wireless communication link or, alternatively, through user data entry of an input device resident on the 35
interactive transponder; and, a means to access, edit, and store the program and/or data
bases to nonvolatile memory operably coupled to the principle transponders for the purpose of automating the creation, delivery, storage, and processing of movement 40
trajectories. Customized user programs and databases would be downloaded from a central repository or relevant website in advance of the training session to the transponder from the user's home location via the Internet or other convenient locales having networked Inter- 45
net access, and transported to the systems remote physi-cal location, and uploaded into the system's memory, and executed as the application program.
This a priori process of remote selection, download, and transfer of programmatic content and database would mini- 50
mize the user's decision making and input during product utilization by offering only relevant and customized programming material of their choosing targeted for their specific exercise, fitness, or rehabilitation goals. Performance data could be saved indefinitely in the database's nonvolatile 55
memory, until an upload process was performed through the said network so the database could be transferred to another location for purposes of, but not limited to, registration, processing, archival, and normative performance evaluation, etc.
6 but may include external media devices, such as USB FLASH Key or other portable media means, that may have interoperability with other computerized devices. The data structures may include:
Modulation & Feedback Thresholds/Triggers Propertiesthe aural, visual, tactile interfaces require threshold settings which determine their excitation or stimulation characteristics. These settings can be derived from previous performance data or defaults determined from normative data, or modified in real-time, by algorithmic methods including moving averages, standard deviations, interpolation based upon goal-oriented objectives, etc.
Normative Performance-performance data collected over a large population of users through controlled studies, that is distilled down into specific user categories based upon certain demographics that the user may compare and rank hislher results. This data may be initially embedded within the transponders or position processor non-volatile memory and may be augmented or modified automatically or by user volition when counected to the Internet.
Competitive Ranking-applications which have a predominate point goal-oriented purpose would allow access to a global ranking file archive accessed through the Internet or automatically via updated executive files. This ranking file would be created through an analysis of user participation and publishing of hi s/her results through Internet Web-based servIces.
Downloadable Executive Programs & Configurationsnew software programs, including new features, enhancements, bug fixes, adjustments, etc., could be downloaded to the transponder through an Internet connection. Graphics images would be stored in compressed or uncompressed binary forms, i.e., bitmap, gif, jpeg, etc. This new programs could be transferred to any suitable computerized position processor unit located at a remote facility via the transponder's wireless link. Therefore, the user's transponder is the node that establishes the portable network capabilities of the system, not necessarily the computerized position processor.
Custom Menu Interfaces-specialized activities may require more advanced (or simplified) interfaces dependent upon the users' cognitive abilities and interactive specificity. This menu may include interactive queries or solicit information regarding the user's daily goals, subjective opinions or overall impression of the activity and ones performance which could be incorporated in the Motivation Index described below.
Report Generation Tools and Templates-XML, HTML or other authoring language used to create documents on the Web that would provide an interactive browser-based user interface to access additional performance data analysis and report generation tools and templates that may not be available or offered with the standard product.
Custom Performance Algorithms---certain applicationspecific performance analysis may require dynamically linked algorithms that process and calculate non-standard or specialized information, values, units, physical measurements, statistical results, predictive behaviors, filtering, numerical analysis including differentiation and integration,
An exemplary list of specific data structures contributing to or affecting the means for automating the creation, delivery, storage, and processing of movement trajectories described below may be stored within the non-volatile memory of the transponder or position processor which may use high-density serial FLASH, although other types of memory may be used such as SmartMedia, Compact Flash, etc. Additionally, the memory device interface should not be limited to internal,
60 convolution and correlation, linear algebraic matrices operations to compute data pose and scaling transformation, and proprietary types. One example of a proprietary type is Motivation Index, a composite numerical value derived from a weighted average of statistical performance indicators and
65 subjective user input including relative scoring improvements, conformity to ROM pattern, lengthy activity access duration, high access rate, relative skill level improvement,
JX-003.0017
A000052
Case: 12-1252 Document: 46 Page: 143 Filed: 08/29/2012
US 7,492,268 B2 7
daily goal achievement, etc., that could represent the overall level of enthusiasm and satisfaction, the user has for a particular activity.
Range of Motion (ROM) Pattem Generator-the ROM pattem requires some key control points to be captured along the desired trajectory and stored in order that the algorithm can calculate an optimally smooth path, in real-time, during the comparative analysis phase.
ROM Pattem Capture & Replay-the ROM pattem can be can saved to memory in real-time by discrete position 10
samples versus time depending upon the resolution desired and memory limitations and later played back on the transponder or remote display for analysis.
Activity Specific Attributes-includes Reps/Sets, Duration, Pause, Heart Rate Limits, intra-activity delay, level, 15
point scalars, energy expenditure, task-oriented triggers, etc., and other parametric data that controls intensity, execution rate and scoring criteria for the activity.
Instructional Information-textual, graphical, or animation-based instruction, advice, coaching, activity description, 20
diagramed transponder deployment and intra-device connectivity, etc. that facilitates the intuitiveness, understanding, and usage of the system. The form of instruction may include music files saved in various formats, including Wave, MP3 or other current or future audio data compression formats, and 25
video files saved in MPEG or other current or future video data compression formats.
8 environments, and especially when the tracking volume likely contains potentially occluding objects, i.e., an uninvolved limbs or clothing, that become potential sources of competing, reflected paths. The preferred embodiment of the registration system utilizes the time of flight (TOF) measurement of ultrasonic acoustic waves due to its immunity from interference from the visible and near-visible electromagnetic spectrum and its superior ability to overcome most multi-path reflections problems by simple gated timing of the initial wave front. Upon command from the processor unit, the transponders produce a few cycles burst of ultrasonic energy and the transducers of the receiver constellation unit are stimulated and mechanically resonate accordingly, upon the wave front arrival. The processor unit's analog signal processing circuits transform the mechanical energy into electrical signals that resemble tapered sinusoidal waveforms, which another electronic circuit triggers upon using an adaptive threshold technique which, in turn, the processor unit detects and calculates TOF timestamps indicating the wave front arrival. In the preferred embodiment, the system overcomes the ultrasonic technology's intrinsic challenge of precisely triggering on same the waveform location and provides consistent unambiguous trigger detection by comple-menting the adaptive threshold technique with a software timestamp correction algorithm, which includes in part, a digital over-sampling and averaging timestamp algorithm, a relative timestamp correction scheme utilizing a predictive algorithm of higher-order Taylor series based derivatives, and
Real-time Data Management-proprietary data management protocols that reside above the communication driver layer that manage the real-time, synchronous and asynchronous exchange of data between transponder(s) and position processor. This would provide an essential real-time sharing
30 an absolute timestamp correction scheme that minimizes the range error based upon discrete biasing of timestamps.
of activity data, analysis, and feedback stimulus thresholds, or coordination of multiple transponder configurations, or for a collaboration of same or different user requirements to 35
complete a similar activity objective. This invention addresses the need for adaptability of the
registration system to different movement measurement scenarios. In one embodiment, it utilizes a versatile, modular configuration and mounting of the transponders onto the user. 40
The efficient deployment of the transducers between different users' and from task to task requires a universal mounting scheme to provide consistent localization and pose of the transponders at the desired measurement sites on user's body. Also, to compensate for the receivers' finite tracking volume 45
when stationary, the receiver constellation unit may be mechanically modified to optimize its tracking properties by conveniently repositioning it in closer proximity to the expected transponders movement trajectories and line-ofsight, thereof. In summary, one embodiment of the present 50
invention is comprised of:
Further, in the preferred embodiment, the processor unit utilizes the absolute and relative trigger timestamps in a multi-modal trilateration algorithm for the measurement of three-dimensional (3 D) translations and rotations of the transponders. The primary trilateration calculation is derived by an application of Pythagoream theorem involving a point position solution based-upon range measurements from at least three (3) points, versus the well-known triangulation method which uses bearing angles of two cameras of known pose. Additionally, the system's main accuracy limitation is mostly affected by the temperature variability of outdoor environments and its influence on the speed of sound in air value. This algorithm mitigates this problem by mathematically computing the speed of sound every analysis period provided at least five (5) receivers and a transponder synchro-nizing means are utilized. If the integrity of the synchronizing signal is temporarily compromised, the system automatically employs a variation of the trilateration algorithm that uses the last known speed of sound value.
In the preferred embodiment, the maximum update rate, and hence the major contributor to the latency of the position calculation, is determined by the typical acoustical reverberation, typically between 20 to 100 ms, encountered in an
a means to quickly and efficiently alter the location of the transponders using a fastening system designed to quickly attach and dispose various forms of transponder assemblies;
a means to augment the physical properties, i.e., weight and length, of the principle transponder with adjunct electro-mechanical components that provide variations in biomechanical leverage for isotonic and isometric utilization; and,
55 indoor environment. Since the transponders are held or fixed on the user's body and, therefore, are mobile, the TOF measurements will experience an additional latency effect. A Kalman filter is used as a prediction/estimation strategy to minimize and compensate for the latency effect. The predic-
a means to allow the user to manually alter the geometry and pose of the receiver constellation unit to facilitate an optimal tracking location based upon collectively maximizing the ultrasonic source's energy received at the transducer interface.
This invention addresses the practicality and robustness of the registration system when used in either indoor or outdoor
60 tion algorithm uses a higher-order Taylor series based derivatives and augmentative inertial sensor data. Its predictive refinement is dependent upon predefined models of expected movement conditions. Because functional movement is episodic, having periods of stillness interspersed with bursts of
65 motion activity, a multi-modal filtering strategy is preferably employed to handle the unpredictable jerkiness at the start of motion and relatively predictable, smooth motion afterwards.
JX-003.0018
A000053
Case: 12-1252 Document: 46 Page: 144 Filed: 08/29/2012
US 7,492,268 B2 9
In sunnnary, the preferred embodiment of the present invention is comprised of:
10 FIGS. 2A-2D illustrate example extension pieces for the
present invention; a means to detect the same carrier wave cycle of ultrasonic
energy using a software correction algorithm requiring multiple, consecutive TOF acquisitions as input for the digital over-sampling and averaging algorithm, the calculation of a higher-order numerical differentiation of the past and current TOF infonnation as input for the predictive algorithm of higher-order Taylor series based derivatives used for the relative TOF correction, and a 10
measurement of the intra-pulse time intervals of consecutive TOF acquisitions as input for the absolute TOF correction scheme that minimizes the range error based upon selective biasing of the TOFs;
FIGS. 3A-3D illustrate one example of process flows for the present invention;
FIGS. 4A and 4B illustrate a sample application of the present invention;
FIG. 5 illustrates a block diagram of the remote processing system of the present invention;
FIGS. 6A-6C illustrate example receiver configurations of the present invention; and
FIG. 7 illustrates a block diagram of the components of one embodiment of the transponder of the present invention.
DETAILED DESCRIPTION OF THE EMBODIMENTS a means to utilize a dual matrix fonnulation of the trilat- 15
eration algorithm, and a calculation strategy thereof, which decision is dependent upon the integrity of the system's connnunication link, synchronization condition, and the desired measurement accuracy; and,
a means to coordinate the information transfer between 20
transponders and the processor unit so that their contribution to the resultant movement vector calculation can
The present invention provides a practical, versatile measurement tool for the assessment of the user's manipulation strategy of the transponder 10 or transponders along a reference movement trajectory. Moreover, the system and methods measure and analyze the kinematics of the relative translations and rotations of the limbs or extremities with respect
be measured without intra-signal interference. These goals will be attained by such system and methods
that are comprised of the user's interaction described by the following steps as set forth as the preferred embodiment:
a. Authenticate user access and open user session from a local or remote database;
b. Setup user training session, i.e., workload limitations, measurement criteria, and audio/visual/tactile stimuli;
c. Select training program and configure its options; d. Deploy the transponders as instructed to predefined loca
tions of users locomotion system to create at least one transponder movement vector;
e. Calibrate the transponder movement vector to establish its reference pose;
f. Create a movement trajectory using learn mode, if required;
g. Initiate the start of session; h. Detennine the instantaneous pose of transponder move
ment vector relative to its reference pose from a periodic temporal iteration of this step;
i. Perfonn qualitative and quantitative statistical analysis of accumulated measured poses of the transponder movement vector relative to the pattern of instantaneous poses defined by the reference movement trajectory;
j. Update the major transponders sensory interfaces to modulate said system parameters in a periodic temporal iteration of this step;
k. End the session once program objectives have been obtained;
1. Analyze the results by interacting with local and/or remote databases;
m. Provide numerical, graphical, and/or animated infonnation indicating desired perfonnance measurements.
BRIEF DESCRIPTION OF THE DRAWINGS
to each other or to a more inertial reference location on or off body as the transponders are manipulated. This infonnation
25 provides useful insight on biomechanical demands and anthropometric factors that influence human movement efficiency and control. Although measurement perfonnance metrics are important design criteria, it's equally important to provide intuitive and motivating program instruction and
30 administration, and to provide comprehensive analysis and integration of the motion data in a fonn that is objective and easily interpreted. This system improves upon the practicality and user interactive aspects of setup, deployment, calibration, execution, feedback, and data interpretation of a tracking
35 system designed for function human movement. Human movement is a response to external enviroumental
forces which requires the accurate coordination of the distal segment( s) to compensate for these forces. Skillful coordination of human movement is dependent upon the cohesive
40 interaction of multiple sensory systems, including visual, vestibular, with the musculoskeletal system. More specifically, the challenges and goals of cognitive spatial mapping, (2) minimization of energy expenditure, (3) maintaining stability, (4) steering and acconnnodation strategies for various
45 environments, (5) dynamic equilibrium, (6) active propulsion and weight support, and (7) core locomotion pattern should be relationally considered to properly assess human movement. Therefore, it is preferable to engage the interaction of these sensory systems during a training session to promote
50 the desired functional movement outcome. Because many movements persist for 400-500 ms, enough time is allowed for the initiation of the movement and for user correction based upon visual and kinesthetic infonnation acquired during the time of the movement. However, the implemented
55 means of visual feedback should be not be distracting or interfering with the task at hand. In the preferred embodiment, this system engages the sensory systems with nondistracting, intuitive, embedded aural, visual, and tactile
The disclosed embodiments will be better understood 60 stimuli which provide real-time indication of the principle transponder pose error with respect to the reference movement trajectory. when reference is made to the accompanying drawings,
wherein identical parts are identified with identical reference numbers and wherein:
FIG. lA illustrates one example of a deployment apparatus of the present invention;
FIG. IB illustrates one example of hand-held fonn for the transponder of the present invention;
In order to conduct a time efficient training session, this registration system attempts to minimize the encumbering experimental setup and calibration procedures characteristic
65 of more complex and higher cost motion analysis technology. These complementary systems serve important academic or clinical oriented research needs or for motion capture for
JX-003.0019
A000054
Case: 12-1252 Document: 46 Page: 145 Filed: 08/29/2012
US 7,492,268 B2 11
computer animation purposes and strive for highly accurate measurement of joint motion data in terms of angular displacement. Therefore, the integrity and reliability of their motion data is dependent upon proper sensor setup and calibration.
For instance, single axis goniometer-based systems usually require specially designed harnesses to hold the monitor and are firmly strapped or taped over the joint to avoid relative motion artifacts. Usually these devices are tethered and their fit, weight, and constraining mechanical linkages can impose limitations on the joint motion and cause discomfort for the user. Most optical or video-based systems require the placement of numerous active or passive markers over landmarks, such as the joints' center of rotation. These systems should guarantee sufficient environmental illumination and contrast between markers and background to function optimally. Also, these systems are severely affected by occluded markers that may disappear for long periods of time due to rotations and line-of-sight limitations. Other video-based systems do not use markers but require the assignment of the body's joints manually or through computerized automation during data analysis, making real-time analysis arduous and real-time feedback virtually impossible.
In the preferred embodiment, the system doesn't require complicated, time consuming sensor setup and calibration by virtue of it minimalist sensor requirements and uncomplicated sensor mounting. Instead, it requires only the deployment of a sensor on the body (in one embodiment a dual sensor group on a combination oflimb(s) and or trunk) and doesn't enforce stringent movement protocol, but encourages free-form, unrestrictive movement of the transponders.
The transponder's preferred deployment means, include either insertion into a universal strap and holster apparatus (FIG.IA) that secures on the user's limb, extremity, or trunk, including, but not limited to, the hip, ankle, knee, wrist, upper arm, neck, waist or an augmentative mechanical attachment to one or a combination of modular extension pieces shaped into a hand-held form (FIG. IB). A strap ortorx-like clip and holster design provides a firm, yet light weight and comfortable mounting location away from areas that clothing and or uninvolved limbs may occlude.
The modular extension piece is either an instrumented sensory type designed to support alternative tactile stimulus device or alternative configurations of aural, visual, and tactile feedback types, or non-instrumented, weighted extension pieces as shown in FIGS. 2A-2D. All modular extension pieces may be of various physical dimensions and intrinsic weight, with a captive handle design that preferably requires zero grip strength to grasp. Alternatively, the modular extension piece may be coupled to the transponder through a fixed or flexible, segmented, articulated coupling to accommodate attachment of additional transponders and/or other modular extension pieces. These components would quickly assemble to each other using a spherical snap joint or twist snap latch, or similar mechanism, to provide quick alteration of form and function when used for different movement trajectory scenarIos.
In one embodiment, the weighted extension attachments (FIG. 2A) are offered in fixed gradations of one (1) kilogram increments or other convenient unit of measure and either be indicated as such with a numerical label, quantitative mark, or color-code feature, or combination thereof. For upper extremity evaluation, the weighted extension piece integrated into a zero-grip handle would enhance the improvement of musculature strength of the limb, while not compromising the user's endurance with a potentially fatiguing hand grasp requirement.
12 In one embodiment (FIG. 2B), the tactile type provides
force feedback functionality by controlling the rotational speed of an embedded vibrator motor in the shaft. Alternatively, the visual type (FIG. 2C) may be comprised of a series of light emitting diodes that could be uniformly embedded along the length of the handle or transponder and their intensities variably controlled therein. It should be appreciated that a simple, economical mirrored or reflective surface placed in front of the user's visual field could provide sufficient real-
10 time indication of the user's subjective conformity to the said movement trajectory while allowing non-distracting viewing of this visual stimulus. For example, a program that requires the user to reposition the principle interactive transponder through an arc-like movement trajectory in the midsagittal
15 plane through out a range of motion beginning from the waist upwards until parallel to shoulder height. As the user performs the movement, the visual sensory interface could be proportionally excited if the user moves too quickly, or hesitates too long, or produces shaky or erratic episodic motions,
20 or is beyond the prescribed bounds of the movement arc. The light stimulus is easily viewed in the mirror and would indicate corrective action in his or her movement strategy, while appropriate aural commands may be issued simultaneously to encourage the same correction. Regardless of the sensory
25 interface type, its control and excitation properties will be determined by some statistical aspect of the user's conformity to and progression through the movement trajectory.
The hand-held transponder may include a modular extension piece with an embedded graphic display device and
30 associated input means to allow the user to setup, operate, provide visual feedback, and view performance results of the device usage without additional remote display means. More specifically, a software-controlled user interface could provide certain visual prompts in a menu oriented presentation,
35 to instruct the user on (1) device setup, i.e., aural, visual, and tactile feedback parameters, types of program start and termination cues, program intensity based on ratio of amount of repetitions, sets, and rest periods or categorical gradation of challenge, learn mode behavior, etc., (2) scrollable program
40 selection with brief descriptions including objective, desired measurement, i.e., range of motion, energy, accuracy, speed, etc., and instructive information, and (3) alphannmeric and/or graphical display of measured performance data and other biophysical data and its analysis thereof, displayed in stan-
45 dard plotted forms including line, bar, and pie charts, etc. It is important to note that the user input process is intuitive and streamlined so as not to detract from the practicality and user friendliness of the system. Only relevant applications and its control thereof will be sequestered from the database and
50 presented to the user. In one embodiment, two or more transponders and exten
sion pieces, or combinations thereof, may be assembled at their endpoints with a universal spring coupling. The assembled device could be grasped in both hands and bent in
55 various rotational angles about the spring coupling's axis. Isotonic strength conditioning programs can be developed due to the force resistance feedback supplied by the spring. A multi-transponder assembly in the form of a flexible rod or staff could provide an indication of balance of upper extrem-
60 ity strength and proprioceptive function dependent upon the angular closure rate and rotational imbalance and orientation deviation from initial starting position.
Additionally, in the preferred embodiment, the modular extension pieces have provisions for other attachable appara-
65 tus (FIG. 2D) that can augment the program's intensity or difficulty. For example, an eyelet is embedded in the end of the extension piece and is designed to attach an elastomeric
JX-003.0020
A000055
Case: 12-1252 Document: 46 Page: 146 Filed: 08/29/2012
US 7,492,268 B2 13
band, such as the type manufactured by Theraband®. By securing the other open stirrup end of the band to the user's foot, isotonic strength conditioning programs can be developed due to the force resistance feedback supplied by the elastomeric band. Moving the transponder through a movement trajectory is now made more restrictive and challenging.
APPLICATION EXAMPLES
An example training session deploying a dual transponder group is now described that may be designed to improve the range of motion, strength, and coordination of shoulder abduction in a user. The training session would primarily serve as an exercise aid that provides essential feedback to the user so that he/she learns to progressively improve the manipulation of the transponder through the reference movement trajectory, while benefiting from increased shoulder range of motion and strength improvement.
In advance of the training session, a software application is operated from a host computer that provides a utility for baseline configuration and management of the system's and transponder's local databases, and/or access to other remote databases, and for the real-time interface to the data flow between the system's components. The application's navigation and selections are presented to the user through a typical graphical user interface like Microsoft Windows® XP operating system. A generalized step-wise procedure requires the administrator or user to (1) select the desired program and features from a menu screen list, and (2) to initiate a communication process that causes the program parameters to be transferred to the processor unit through a standard computer communication protocol, i.e. serial, USB, ethernet, etc., whereupon, (3) the information is subsequently processed and transferred into the transponders local memory via a wireless communication link, and, finally, (4) the transponder's software program accesses this database to manage the device utilization and configuration of the local display means. Alternatively, a Compact FLASH-based memory card, embedded serial FLASH, or a similar nonvolatile memory device provides the user an additional specialized database supporting remote data collection capabilities. This database would be preprogrammed in advance and the resultant performance data retained, even if the device's power is lost, or for extended unsupervised exercise sessions conducted remotely from the host computer system or when the host computer system is unattached or unavailable. After the session is completed, the user would be queried if the results are to be saved for later analysis or would automatically be saved, dependent upon device setup. This data could be retrieved at a later time when the system is once again attached to a host computer system, and the software utility could be commanded to upload the database.
Henceforth, the following procedural description refers to the activity dependencies diagrammed in FIGS. 3A-3D that the user would encounter while operating the system.
During the Security Phase (FIG. 3A), the user may be requested to provide a security authentication code for validation, which opens access to hislhers custom programs in the training session. Next, during the Setup Phase (FIG. 3A), the user can configure global options or select the desired program. The global options may include, but are not limited to, workload intensity, measurement criteria, sensory interface properties, and reporting features. A program menu list would indicate name, ID, and a brief description, or alternatively, be represented by a detailed graphical icon. When the program is selected, other program-specific options can be setup.
14 During the Deployment Phase (FIG. 3B), and dependent
upon the program's objectives, a suitable combination of transponder types will be mounted on the user's body as instructed by the program. This example requires the assembly of a hand-held interactive transponder with graphical display, and a weighted extension piece coupled therein to be grasped by the hand on the same side as the affected shoulder. Another subordinate transponder 12 is placed into a holster assembly strapped around the lower quadriceps on the same
10 side. This setup is shown in FIG. 4. During the Calibration Phase (FIG. 3C), a simple calibra
tion procedure may be requested to evaluate transponder function and specific user range of motion constraints. Typically, this information is determined beforehand and saved in
15 the system's database. Also, practicality of this system is claimed for lack of extensive calibration requirements.
Dependent upon the program's options, a user-defined movement trajectory may be created prior to program start in lieu of executing the predefined version. The learn mode
20 could be utilized to quickly choreograph free-form movement trajectories and save them into the transponder's nonvolatile memory for later execution. The learn mode would be accessed through the user interface and instruct the management of the control point assignment by pressing the push
25 button switch at the appropriate junctures of movement discontinuity or, preferably, allowing automated assignment by the software. In the preferred embodiment, a computationally efficient Catmull-Rom spline is used to define a three dimensional (3D) curve that passes through all the control points
30 along the movement trajectory path. If manually interceding, the user is instructed to press the push button once at each major juncture in the movement trajectory, but, preferably, for no more than a few locations, until the desired end of range of motion is reached as shown in FIG. 4B. Similarly, the return
35 path may be similarly defined or he/she may elect to use the same forward path in reverse. These control points are registered by the processor unit and transferred and saved to the transponders' memory to serve as the control points for the real-time calculation of a Catmull-Rom spline. The Catmull-
40 Rom spline is calculated in real-time from the desired initial starting point to provide a continuous set of position points representing the "learned" reference movement trajectory.
After the program is selected or the learn mode complete, the user may be instructed to alter the pose of the transponders
45 to satisfY the initial starting conditions of the program. Either one or a combination of sensory interfaces could be excited by the principle transponder to cause the user to direct or steer it towards the desired start point. For instance, the visual sensory interface could sequentially extinguish or dim its
50 peripheral light sources to converge to a central light source as the principle transponder is positioned closer to the desired start point. Alternatively, the aural sensory interface could change its tonality and loudness as the start point is approached. Or alternatively, the tactile sensory interface
55 could be modulated to provide less force feedback as the start point is approached.
During the Execution Phase (FIG. 3D), the transponders are continually manipulated along the reference movement trajectory to the best of the user's skill and fidelity, within the
60 bounds of the user's physical limitation, until an aural, visual, or tactile response is given that indicates the activity volume has been successfully completed or a sufficient number of conformity violations or failures have been registered. The processor unit calculates the instantaneous pose coordinates
65 of the transponders every analysis interval and periodically communicates this information to the transponders via the wireless communication link. As the principle transponder is
JX-003.0021
A000056
Case: 12-1252 Document: 46 Page: 147 Filed: 08/29/2012
US 7,492,268 B2 15
moved in mimicry to the reference movement trajectory the conformity error between the actual and reference movement trajectory is calculated periodically in real-time to determine the characteristics of feedback quality to be elicited by the sensory interfaces for the user's closed -loop control to correct hislher manipulation strategy. For example, the conformity error may be calculated from statistical processes based upon the standard deviation of the least mean squared (LSM) principle transponder's position error compared to the reference movement trajectory, or based upon, or combination thereof, 10
a threshold magnitude of some multi-order numerical differentiation of said movement to indicate a "smoothness" qual-ity of translation and rotation along the movement trajectory path.
16 One testing and training scenario for postural stability
would be to measure frequency and amplitude of body sway in three dimension (3D) space while feet remain in a fixed position. This task can be performed in both a double or single leg stance to test for bilateral symmetry relating to balance. Another modification of the test would be to perform each test with eyes both open and closed to help determine the contribution of the visual component to overall balance ability. Tracking body sway while creating the illusion of motion through proper visual cueing on a display means would be another test to help determine the reliance on specific sensory components of balance. Delivering repetition of protocols with increasing difficult oscillation thresholds with biofeedback of successes and failures of such predetermined goals is
Alternatively, a host computer system could provide an auxiliary processing and display means to allow another software program to access the transponder's calculated positional data through an application programmer's interface and use this data to alter the pose of a graphical primitive in proportion to the motions of the transponders within the context of computer generated virtual enviroument. The dynamic control of objects in the computer generated virtual environment could be used to augment the local sensory interfaces of the transponders through an interactive, goal-oriented video game modality. The video game challenges could be increased over time based upon some scoring criteria of successful manipulation of the principally controlled on-screen graphical object with respect to cueing derived from other secondary static or dynamically moving objects. It is important to note that only primitive forms of video game challenges would be considered, to take into account the user's cognitive awareness and physical limitations, and the economics of software development for photo realistic virtual environments and animation. Also, this auxiliary computer display means would offer an alternative visualization method of interactive and immersive video feedback aid to enhance the application presentation.
15 one way to train to improve balance.
Additional examples of how the present invention may be applied are described as follows:
Balance The body has the ability to maintain balance under both
static and dynamic conditions. In static conditions, such as in standing, the body strives to efficiently maintain posture (often referred to as postural stability) with minimal movement.
The transponder can deliver aural, visual, and tactile stimuli to queue the individual to the degree of frequency and amplitude of body oscillations. The aural and tactile components provide the only means of feedback when the testing
20 and training are performed with eyes closed or the visual field is compromised. Examples include, but are not limited to, (1) an audio signal increasing and decreasing in volume proportional to the amplitude of body sway, (2) a vibration action proportional to frequency of body oscillations, and (3) a light
25 source illuminated when both frequency and amplitude goals are achieved. Multiple transponders can be used to evaluate and reinforce proper balance posture by communicating position information of certain body segments in relationship to others. An example would be the comparison of position of
30 the head with respect to the hips while generating a vibration action if an excessive forward lean of the head as compared to the hips is recognized.
Another test for balance would be to test ones Limits of Stability (LOS). This test refers to ones ability to effectively
35 operate within their sway envelope. The sway envelope or LOS is defined as the maximal angle a person's body can achieve from vertical without losing balance. An individual with healthy balance is capable ofleaning (swaying) within a known sway envelope and recover back to a centered position
40 without the need for a secondary maneuver such as a step, excessive bend at the torso or arm swinging. LOS for bilateral stance in normal adults is 8 degrees anterior, 4 degrees posterior and 8 degrees laterally in both directions.
The present invention described can be used as a testing 45 and training device for balance control during movement or
perturbations within a desired sway envelope. Through proper visual queuing represented on the display means that defines a normal sway envelope, the amount of body displace-
In dynamic conditions such as in walking or sports play, the body strives to maintain balance while performing in an ever changing environment. The ability to maintain balance is a complex process that depends on three major sensory components. The sensory systems include visual, vestibular and 50
proprioception. For example, we rely on our visual system (eyes) to tell us if the environment around us is moving or still;
ment can be measured from vertical stance. The transponder can deliver aural, visual, and tactile
stimuli to queue the individual as to when he or she has achieved the desired range of their sway envelope, then assess the individual's ability to return back to a vertical stance. we rely on our vestibular system (iuner ears) to tell us if we are
upright or leaning, standing still or moving; and we rely on our proprioceptive system (feet and joints) to tell us if the 55
surface we are standing on is uneven or moving. If balance problems develop, they can cause profound disruptions in your daily life. In addition to increased risk for falls, balance disorders can shorten attention span, disrupt normal sleep patterns, cause excessive fatigue, increase dependence on 60
others and reduce quality of life. It is not uncommon for individuals with a history of balance problems to regain their balance control through accurate diagnosis followed by specific medical treatment and/or rehabilitation exercises.
The present invention described can be used as a testing 65
and training device for balance improvement under both static and dynamic conditions.
Examples include, but are not limited to, (1) a vibration action when the user varies (meanders) from the desired movement path, (2) an array oflights change intensity and pattern as the individual successfully approaches the intended target, (3) an audio signal is generated when the individual has maintained a stable position with respect to proper visual queuing represented on the display means for a selected period of time. Multiple transponders can be used to evaluate and reinforce proper balance posture by communicating position information of certain body segments in relationship to others. An example would be the comparison of position of the head with respect to the hips while generating a vibration action if an excessive forward lean of the head as compared to the hips is recognized.
JX-003.0022
A000057
Case: 12-1252 Document: 46 Page: 148 Filed: 08/29/2012
US 7,492,268 B2 17
Dynamic balance can be evaluated while having the individual perform coordinated movements which specifically challenge the various components of balance in a dynamic nature. Such movements include, but are not limited to jumping, hopping, and walking. These movements can be performed with eyes both open and closed, during interaction with static or dynamic visual queuing on the display means. The ability to perform these dynamic balance tasks with comparisons to others of similar sex, age or disability can be assessed. Example measurements may include, but are not 10
limited to, (1) amount of body sway in three dimension (3D) space, (2) time to complete specific task, and (3) effects of fatigue on balance ability.
Balance training in both static and dynamic conditions can be easily achieved by providing specific visual queuing on the 15
display means, which challenge the individual to perform repetitive and progressively more difficult balance drills. Performance reports can be generated to establish a baseline, isolate specific strengths and weaknesses within the specific sensory and motor control aspects of balance, and document 20
progression and improvements. The transponder can deliver aural, visual, and tactile
stimuli to queue the individual as to when he or she has achieved the desired balance task. By example, a vibration action is produced proportional to the frequency of a body 25
segment oscillation after the user lands from a hop test and attempts to stabilize and maintain proper postural balance. When the individual finally stabilizes and achieves correct postural balance, an audio signal indicates the task has successfully completed. Multiple transponders can be used to 30
evaluate and reinforce proper balance posture by communicating position information of certain body segments in relationship to others. An example would be the comparison of position of the head with respect to the hips while generating a vibration action if an excessive forward lean of the head as 35
compared to the hips is recognized.
Range of Motion (ROM)
18 vidual's movement trajectory varied from the intended two dimensional (2D) reference movement trajectory by deviation from the planar path into the uninvolved spatial dimension. An array oflight sources could increase illumination in intensity and repetition as the ROM goal was approached and an audio tone could signal the individual they have achieved the desired pause time at the proper ROM.
Multiple transponders can be deployed to determine the contribution of each joint or anatomical structure where more then one joint is involved in the ROM movement (example; shoulder and scapular in overhead reaching). The vector sum of each transponder movement in a specific axis can be added together to determine the total ROM. The ROM of one j oint in a two joint motion can be subtracted from the total ROM to determine the contribution of a single joint in a two joint movement.
Human Performance Testing and Training
There are many devices that test the strength and speed of isolated joint movements, for example, the leg extension and bicep curl. This information has value in testing both healthy individuals, athletes and individuals whose strength and speed capabilities may be compromised by injury, disease, poor conditioning or simply age. Recently in the field of human performance, it has been recognized that the analysis of the mobility of the isolated joint, although providing some value, does not offer enough information to determine how the body will perform during functional movements. Functional movements are defined as movements equal to those encountered on the athletic field, in the work environment or while performing activities of daily living. Functional movements involve the movement and coordination of multiple joints and muscle groups acting together to perform a more complex task then a single, isolated joint movement.
The present invention described can be used as a testing and training device for functional movement improvement. By tracking various registration points on the body with The present invention described can be used as a testing
and training device to determine the range of motion within a joint. Range of Motion is the normal distance and direction through which a joint can move. Limited ROM is a relative term indicating that a specific joint or body part cannot move through its normal and full ROM. Motion may be limited by
40 respect to each other or to an off-body registration point, performance measurements of functional movements can be assessed, such as jumping, cutting, turning, bounding, hopping, shuttling, etc.
a mechanical problem within the joint that prevents it from 45
moving beyond a certain point, by swelling of tissue around the joint, by spasticity of the muscles, or by pain. Diseases that prevent a joint from fully extending, over time may produce contracture deformities, causing permanent inability
The present invention described can be used as a testing and training device for individuals involved in physical rehabilitation, general fitness or sports performance enhancement to improve their functional movement abilities. Proper visual queuing can be represented on the display means to instruct and motivate individuals to perform specific functional move-
to extend the joint beyond a certain fixed position. 50 ments. The present invention described can be used to test the
starting point and end point which an individual is capable of moving a body part, typically a limb and associatedjoint(s). Comparisons to age and sex based normative data can be made. Proper visual queuing can be represented on the dis- 55
play means to instruct and motivate the individual through the proper testing procedure.
The present invention described can be used as a testing and training device for individuals involved in physical rehabilitation or general fitness to improve ROM. Proper visual 60
queuing can be represented on a display means to motivate individuals to extend their range of motion beyond their current capabilities.
The transponder can deliver aural, visual, and tactile feedback that alerts the individual to successes or failures in 65
proper execution of each repetition. An example of tactile feedback would be the transponders are vibrated if the indi-
The transponder can deliver aural, visual, and tactile feedback of proper movement execution. Examples include, but are not limited to, (1) an audio signal alerting the user that the desired performance stance is incorrect, (2) the light sources illuminate when the desired speed is achieved in a first step quickness drill, (3) a vibration action to indicate the limits of tracking range, (4) a vibration action proportional to the magnitude of a biophysical measurement during the interaction with visual queues represented on the display means, (5) a vibration action when the body or limb position does not correlate well to the desired body or limb position of the visual queuing represented on the display means, (6) an audio signal indicating start, stop and pause periods or other controlling events, (7) an audio signal indicating proper body alignment or posture has been compromised, and (8) an audio signal indicating the relationship of desired target heart rate to a desired threshold.
JX-003.0023
A000058
Case: 12-1252 Document: 46 Page: 149 Filed: 08/29/2012
US 7,492,268 B2 19
Hardware Description In the preferred embodiment, the processor unit is com
prised principally of a constellation of five (5) ultrasonic transducers and signal processing circuitry, thereof, and a signal processor that interfaces to this receiver group, performs the pose calculations, and interfaces to the transponders and host computer databases. The following interface descriptions for the processor unit are based upon the dependency flow represented by FIG. 5.
The sensors 14 preferably used for the receiver constella- 10
tion unit are cylindrically-shaped ultrasonic transducers, for example, the model US40KR-0l 40 kHz PVDF ultrasonic receivers manufactured by Measurement Specialties Inc., which provide adequate acoustic pressure sensitivity and exhibit 360 degree onmidirectional broad beam response 15
along the horizontal plane. The onmidirectional characteristic, albeit in one plane only, is very desirable to minimize line-of-sight occlusion. Because of its low resonance Q value, the rising and decay times are much faster than conventional ceramic transmitters. This reduces its power requirements 20
since less burst drive duration is needed to achieve sufficient triggering thresholds at the receiver. This transducer type is also utilized similarly in the transponders to provide the potential for the most optimal acoustic coupling.
The receiver constellation unit is preferably mounted on a 25
fixed support base, and has a pivoting and/or swiveling mechanical linkage which provides an adjustable mechanism for configuration of the receiver constellation unit's inertial frame of reference relative to the tracking field. In the preferred embodiment, it is strategically positioned and oriented 30
in proximity to the tracking field in order (1) to minimize line-of-sight degradation with respect to the expected transponder orientation, (2) to optimize registration resolution with respect to field volume size, and (3) to satisfy the mathematical restrictions of performing trilateration calculations 35
based upon the solution of simultaneous linear equations. It should be noted that the trilateration matrices may be solved if the matrices have a rank of five, and are non-singular, i.e., the matrix determinant is non-zero. In the preferred embodiment, the geometric parameters and their coordinate location 40
of the receiver constellation must insure linear independence of the colunms of the matrices and to avoid the matrices from becoming singular.
One example geometrical permutation of the receiver constellation unit that satisfies these rules is shown in FIG. 6A. It 45
20 trilateration requires the precise resolution of the common intersection of multiple spheres circumscribed by the distance between each transmitter and receiver transducer. Each sphere has an inexact radius due to system noise and measurement resolution. Therefore, the intersection becomes a volume instead of a point and the size of the volume is dependent upon the radii of the intersecting spheres as well as the distance between the spheres' centers. As the radii get larger with respect to the distance between the centers, i.e., the transmitter is farther down range, the spheres begin to appear more and more tangential to one another and the intersection volume increases, although not necessarily symmetrical in all dimensions. Therefore, to minimize position uncertainty, the receiver transducers should be separated from each other as much as practical proportions allow with respect to the confines of the tracking field volume as the above said geometric examples provide.
This receiver constellation unit can be repositioned with respect to the tracking field by a simple mechanical adjustment as shown in the preceding figures. The mechanical adjustment raises and lowers and changes the length and pivot axis of the cantilever arm which is fixed to a ground base support.
Because the receiver constellation unit operates a distance from the processor unit, each receiver preferably has an associated pre-amplifier circuit to convert the high-input impedance piezoelectric signal into a low-level voltage proportional to the acoustic signal energy impinging the transducers sufficient in order to accurately transmit the signals to the pro-cessor unit. In one embodiment, a high-input impedance AC amplifier design with 30 dB gain can be utilized. The pre-ferred operational amplifier is the OPA373 manufactured by Texas Instruments. It was chosen for its low 1 pA input bias current, high 6 MHz GBW, and low-voltage single supply operation. The amplifier is configured as a non-inverting type with the high-pass cutoff frequency set at 1 kHz. The overall circuitry is preferably enclosed in a metal shield to minimize electromagnetic noise coupling into the highly sensitive amplifier inputs. In addition, a local, regulated power supply is included to allow for a wide range of input voltage supply and provide sufficient power supply rejection to compensate for the noise susceptibility of remote power distribution. All the pre-amplifier circuits' power and signal connections preferably originate from the processor unit.
The processor unit subsystem preferably consists of an analog signal processing interface that provides (1) additional voltage amplification and filtering of base band signal from the preamplifiers, (2) absolute value function, (3) peak detection function, and (4) analog-to-digital comparator function
occupies a volume of approximately 8 cu. ft. and essentially fixes the transducers in a way that defines two primary orthogonal, bisecting planes defined by three non-collinear points each. Another preferred implementation that occupies nearly the same volume is shown in FIG. 6B and is characterized by its S-shaped curve and tilted with respect to the horizontal plane. Another preferred implementation that occupies nearly the same volume is shown in FIG. 6C and is characterized by its helical or logarithmic spiral shape oriented perpendicular to the horizontal plane. Further, as indicated in the preceding figures, the transducers vertical axes are oriented 90° with respect to the typical vertical axis orientation of the transponder's transmitter to improve acoustic coupling in the vertical plane, a consideration for overhead, upper extremity tracking. Although this causes some reduction in the lateral registration bounds, the compromise provides a more symmetric field about the middle or primary location of tracking interest.
50 to provide support for an adaptive threshold means. The adaptive threshold technique provides robust triggering of the most proximal ultrasonic source at a precise temporal point along the traversing sinusoidal waveform of the electrical signal. Essentially, a new threshold signal is recalculated each
In the preferred embodiment, the overall size of the receiver constellation unit is predicated on a phenomenon referred to as Geometric Dilution of Precision (GDOP). The solution of a unique three-dimensional location based upon
55 analysis period based upon a small percentage reduction of the last peak waveform detected. Therefore, the tracking range is not necessarily restricted due to an arbitrarily high threshold setting and the noise immunity is improved as the threshold tracks the waveform envelope and not transient
60 disturbances. An alternative automatic gain control strategy for the amplification function is unnecessary since the trigger threshold will adjust to the signal level instead. In the preferred embodiment, the threshold faithfully tracks the peak to minimize integer period phase errors, so the amplifier's gain
65 is set to prevent signal saturation from occurring when the receiver constellation unit and transponders are in closest proximity during normal use.
JX-003.0024
A000059
Case: 12-1252 Document: 46 Page: 150 Filed: 08/29/2012
US 7,492,268 B2 21
In one embodiment, an amplifier and BW (band width) filter circuit receives the output from the sensor and preamplifier circuit and provides additional amplification and lowpass filtering to condition it for reliable threshold triggering and input to other analog signal processing circuitry. A dual amplifier configuration may be used to provide an additional gain of 40 dB, AC coupling to remove DC offsets of the preamplifier outputs and long cable losses, and low-pass filter to reject noise beyond the interest signal's bandwidth. The first stage amplifier may be configured as a non-inverting type with a gain of 20 dB. The low-impedance DC input signal is effectively blocked by the coupling capacitor in series at its non-inverting input with a high-pass frequency cutoff set at 20 kHz. This gain stage feeds a second amplifier configured as low-pass, 2nd order Butterworth MFB filter. This filter type provides smooth pass band response and reduced sensitivity to component tolerances. The second stage low-pass frequency cutoff is set at 80 kHz with a pass band gain of20 dB.
An absolute value circuit receives the output of the amplifier and BW filter circuit and converts the bipolar signal into a unipolar form for magnitude detection. A dual amplifier configuration may be used to provide highly accurate full wave rectification of the millivolt-level signal. The first stage amplifier feedback switches to control the distribution of input current between the two signal paths dependent upon the input signal polarity. For a positive input voltage the input current will be positive which forward biases Dl and reverse biases D2. This configures the 1st stage as an inverter driving the inverting input resistor of the 2nd stage, which is also configured as an inverter because its non-inverting input is held at virtual ground due to the non-conducting path ofD2. This effectively creates a combined circuit of two cascaded inverters for an overall gain of + 1. For a negative input signal
22 In the preferred embodiment, a comparator circuit receives
the output from the peak detect and sample-hold circuit to convert the analog signal to digital form for high-speed triggering operation of the processor. The preferred device is the MAX941 which is manufactured by Maxim. A percentage of the peak threshold is used to set the inverting input. When the non-inverting voltage exceeds the inverting voltage, the comparator's output will trip and produce a high-true logic pulse that triggers the processor. A latch control input allows the
10 processor to disable the comparator action to prevent unnecessary triggering during the reverberation phase and to prevent potentially disruptive noisy output chattering near threshold crossover beyond its hysteretic immunity. The percentage of threshold level is predetermined through the scal-
15 ing resistors to be set low enough to trigger on the rising edge of the signal's first crest at the furthest range of transponder operation, but high enough above the intrinsic system noise level and external noise caused by reverberation and other ultrasonic sources. Once the first crest is registered, subse-
20 quent crests may be triggered at their zero-crossing representing the most precise timing registration by momentarily disabling the sample-hold circuit. Because of the longer duration trigger receptivity window, early multiple reflections are mitigated by transducer placement at least 3.5 cm away from
25 adjacent planar surfaces, so the reflected acoustic energy doesn't produce a canceling effect of the direct acoustic energy of the later crests. Once a sufficient number of crests have been registered, then the triggering window is blanked for the remainder of the analysis period by latching the com-
30 parator's value. In the preferred embodiment, a digital signal processing
interface is connected to the analog signal processing interface to transform the analog trigger processing into digital position information.
The digital filter circuit receives output from the compara-tor circuit and preferably consists of a digital low-pass filter implemented in a complex programmable logic device (CPLD) that serves to precondition the comparator circuit's digital outputs. The preferred device is an AT1504ASVL
its input current is negative which forward biases D2 and 35
reverse biases Dl. This configures the 1st stage as an inverter driving the non-inverting input of the 2nd stage which changes the sign of the circuit gain. In this mode, the input current is shared between two paths to the input of the 2nd stage, where _2/3 of the input current flows around the 1 st feedback stage and -1f3 flows in the opposite path around the 2nd stage feedback path for a net gain of -1.
40 CPLD which is manufactured by Atmel. Base band system noise or other glitches potentially occurring in the analog signal processor interface, but prior to the actually arrival of the ultrasonic signal, could cause a threshold disruption that registers a "runt" pulse as a false trigger condition. The "runt"
In the preferred embodiment, a peak detect and samplehold circuit receives the output of the absolute value circuit and registers a peak value that is required to set a magnitude threshold precisely at some percentage of full-scale of the peak. A dual amplifier configuration may be used to provide the highest ratio of high output slew rate to low droop. The first stage is typically in negative saturation until the input voltage rises and exceeds the peak previously stored on the sample capacitor at the inverting input. Now the amplifier acts
45 pulse would be misinterpreted as the actual TOF trigger and cause serious error in the position calculation. AnANDINOR one-hot state machine design may be used to ignore level transitions that are not stable for at least Ih system clock frequencyx8 states, so only transitions of 4 flS or greater are
50 passed through. The system clock delays introduced by the digital filter's synchronous state machine affect all channels the same and are, therefore, effectively eliminated by the inherent dependency on relative measurement.
as a unity gain buffer and the input voltage charges the sample capacitor which faithfully tracks the rising voltage. Once the input voltage diminishes in magnitude, the first blocking diode reverse biases and the sample capacitor holds an accurate replica of the highest voltage attained with minimal droop because of the low input bias current of the amplifier and elimination ofleakage altogether in the second blocking diode by bootstrapping its cathode at the same potential provided by the low-impedance buffer of the second output stage. An electronic switch and bleed resistor allow the voltage across the sample capacitor to be reset by the processor during power up and after the triggering event is recorded so the adaptive threshold value can be refreshed each cycle. A 1 st
order Butterworth filter may be used at the input to smooth 65
false in-band transients that could disrupt the peak accuracy detection.
In the preferred embodiment, the processor and digital 55 filter circuits receive the output from the analog processor and
provide controlling signals therein. The preferred processor circuit is the MC9S08GB60 which is manufactured by Motorola Inc. It is a low-cost, high-performance 8-bit microcontroller device that provides all the aforementioned hard-
60 ware circuits integrated into one convenient device. The calculation circuit is abstracted from embedded 60 KB FLASH for program memory with in -circuit programmable capability and 4 KB RAM for data memory. The time base circuit is preferably comprised of an external, high-noise immunity, 4.0 MHz system clock, which multiplies this by the internal frequency-locked loop for a bus clock of 40.0 MHz and single instruction execution time of251lS. This clock also provides
JX-003.0025
A000060
Case: 12-1252 Document: 46 Page: 151 Filed: 08/29/2012
US 7,492,268 B2 23
all the capture and control timing functionality for the other specified circuits. Multiple parallel I/O ports and dedicated asynchronous serial communication signals provide for the digital control of the analog signal processing and communication interfaces, respectively.
24 power output, etc. or get infonnation about parameters such as battery, PLL lock state, etc. In nonnal mode, any data entering its input channel is immediately radiated or any desired signal collected by the aerial is demodulated and transferred to the microprocessor as reshaped register bit information. In wake-up mode, the device periodically scans for an expected message sequence and broadcasts an interrupt if a correct message is detected.
In the preferred embodiment, at least three (3) consecutive TOF timestamps are registered for each receiver during the acquisition phase. Preferably, the transponder's transducer emits a multi-cycle ultrasonic acoustic burst of at least ten cycles in duration so that sufficient energization of the receiver transducer is realized and at least three crests of the
The timing capture-control circuit receives the output from the digital filter circuit representing the arrival of the TOF triggers to detennine the relative TOF propagation of the ultrasonic acoustic wave as it passes through the receiver constellation unit. More specifically, it is comprised of a five 10
channel l6-bit timer input capture module with programmable interrupt control that provides edge detection and 50 11 s timing precision to automatically register the TOF triggers timestamps asynchronously without using inefficient and less accurate software polling means. 15 waveform can be properly registered. At low signal levels
when ultrasonic acoustic coupling is poor, this requirement may fail and an invalid tracking status is asserted. Preferably, the reference receiver transducer of the receiver constellation
The phase-locked loop circuit receives the output from the timing capture-control circuit and is preferably comprised of a three channel, l6-bit timer compare module is implemented as an all-digital phase locked loop (ADPLL), which synchronizes the capture window and blanking functions with respect 20
to the reference input channel. It is comprised primarily of a free-running l6-bit timer configured to periodically interrupt the processor dependent upon a precise convergence of its period and phase to the reference trigger source, by means of
unit is positioned in closest proximity to the acoustic signal source so that it is the first transducer to be affected by the initial wave front. This reference receiver provides the overall system timing and state machine control for the phase-locked loop circuit, so that the processing, calculation, and communication tasks are executed in a deterministic and efficient
an over/under count matching and correction technique. 25 fashion. The AID conversion circuit receives the output from the
amplifier and BW filter circuit and consists of an eight channell O-bit analog-to-digital converter used to monitor channel offsets and magnitudes for range and polarity errors and correction. This infonnation is utilized by the calculation 30
circuit as input to the TOF software correction algorithm to determine the slope of the wavefonn crest.
It should be appreciated that a high-resolution ultrasonic acoustic tracking system that depends upon threshold detection means has an inherent uncertain trigger dilemma. This uncertainty arises because of the multi-cycle nature of the transmitted signal's wavefonn and the associated difficulty detecting the exact temporal location for consecutive analysis periods when the signal's magnitude may vary greatly depending upon the efficiency of the acoustic coupling, the distance between transmitter and receiver, and signal-tonoise ratio of the signal processing techniques. If a threshold is set near one of the minor crests of the wavefonn during the
In the preferred embodiment, the serial communication circuit is comprised of two asynchronous serial communication interfaces that are connected between the calculation 35
circuit and host link and radio link circuits of the communication interface. The host link provides a l15K bit per second (baud) bi -directional communication link to an auxiliary host computer system through a Serial-to-Universal Serial Bus bridge. The preferred device is the CP210l which is manufactured by Silicon Laboratories. It supports the conversion of a fully asynchronous serial data bus protocol, with buffering and handshaking support, to an integrated Universal Serial Bus (USB) Function Controller and Transceiver and internal clock providing USB 2.0 full-speed compliancy. An integrated 512 bit EEPROM stores the required USB device descriptors, including the Vendor ID, Product ID, Serial Number, Power Descriptor, Release number and Product Description strings. A host computer may enumerate and access this device utilizing the manufacturer's virtual COM port device drivers using a USB channel.
In the preferred embodiment, the radio link circuit is comprised of a wireless bi-directional communication interface to preferably (1) broadcast a synchronization signal to control the transponders interoperability, (2) to receive other transponder sensor data, including, but not limited to, accelerometer, heart rate, battery, user I/O status, (3) to provide control messages for the transponders' sensory interfaces, and (4) to provide means to configure transponders' local databases. The preferred wireless communication link is based upon the AT86RF2ll, a highly integrated, low-power FSK transceiver optimized for license-free ISM band operations from 400 MHz to 950 MHz. and manufactured by Atmel. It supports data rates up to 64 kbps with data clock recovery and no Manchester Encoding required. The device has a three wire microprocessor interface that allows access of read/write registers to setup the frequency selection, transmission mode,
last analysis period, then it is conceivable a slight reduction of magnitude of the wavefonn during the next analysis period may fall slightly below the set threshold and actually not be
40 triggered until the next larger excursion of the wavefonn occurs. This would create a TOF error proportional to the period of the acoustic wavefonn or its intra-pulse interval and have a detrimental affect on the measurement accuracy. This analog processing described above establishes trigger thresh-
45 olds that allow no more than a single intra-pulse interval of uncertainty, but that is still inadequate for high-resolution measurements. Although a technique is known that controls the largest peak profile of the transmitter acoustic signal and claims to provide an absolute trigger condition, this proce-
50 dure is difficult to reliably tune and control among different transducer types.
In the preferred embodiment of the invention, no modulation of the acoustic signal is required. Rather, the adaptive threshold method is augmented with a TOF software correc-
55 tion algorithm that unambiguously determines the correct TOF based upon a means to detect the same carrier wave cycle of ultrasonic energy every period. The software correction algorithm requires multiple, consecutive TOF acquisitions as input for the digital over-sampling and averaging
60 algorithm, the calculation of a higher-order numerical differentiation of the past and current TOF information as input for the predictive algorithm of higher-order Taylor series based derivatives used for the relative TOF correction, and a measurement of the intra-pulse time intervals of consecutive TOF
65 acquisitions as input for the absolute TOF correction scheme that minimizes the range error based upon selective biasing of the TOFs.
JX-003.0026
A000061
Case: 12-1252 Document: 46 Page: 152 Filed: 08/29/2012
US 7,492,268 B2 25
The calculation circuit preferably processes multiple, consecutive TOF acquisitions to effectively improve the timing resolution that proportionally affects position accuracy and precision. The digital filter discussed above introduces quantization errors because of its discrete operation. And minor fluctuations in the acoustical coupling produces timing jitter or uncertainty in the triggered zero-crossings of the acoustic sinusoidal. A Gaussian average or mean value of multiple TOF is a simple and effective filter strategy. Due to the possibility of poor acoustic coupling or misalignment, and dis- 10
tant transponder location from the processor unit, the nnmber of detectable triggered zero-crossings may vary for a fixed duration of multi-cycle ultrasonic acoustic burst. The averaging algorithm automatically adjusts to this condition by only including TOFs whose delta changes fall within the expected 15
range of the nominal intra-pulse interval defined by the transmission properties of the acoustic source. The nominal intrapulse interval is determined and utilized by the following compensation schemes.
The calculation circuit preferably processes a relative TOF 20
correction algorithm based upon a predictive tuned algorithm that requires higher-order numerical differentiation calculation of the past and current TOFs. This compensates the TOFs that may have registered one intra-pulse interval earlier or later than the nominally expected time due to the trigger 25
dilemma described above. By formulating these derivatives into a truncated 2nd order Taylor series expansion and weighting the terms contribution, an estimate of expected TOF is calculated and compared to the actual TOF through an iterative error minimization calculation. A minimized error that 30
results in a delta time change indicative of a discrete intrapulse interval increase or decrease due to an early or late TO F, respectively, produces a characteristic value that directs the algorithm to compensate the actual TOF by the intra-pulse interval and restore it to its correct value. In the preferred 35
embodiment, this relative compensation algorithm works most effectively when, (1) the maximally expected interperiod TOF change is less than the discrete intra-pulse interval, (2) the TOF inter-period processing is contiguous, (3) the TOF increase or decrease is no more than a single intra-pulse 40
interval, and (3) the Taylor series terms are suitably weighted in the prediction algorithm.
26 ferred, embodiment, this absolute compensation algorithm works most effectively when (1) the wireless synchronization means is tightly coupled to the excitation of the acoustic source, (2) the synchronizing signal's arrival is timed by the same mechanism that times the arrival of the reference transducer's acoustic signal, and (3) the coordinate locations of the sensors of the receiver constellation are established to a high degree of accuracy.
The calculation circuit preferably employs two software methods of trilateration calculation to estimate transponder position, wherein the particular method used depends upon the availability of a synchronizing signal and the accuracy desired. The first method is based on a relative TOF calculation and the speed of sound is treated as a constant estimated at ambient indoor room temperature. The second method requires calculation of an additional TOF timestamp between the transponder and reference receiver, but calculates the speed of sound as an unknown every analysis period, and thus improves measurement accuracy. The first method eliminates the global system timing variances and delays due to the multiplicity of signal conditioning circuitry and eliminates the need for a controlling signal means synchronized at the generation of the transmission of the ultrasonic acoustic wave. The second method also employs relative TOF calculation but requires an additional synchronization signal from the processor unit to determine the absolute TOF between transponder and reference receiver. Since the absolute TOF is based upon a single channel only, its timing latencies can be readily accounted for and easily corrected. This method computes the speed of sound every analysis period, provided the synchronization signal is detected, without need for additional hardware temperature processing or requiring more then five (5) receivers, and automatically accounts for the system's main accuracy limitation of speed of sound in air as defined by Eq. 1.1, if uncorrected, yields a 1.6 mmlmranging error for every 10 C. temperature shift. If the synchronization signal is not detected and, therefore, the second method is not resolvable, the last calculated speed of sound can be utilized within the first method's calculation to minimize error.
c=34.6 mis+0.5813 m/s(Tc-25C C.) (1.1)
The calculation circuit preferably processes an absolute TOF correction algorithm at least once initially, when the phase-locked loop is stable, but may be performed every 45
analysis period depending on computational resources, that determines the initial set ofTOF values for the relative correction algorithm. The initial condition that precedes the start
The TOF timestamps and speed of sound values are input into linear independent algebraic equations in a matrix formulation to solve for the unknown transponder(s) position, in
50 a form as shown in Eq. 2.1, of the relative compensation algorithm may be due to the resnmption of a stable, locked tracking state after recovery from a fault condition and, therefore, requires computation of
55
a set of reference TOFs producing minimum range error as a starting basis. The algorithm utilizes a wireless synchronization means to determine a reference TOF calculation between the transponder and reference sensor of the receiver constellation. By computing the reference range distance by the product of the reference TOF and speed of sound in air, this reference range may be compared to the range calculated from the matrices solutions described below. By iteratively and sequential increasing and decreasing the TO F s by a single 60
intra-pulse time interval and applying the input to matrices formulations described below, all possible combinations of compensation are permutated and tested, which produces a unique set ofTOFs that minimize the error between the calculated range distance with respect to the reference range 65
distance. This unique set of initial TOFs serves as the starting basis for the relative compensation algorithm. In the pre-
all al2 a13 al4 Xl bl (2.1)
A·X =B A= a2l a22 a23 a24
X= X2
B= b2
a31 a32 a33 a34 X3 b3
G41 a42 G43 G44 X4 b4
To solve for the unknowns X, Eq. 2.1 is rearranged as shown in Eq. 3.1, whereas the inverse of A requires computation of the cofactor matrix AC for the adjoint and determinant calculations for Eq. 3.2 and Eq. 3.3, respectively,
(3.1)
JX-003.0027
A000062
Case: 12-1252 Document: 46 Page: 153 Filed: 08/29/2012
US 7,492,268 B2 27
-continued All A2l A3l A4l (3.2)
(A'l = Al2 A22 A32 A42
Al3 A23 A33 A43
A14 A24 A34 A44
(3.3) 10
To setup the coefficient matrix A, the utilization of five (5) receivers produces the following set of relative TOF equations defined by Eqs. 4.1-4,
Ll.T12=Tr Tl (4.1)
Ll.T13=TrTl (4.2)
Ll.T14=T4-Tl (4.3)
Ll.TlS=Ts-Tl (404)
15
20
The receiver locations are fixed within the system's inertial reference frame, while the transponder(s) are mobile with 25
respect to the same reference frame and are defined as follows,
S(Xi'Yi,zJfor 5~i~1 =>fixed receiver locations
30 S(xo,Yo,zo)=S(u,v,w)~>unknown transponder location
Since each receiver is fixed at a distance Di from the transponder as determined by the receiver constellation geometry and because the acoustic waves propagate spherically, by 35
using Pythagorean's theorem the following set of range equations are defined in Eqs. 5.1-5,
(x l-U)2 +(Yl-V)2 +(z 1-w)2~D12 (5.1)
(xru)2+(yrv)2+(zrw)2~D22 (5.2) 40
(xru)2+(yrv)2+(Z3-W)2~D32 (5.3)
(X4-U)2+(y4-V)2+(Z4-W)2~Dl (504) 45
(xs-u )2+(yS-V)2+(ZS-w)2~Ds 2 (5.5)
Equivocally, the four (4) non-reference receivers are preferably located at an incremental distance relative to the reference receiver, so by substitution of the incremental distance 50
defined by Eq. 6.1, the following set of relativistic range equations are defined by Eqs. 6.2-5,
Di=Dl+C~Tll for 5~i~2 (6.1)
55
(X2-u)2+(y2-V)2+(zrw)2~(Dl+cLl.Td2 (6.2)
(xru)2+(yrv)2+(zrw)2~(Dl+cLl.T13)2 (6.3)
(X4-U)2 +(y 4-V)2 +(Z4-w)2~(Dl +CLl.T14)2 (604) 60
(xs-u )2+(yS-V)2+(ZS-w)2~(Dl +cLl.TlS )2 (6.5)
28
Xl -X2 Yl - Y2 Zl - Z2 -c!:J.T12
2 Xl -X3 Yl - Y3 Zl -Z3 -c!:J.Tl3
Xl -X4 Yl - Y4 Zl - Z4 -CLl.T14
Xl -XS Yl - Ys Zl -ZS -CLl.TlS
where R?=x/+y?+z? for 5~i~1
U
W
Dl
(cLl.T12)2 + Ri - R~
(cLl.Tl3)2 + Ri - R~
(cLl.T14)2 + Ri - R~
(cLl.Tld + Ri - R~
(7.1)
(7.2)
Alternatively, if the second method algorithm is used, the unknown range of the reference receiver D 1 can be substituted by Eq. 8.1,
Dl =cTOl> Ll.Tol~>time of flight (TOF) from S(u,v,w,) to S(XVYi,Zl) (8.1)
And, by rearranging terms, it is depicted in the matrix form defined by Eq. 9.1,
a.
Xl -X2 Yl - Y2 Zl -Z2 -( Ll.TOl Ll.T 12 + 0.5Ll.Tf2) u R2 1 (9.1)
Xl -X3 Yl - Y3 Zl -Z3 -(Ll.TOlLl.Tl3 + 0.5Ll.Tf3) R2 1 2
Xl -X4 Yl - Y4 Zl -Z4 -(Ll.TOlLl.T14 + 0.5Ll.Tf4) W R2 1
-(Ll.TOlLl.TlS + 0.5Ll.Tfs) c2 R2 1 Xl -Xs Yl - Ys Zl -Zs
Although similar results may be obtained by application of more computational efficient processes such as pivotal condensation or Crout's decomposition, the application of Cramer's rule was used to evaluate the first-order determinant in Eq. 3.3 using second-order determinants from Laplace expansion. The final transponder(s) position equations are defined by Eqs. 10.1-8.
IA21 u=w IA21
V=-IAI
(10.1)
(l0.2)
(l0.3)
(lOA)
(10.5)
(10.6)
(10.7)
(10.8)
By expanding and rearranging the terms ofEqs. 6.2-5, a set of four linear algebraic equations and four unknowns for the first method algorithm, depicted in the matrix form ofEq. 2.1, is defined by Eq. 7.1,
If the first method is used, D, the range of the transponder to the reference receiver from Eq. 10.4 may be calculated as a
65 redundant confirmation of the Eqs. 10.1-3 calculations, provided the frame of reference origin and location of the reference receiver are identical or their offsets accounted for. If the
JX-003.0028
A000063
Case: 12-1252 Document: 46 Page: 154 Filed: 08/29/2012
US 7,492,268 B2 29
second method is used, C, the speed of sound in air, from Eq. 10.4 must be computed every analysis period if its value is anticipated to be used in the first method in the absence of a synchronization signal.
The orientation of the transponders can be derived from a similar utilization of the above algorithms for a transponder configured with a triad of ultrasonic transmitters. The transducers are preferably arranged in a triangular plane at the transponder of sufficient area for the desired angular resolution. The sequential excitation of each transducer and subsequent calculation of position by the aforementioned methods provides sufficient information to detennine orientation by the inverse kinematic calculations of Eqs. 11.1-4, where the analysis is simplified by assuming the origin of rotations occurs about Tl and Tl23 represents the initial relative position matrix from this origin and T\23 is the transfonned or forward kinematic position matrix.
(11.1)
X2COS()y + Z2sinBy
sin8Ax2sin8y - z2cos8y)
-cos8x(x2sin8y - z2cos8y)
X3cos8y [(11.2)
x3sinBxsinBy
-x3cosOxsinBy
[
Xl cos(h + Yl sinB2
- Xl sinB2 + Yl cos(h
z;
X2 cos(h + Y2 sinB2
- x2 sinB2 + Y2 eosB2
Z2
X3 eose2 + Y3 sinB2 [
- x3 sinB2 + Y3 eose2
Z,
(11.3)
(1104)
By examination of the matrices element equivalency of Eqs. 11.2-3 and manipulation of terms so that the angles may be found using the inverse tangent function, the following rotation equations Eqs. 12.1-3 are derived,
30 functions may be evaluated through a conventional look-up table or by a power series expansion.
Preferably, the overall analysis period duration is effectively trebled until the three (3) transducers' positions are calculated, which reduces the system's frequency response and imposes an increased latency effect. Typically, robust absolute orientation processing requires more stringent lineof-sight operation and is reserved for more sensitive, less dynamic, and reduced ROM movement trajectories, e.g., bal-
10 ance and sway. Therefore, the latency effect is less noticeable upon the real-time perfonnance of the sensory interfaces.
In the preferred embodiment, the interactive hand-held transponders support a dual axis inertial sensor, which is operably configured to provide tilt (pitch and roll) orientation
15 in its horizontal mounting plane. The inertial sensor is mounted in the intended operational horizontal plane with respect to the systems inertial frame of reference. Once the sensors signals has been converted to an acceleration value that varies between +/-1 g the tilt in degrees is calculated as
20 shown in Eqs. 13.1-2, for pitch and roll, respectively.
25
<I>~a sin(A)l g) (13.1)
<I>~a sin(A)l g) (13.2)
This outside-in ultrasonic tracking implementation, where the transponders are mounted on the mobile object, produces inherent temporal delays due to the finite TOF registration and calculation delays after the transponder has already moved into a different position before the measurement is
30 complete. This overall latency period is compensated and minimized through use of a Kalman filter data processing algorithm to estimate the pose of the transponder by optimally and recursively combining past history, new measurements, and a priori models and information. Generally speak-
35 ing, the Kalman filter is a digital filter with time-varying gains that are optimally determined through a stochastic dynamical model of the motion. The overall goal is to minimize filter lag while providing sufficient smoothing of the motion data.
(12.1) 40
An adaptive, multi dynamic model is developed based upon the kinematic quality of the expected movement trajectory. The predictive kinematic model for the Kalman filter is depicted in matrix fonn utilizing a truncated 2nd order Taylor series expansion as below in Eqs. 14.1-2, ( -z, ) By = atan , , .
cos8x(x3cos82 + Y3 sm8,)
(12.2)
(
, x3sin81 sin8y ) Y3 - -c-O--;S8;-23----'-
(}z = atan " x3
(12.3)
COS 82 _ 1 =cos 82 from previous iteration (1204)
82 = atan(~) for 1st iteration (12.5)
These calculations are perfonned through iterative step processes which inherit angular approximations of the preceding steps until the final desired angular accuracy is achieved by assuming the conditions of Eqs. 12.4-5. Therefore the rotation 8z ' roll, is first approximated by Eq. 12.5; then the rotation 8x ' pitch, is approximated by Eq. 12.1; and then the final rotation 8y , yaw or tum, is approximated by Eq. 12.2. The next approximation of8z utilizes the previous value of8z in Eq. 12.3 and the similar steps are preferably repeated until the desired accuracy is achieved. The transcendental
45
[:L =[~ ~ttl +[:l (14.1)
[1°[: :' O~[[l[:[ (14.2)
50
The Kalman filter is now described for a single dimension, 55 although it is utilized for prediction and smoothing for all
position dimensions. The predictor stages consist of the calculation of the state and the error covariance projection equations. The state projector equation, Eq 15.1, utilizes a discrete time-sampled difference equation of r calculated from Eq.
60 15.2. In other words, the numerically derived velocity and acceleration components of motion are linearly combined with the previously a priori position to estimate the new position. The corrector stages consist of sequential computation of the gain, updated state estimate, and updated error
65 covariance equations. The a posteriori state estimate, Eq. 15.4, is based on a linear combination of the weighted measurement residual and the last state estimate.
JX-003.0029
A000064
Case: 12-1252 Document: 46 Page: 155 Filed: 08/29/2012
US 7,492,268 B2 31
(15.1)
(15.2)
Fk Kk =---(Pk + Rk )
(15.3)
10
(15.4)
(15.5)
The new error covariance projector, Eq. 15.2, is it's previ- 15
ously computed value combined with the current process noise covariance, Qk' which is tuned by an example model derived from the measured motion dynamics shown in Eq. 16.1. The gain's measurement noise covariance, Rk , is defined as a small constant and based upon the actual static 20
timing variance empirically measured. The smaller this value the more confidence there exists in the systems' measurement capability.
32
1 2 0 -5 3 J.l Ck(J.l) = Gk 2 0 1 4 -3 J.l2
o -1 2 -1 11
Ck(J.l)=Gk
o 0 -1 1 J.l3
-O.SJ.l + J.l2 - O.SJ.l3
1 - 2.SJ.l2 + l.SJ.l3
O.SJ.l + 2J.l2 - I.SJ.l3
-O.SJ.l2 + O.SJ.l3
(17.1)
(17.2)
(17.3)
The fl value is normalized and represents the % value between the 2nd
3rd control points. To calculate the interpolated value between the 1 st and 2nd orthen-l th and nth control points, the value of first control point of the pair and the value of the last control point pair are doubly entered into the geometry matrix, respectively. The appropriate dflldt is determined by the desired rate of playback of movement trajectory.
25 To playback at the same rate as the recorded session, and assuming fairly constant velocity, a timestamp should also be saved at each control point registration so that the fl calculation is correctly scaled by the delta time interval. The n-length set of control points would be manually registered by the user
In the preferred embodiment, the product of the numerically-derived 1st and 2nd order derivatives of the measured position scaled by a frequency dependent gain provides a computationally practical adaptive dynamic process noise estimate model. The derivative product term increases Qk proportionally for higher velocity and acceleration components of motion, e.g., quick, abrupt directional changes, which effectively increases the gain and, therefore, means more confidence exists in the measurement rather than the estimate. This provides faithful, low-latency response to high-frequency motions. Conversely, the frequency scaling 35
term decreases the predictive "overshoot" characteristic of lower power, repetitive motion, e.g. slower, cyclic, ROM trajectories, which effectively decreases the gain and, therefore, means more confidence exists in the estimate rather than 40
the measurement. It should be appreciated this filter implementation provides superior tracking fidelity and comparable smoothing characteristics as compared to practical lengths of finite impulse response running-average filters and various low-orders infinite impulse response filters. It achieves 45
enough predictive response to compensate for the inherent TOF and computation latencies, while providing and comparable smoothing properties of other filter types.
30 pressing a switch or automatically post processed by a sorting method where a control point is registered at the tangents of the trajectory having sufficient magnitude and/or experience sign changes which indicates discontinuous or non-mono-tonic movement.
The major functional interfaces of the transponder unit preferably include the sensory interface, transducer interface, processor, and communication interface. The following descriptions of the transponder unit are based upon the dependence flow represented by FIG. 6.
The sensor interface refers to the collective support for the ultrasonic transmitter, heart rate receiver, and accelerometer circuits. The ultrasonic transmitter circuit is preferably gated by a pulse-width modulated (PWM) digital signal at nominally 0.8% duty cycle of the 40 kHz resonant frequency, e.g., a single 250 flS pulse every analysis period, by the processor
(16.1) 50
circuit. The radiated ultrasonic signal strength is controlled by gating a MOSFET transistor switch at a duty cycle which optimally energizes the transducer's series resonant tank circuit for sufficient duration. The resonant circuit's reactive components include an impedance matching inductor, the
(16.2)
In the preferred embodiment, a three dimensional (3D) piecewise cubic curve interpolates a movement trajectory for smoothing and reduced sample storage for greater memory efficiency. Preferably, four (4) sequential discrete control points of the n-length set of control points, the sample resolution dependent upon the desired movement granularity, and corresponding timestamp are needed to calculate in real-time the interpolated position between any pair of control points. A Catmull-Rom spline algorithm is the preferred method in that the path intersects the control points and would best approximate a movement that may have acute directional changes. The Catmull-Rom spline algorithm is defined by Eqs. 17.1-3, where the geometry matrix G k represents the matrix of three dimensional (3D) control points.
transducer's intrinsic capacitance, and a small damping resistive load. At resonance, a electrical damped sinusoidal with a potential up to -400 V pk_pk is developed across the transducer
55 to sufficiently drive it at acoustical power levels practical for the system's intended range of operation. Enabling a lower duty cycle control through means of a software algorithm monitoring the transponders range would effectively lower the transponders power consumption and radiate less ultra-
60 sonic acoustic energy for close range operation when signal saturation and clipping is undesirable. Conversely, a higher duty cycle control would radiate greater ultrasonic energy to compensate for less efficient, non-optimal acoustical coupling orientations of the transponder with respect to the
65 receiver constellation. Optionally, two additional transducers may be driven in unison or sequentially from a different transponder assembly to support measurement of absolute
JX-003.0030
A000065
Case: 12-1252 Document: 46 Page: 156 Filed: 08/29/2012
US 7,492,268 B2 33
rotation about a single or multiple axes, or provide calculated positional redundancy for certain difficult line-of-sight applications.
34 clock also provides all the capture and control timing requirements for the other specified circuits. Multiple parallel I/O ports and dedicated asynchronous serial communication signals provide digital control for the circuits of the parallel/ serial I/O circuit.
In the preferred embodiment, the graphic LCD and touch screen circuit is the primary user input device for database management for an interactive transponder configuration. For example, it may be a 128x64 graphical liquid crystal display
The heart rate receiver circuit wirelessly receives a 5 kHz heart rate signal from a POLAR® transmitter belt. The transmitter, wom around the chest, electrically detects the heart beat and starts transmitting a pulse corresponding to each heart beat. The receiver captures the signal and generates a corresponding digital pulse which is received by the timing capture-control circuit of the processor interface. A software algorithm processes the signal with known time-based averaging and an adaptive window filter techniques to remove any extraneous artifact or corruption caused by interfering sources.
10 system (LCD) and associated 4-pin touch screen input device. A preferred LCD device is the 51553 manufactured by Optrex and the preferred touch screen device is the TSG-51 manufactured by Apollo Displays. LCD display information, configuration commands, and bitmaps images can be loaded
The accelerometer circuit consists of a low cost +/-1.5 g dual axis accelerometer that can measure both dynamic, e.g. vibration, and static, e.g. gravity or tilt, acceleration. If the accelerometer is oriented so both its axes are parallel to the earth's surface it can be used as a two axis tilt sensor with a roll and pitch axis.
15 through the software calculation engine via a parallel memory interface to emulate a graphical user interface. A touch screen input device is connected to a controller circuit to decode soft key presses at areas over the graphical object. Preferably, the key presses are registered, filtered, decoded,
20 and processed by the controller and then transferred to the software calculation engine via an interrupt driven asynchronous serial communication channel of the I/O interface. A preferred LCD controller is the UR7HCTS manufactured by Semtech.
The stimuli interface circuit provides the primary visual sensory interface preferably comprised of a linear array of five (5) bright, white light emitting diodes (LED) and associated drivers. The preferred LED device is a CMD87 manufactured by Chicago Miniature Lamp. These LEDs' intensity 25
is controlled by a white LED driver. The preferred white LED driver device is a MAX1570 manufactured by Maxim. The white LED driver provides a maximum 120 rnA constant current source to each LED for optimal uniform luminescence. The drive current can be proportionally regulated 30
through external pulse width modulation (PWM) means from the processor circuit to modulate its brightness level. Additionally, an electronic switch is connected in series to each LED drive to individually control its active state. By simultaneously controlling the PWM duty cycle and active state of 35
each LED, the light strobe can appear to smoothly migrate along the linear array in spite of its discontinuous operation.
Preferably, the stimuli interface circuit provides the primary aural stimulus by means of a 4 kHz piezo buzzer. The preferred device is SMT-3303-G manufactured by Projects 40
Unlimited. This electro-mechanical buzzer requires an external transistor drive circuit and digital control signal gated at a rate near its resonant frequency. The buzzer inputs are connected to and controlled by PWM means from the processor circuit to provide a gross volume adjustment which is depen- 45
dent upon the amplitude of the drive signal.
The timing capture-control circuit provides controlling means for the stimuli interface and portions of the sensor interface. The stimuli interface is preferably comprised of a five channel 16-bit timer PWM module with programmable interrupt control which provides 250 llS timing resolution to automatically modulate the circuits' drivers through variable duty cycle control.
In the preferred embodiment, the AID conversion circuit receives the output from the accelerometer circuit and consists of a two channell O-bit analog-to-digital converter used determine the rotational angle of roll and pitch in the accelerometer deviates from its horizontal plane orientation. This information is communicated to the signal processor via the radio link.
In the preferred embodiment, the radio link circuit is comprised of a wireless bi-directional communication interface (with a receiver and transmitter shown generally at 20 and 30) to (1) receive a synchronization signal for control of the transponders interoperability, (2) to transfer acquired local sensor data, including, but not limited to, accelerometer, heart rate, battery, user I/O status, to processor unit and (3) to provide means to configure its local database from command of processor unit. The preferred wireless communication link is based upon the AT86RF211, a highly integrated, lowpower FSK transceiver optimized for license-free ISM band
Additionally, the stimuli interface circuit provides the primary tactile stimulus by means of a vibrator motor. The driver for the vibrator motor enables a 120 rnA DC current source to excite the motor armature. The preferred driver device is the MAX1748 manufactured by Maxim. The rotational speed of the motor's armature is controlled by PWM means from the processor circuit.
50 operations from 400 MHz to 950 MHz. and manufactured by Atmel. Its key features are described above.
In the preferred embodiment, the switch I/O circuit uses a SPST push button switch for user input to control the system's operational states, start and stop program execution,
55 and function as feedback input to the program. A preferred device is the KSS231 SPST pushbutton switch manufactured by ITT Industries.
The processor circuit preferably receives input from the stimuli interface, sensor interface, and the communication interface and provides controlling signals therein. The preferred processor circuit is the MC9S08GB60 which is manufactured by Motorola Inc. It is a low-cost, high-performance 8-bit microcontroller device that integrates the specialized hardware circuits into one convenient device. The software 60
calculation engine circuit operates from an embedded 60 KB FLASH for program memory with in-circuit programmable capability and 4 KB RAM for data memory. The time base circuit is preferably comprised of an external, high-noise immunity, 4.0 MHz system clock, which multiplies this value 65
by the internal frequency-locked loop for a bus clock of 40.0 MHz and single instruction execution time of 25 llS. This
What is claimed is: 1. A system for tracking position of a user, comprising: a first communication device, adapted for being attached
to, or held by, the user, comprising: a transmitter for transmitting signals; a receiver for receiving signals; and an output device; and
a processing system, remote from the first communication device, for wirelessly receiving the signals transmitted by the transmitter, determining position information for
JX-003.0031
A000066
Case: 12-1252 Document: 46 Page: 157 Filed: 08/29/2012
US 7,492,268 B2 35
the first communication device and sending data signals to the receiver to provide feedback or control data to the user;
wherein the first communication device receives and processes the received data signals and generates sensory stimuli for the user, based on the received data signals and delivered through the output device.
2. The system of claim 1, further comprising:
a second communication device, in electrical communica- 10
tion with the first communication device and in wireless communication with the processing system, and adapted for being attached to, or held by, the user;
wherein the processing system is remote from the second communication device and is adapted for detennining 15
the position of the second communication device relative to at least one of the first communication device and the processing system.
3. The system of claim 1, wherein:
the first communication device comprises a user input device adapted for communication with the processing system through the transmitter.
4. The system of claim 3, wherein:
20
the user input device is adapted for calibrating the first 25
communication device to establish a reference position.
5. The system of claim 1, wherein: the processing system is adapted to determine acceleration
information of the first communication device.
6. The system of claim 1, wherein:
the sensory stimuli are at least one of: aural, visual or tactile.
7. The system of claim 1, wherein:
30
35 the processing system determines posItion infonnation
without interference from occluding objects.
8. A method for tracking position of a user, comprising the steps of:
providing a system according to claim 1;
establishing a wireless communication between the first communication device and the processing system;
40
exchanging data signals between the first communication device and the processing system, the first communication device sending position information data signals to 45
the processing system and the processing system sending feedback or control data signals to the first communication device; and
providing sensory stimuli to the user through the output 50
device, the sensory stimuli based upon the received feedback or control data signals.
36 9. The method of claim 8, wherein: the system further comprises a user input device adapted
for communication with the processing system through the transmitter; and
the step of exchanging data signals further comprises sending user input data signals to the processing system.
10. An apparatus for use in conjunction with a remote processing system for tracking position of a user, comprising:
a transmitter for transmitting position information signals to the remote processing system;
a receiver for receiving feedback or control data signals wirelessly from the remote processing system, the data signals derived from processed signals from the transmitter; and
an output unit for delivering sensory stimuli to the user, based upon the data signals.
11. The apparatus of claim 10, wherein: the data signals are derived by comparing the position
information transmitted to reference position infonnation.
12. The apparatus of claim 10, further comprising: a first and a second communication device, each adapted
for being attached to, or held by, the user; and each of the communication devices is in communicative
contact with the processing system directly or through the other communication device.
13. The apparatus of claim 12, further comprising: a user input device deployed on at least one of the commu
nication devices and adapted for communication with the processing system through the transmitter.
14. The apparatus of claim 10, further comprising: an interactive interface such that movement of the appara
tus controls the movement of an object in a computer generated virtual environment.
15. A system for tracking movement of a user, comprising: a first communication device, adapted for being attached
to, or held by, the user, comprising: a transmitter for transmitting signals; a receiver for receiving signals; and an output device; and
a processing system, remote from the first communication device, for wirelessly receiving the signals transmitted by the transmitter, detennining position information for the first communication device and sending data signals to the receiver to provide feedback or control data to the user;
wherein the first communication device receives and processes the received data signals and generates sensory stimuli for the user, based on the received data signals and delivered through the output device.
* * * * *
JX-003.0032
A000067
Case: 12-1252 Document: 46 Page: 158 Filed: 08/29/2012