Download - 4. Larkin Sensory
-
8/7/2019 4. Larkin Sensory
1/29
SENSORY INFORMATIONSYSTEMS14 March 2011
WILLARD LARKINProgram Manager
AFOSR/RSL
Air Force Office of Scientific Research
AFOSR
Distribution A: Approved for public release; distribution is unlimited. 88ABW-2011-0780
-
8/7/2019 4. Larkin Sensory
2/29
2
2307/C PORTFOLIO OVERVIEW
Program Manager: Willard Larkin
BRIEF DESCRIPTION OF PORTFOLIO:
Auditory modeling for acoustic analysis
Biological polarization optics & vision Sensori-motor control of bio- flight & navigation
SUB-AREAS IN PORTFOLIO:Sensory Information Systems (2307/C)
-
8/7/2019 4. Larkin Sensory
3/29
3
Program Trends and Strategy:TOPIC AREA OVERVIEW
Polarization Vision & Optics:
Sensorimotor Control of Flight & Navigation:
Scientific Question: How do natural photoreceptors detect and how doanimal brains interpret polarization information? How is it used for nocturnalnavigation or recognition of obscured targets? Can these unique bio-opticalstructures be emulated?
Scientific Question: How does neural control make natural, low-ReynoldsNo. flight autonomous, efficient, and robust? Discover principles ofmultisensory fusion, distributed sensors and actuators. Develop control lawsfor emulation in MAVs.
Advanced Auditory Modeling:Scientific Question: How does the auditory brain parse acousticlandscapes, bind sensory inputs, adapt its filters, hear through noise anddistortion? Could autonomous listening devices emulate neurology to match orexceed human auditory analysis, e.g., to detect and identify speech targets innoise and reverberation?
42%
11%
47%
Primary Strategy: Forge useful connections between math and biology
3 +6
-
8/7/2019 4. Larkin Sensory
4/29
4
Other OrganizationsThat Fund Related Work
Funds 5 current AFOSR P.I.s in a new MURI, Animal Inspired Robust
Flight with Outer and Inner Loop Strategies. (M. Steinberg)ONR
ONRProgram, Bio-Inspired Autonomous Systems, (T. McKenna) has focuson aquatic environment, held joint review with AFOSR in May 2010
WRAMCCoordinates psychoacoustics with AFRL/RH, and extends our
6.1 research to hearing-impaired populations. (D. Brungart)
ARL MASTMicro Autonomous Systems and Technology program forurban and complex terrain funds 6.2 work on MAVs.
NSF Partners with AFOSR to support the annual workshop onNeuromorphic Modeling, involving several AFOSR P.I.s
NSF Broad programs in biomathematics (M. Horn), perception, action &cognition (B. Tuller), NSF-NIH computational neuroscience
5
NRL Applies AFOSR 6.1 to MAV development; funds related 6.2, 6.3
. . . plus international & 6.2 coordination
-
8/7/2019 4. Larkin Sensory
5/29
5
International and 6.2Coordination
U.K.
T. Daniel
S. Humbert
M. Willis
U.S.AFRL-Dstl Working Group
Biologically-Motivated Micro-Air-Vehicles
STATE OF THE ART REVIEW
Georgia Tech15-18 June, 2010
Organizers: M. Wehling, AFRL. P. Biggins, Dstl
Presentations:https://livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&objId=24091294
&objAction=browse&viewType=1
30 Participants from UK, US, Industry, Academia, & Gov.
H. Krapp
J. Niven
G. Taylor
https://livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&objId=24091294&objAction=browse&viewType=1https://livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&objId=24091294&objAction=browse&viewType=1https://livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&objId=24091294&objAction=browse&viewType=1https://livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&objId=24091294&objAction=browse&viewType=1http://en.wikipedia.org/wiki/File:Georgia-Tech-Insignia.svg -
8/7/2019 4. Larkin Sensory
6/29
6
Recent Highlights
Best Paper, 2010Journal of
ExperimentalBiology
Chiu, et al., Paper oncompetitive target capture
by echolocating bats
AFRL Fellow2010
Richard McKinleyHuman Effectivenes Directorate
Acoustic Research
Special IssueHearing Research
Mechanics ofHearing Workshop
-
8/7/2019 4. Larkin Sensory
7/29
7
2 Photoreceptor
Rows havemembrane for wave phase delay
Crossed parallelmicrovilli in lowersection conferlinear pol
sensitivity
Photoreceptorrows 1-6 in O.Scyllaruscompound eye
Visual Photoreceptor
Discriminates L vs RCircular Polarization
Receptor Membrane
Imparts a Uniform 1/4Wave Delay, 400 700 nm
Comparison with Best Available Materials
QUARTZOPTIMAL GRATING
Two Recent Discoveries
T. Cronin, (UMBC), J. Marshal (Queensland), N. Roberts (Manchester)
http://en.wikipedia.org/wiki/File:Circular.Polarization.Circularly.Polarized.Light_Right.Handed.Animation.305x190.255Colors.gif -
8/7/2019 4. Larkin Sensory
8/29
8
AFOSR Young Investigator DevelopsTransformative Science
Math predicts a previously unknown &unexpected mode of neural behavior:prolonged electrical silencing, thought tobe physiologically unsustainable.
Math captures details of intracellular
gene expression / transcription andmembrane electrochemistry.
Confirmed during 2010 in several labs
Finding revises basic assumptions forneural computation in cells & networks.
MEMBRANEDEPOLARIZATION
Science 326, 9 Oct. 2009D. Forger, AFOSR YIP.
NORMALACTIONPOTENTIALS
ELECTRICALLYSILENT
~ 5 Hrs.
http://www.umich.edu/http://www.umich.edu/ -
8/7/2019 4. Larkin Sensory
9/29
9
Recent Transitions
To Army: VLSI implementation of insect visual sensory fusion for a motion-
sensitive guidance system. Contract W56HZV-100192. (Dr. P. Shoemaker)
To Oticon A/S: Method to boost directional hearing via monaural tandemanalysis of pitch and voice segmentation. Dr.Ulrik Kjems, J. Woodruff. (Dr. Wang)
To NRL: Methods for cooperative steering of autonomous surface & air vehicles,from control law mathematics based on bats & dragonflies. Dr. Justh, NRL/TEMD
To AFRL 6.2: Spatial audio sorting, annotation, and retrieval system enhancesvoice communications for multiple sources. Victor Finomore, RHCB. (Dr. N. Iyer)
To NSMRL: Simulators and technical method to calibrate underwater acousticstimulation of the human head. Dr. Michael Qin. (Creare, Inc. STTR, Dr. A. Dietz.)
To AFRL 6.2: Speaker recognition method, based upon binary mask technique,boosts performance in additive noise. Brett Smolenski, RADC/Rome Lab. (Dr. Wang)
To EmergentViews.com: New technique for polarization imaging, basedupon 6.1 AFOSR work accomplished under the BioInspired Theme. (Dr. N. Engheta)
To NSA & CIA: Method to sort speech from non-speech in noisy electronicsignals, based upon cortical model from experiments with ferrets. (Dr. S. Shamma)
7
-
8/7/2019 4. Larkin Sensory
10/29
10
150 dBA SPL
Most AF Flight-Line Crews
suffer Hearing Loss
A 50 dB Technical Transition
ResearchGoal:
AFOSR LaunchedMultiple Efforts:
Discover how high-level soundtransmits through air, bone, andtissue to the human cochlea.
Enable 50 dB acoustic isolation,with no sacrifice of voicecommunications.(Legacy level was 30 dB)
6.1 Research in Support of DTO HS-33
Measure nonlinear loudness compression in bone conduction Model acoustic wave propagation through skull Develop a physiologically realistic, instrumented human head simulator Measure and model dynamics of middle ear transduction
Develop new techniques for noise cancellation, active and passive
6.16.1STTRSTTR
STTR
-
8/7/2019 4. Larkin Sensory
11/29
11
Technical TransitionAchieved 50 dB Attenuation Goal of DTO HS-33
Program Coordination with: NATO, ONR, ARO, AFRL 711th HPW
Wright-Patt Bioacoustics Lab tests New
Helmet for Navy Carrier Crew
Instrumented Head Simulatorenabled acoustic tests withoutrisk to human listenersHanover, N.H.
Coordinated 6.1, 6.2efforts enabled a majorbreakthrough in hearing
protection
http://en.wikipedia.org/wiki/File:CardSeal-1.gifhttp://en.wikipedia.org/wiki/File:CardSeal-1.gifhttp://en.wikipedia.org/wiki/File:UIUC_seal.svghttp://en.wikipedia.org/wiki/File:Dartmouth_College_shield.svg -
8/7/2019 4. Larkin Sensory
12/29
12
Auditory Modeling for Acoustic Analysisand Audio Displays
0 0.2 0.4 0.6 0.8 1 1.2-0.1
-0.05
0
0.05
0.1
Time (s)
Amplitude
Speech signals encode information in low-frequencyenvelopes modulating high-frequency carriers
Cortical theory for speech detection and recognition
Innovative signal processing ideas
based upon neural mechanisms
ResearchTopics
Modulation analysis of acoustic signals
3D spatial audio displays to optimize human performance
Biophysical basis of3D spatial hearing.
(Wm. Hartmann, MSU)
Computational auditory scene analysis
3 +6
REPORTEDIN RECENT
AFOSRREVIEWS
NEW IN 2011:
NEXT SLIDES:
-
8/7/2019 4. Larkin Sensory
13/29
13
INNERROW
OUTERROWS
COCHLEAR HAIR CELLS
Cochlea behaves as a system ofadaptive, tunable oscillators.
Triplet Tuning
Adjacent groups of neural fibersrespond strongly to the same key
frequencies in a vowel sound
R. Kumaresan, V. Peddinti (U. Rhode Island) & P. Cariani (Harvard), ICASSP 2011
Synchrony Capture in Auditory Nerve:inspires method for multi-frequency tracking
-
8/7/2019 4. Larkin Sensory
14/29
14
Auditory Model emulates SynchronyCapture to achieve multi-pitch tracking
R. Kumaresan, V. Peddinti (U. Rhode Island) & P. Cariani (Harvard), ICASSP 2011
New architecture for
auditory signal processingemploys synchrony capture
via adaptive band-passfilters that emulate cochlear
mechanics.
NEXT GOAL: Use model to explain other phenomena, e.g., distortion
products, two-tone suppression, gain control.
Spectrogram of a speech fragment overlaid
with multiple frequency tracks obtained viasynchrony capture.
http://en.wikipedia.org/wiki/File:Uriseal.pnghttp://en.wikipedia.org/wiki/File:Uriseal.png -
8/7/2019 4. Larkin Sensory
15/29
15
Neural Model generates Auditory Nonlinear
Resonances to match Neural Brain Signals
Florida Atlantic Univ., E. W. Large, F. Almonte, unpublished (used with permission.) Data from Lee, et al. 2009
INPUT ENERGY
F1 F2Background:
Conventional auditory modelingdoes not account for highlynonlinear neural responsepatterns, e.g. in inferior colliculusor brainstem EEG.
Hypothesis:These nonlinear patterns arekey to auditory cognition.
Progress:Arithmetic combinations of F1 and F2,arise here in brainstem data and in anonlinear coupled oscillator model,fitted with one free parameter (gain).
Neural Responses to a two-tone complex
F2 / F1 = 1.7
STTR:CircularLogic
http://en.wikipedia.org/wiki/File:Florida_Atlantic_University_seal.svg -
8/7/2019 4. Larkin Sensory
16/29
16
1 1( ) ( ) ( ( ))k
K
k
k k
k
K
m tx t x t c t
Complex Modulators Carriers
AFOSR / Rome Lab Symposium onCoherent Modulation Analysis
Acoustic Signal
. . .A key problem in decomposition of speech & other acoustic waveforms
Dr. Les Atlas, University of Washington
Coherent Demodulationis a new
approach for feature extraction andimproved intelligibility. Out-performs conventional incoherent,AM/FM, and phase vocoding methods.
Participants8 July 2010 J.Grieco, D. Harris, B. Pokines, J. Parker, S. Wenndt, K. Godin, A. Noga,J. Cupples, S. Johns, B. King, P. Clark, L. Atlas, W. Larkin
Time
Frequency
Eight harmonics
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.10
500
1000
1500
2000
2500
3000
8-Component Modulation Spectrum for aSpeech Sample
http://en.wikipedia.org/wiki/File:University_of_Washington_Seal.pnghttp://en.wikipedia.org/wiki/File:University_of_Washington_Seal.png -
8/7/2019 4. Larkin Sensory
17/29
17
ResearchTopics
NEXT: BIOSENSORY
Flight and Navigation
Neuromorphic emulation of inner- and outer-loop control
Airfoil mechanosensors in bats
Nocturnal navigation by echolocation or optical polarization
Airfoil, antennae mechanosensors inhawkmoths, insects Sensorimotor reflex basis forcooperative formation control
Bio-clock compensated opticalcompass navigation
Insect flight stabilization & self-motion tuning
Target tracking and pursuit in the dragonfly
REPORTEDIN RECENT
AFOSRREVIEWS
NEW IN 2010 & 2011:
NEXT SLIDES -- RECENT DISCOVERIES:
-
8/7/2019 4. Larkin Sensory
18/29
-
8/7/2019 4. Larkin Sensory
19/29
19
Virtual Reality wind tunnelallows simultaneous tracking offree flight & computer-projected
imagery on the walls and floor
Discovery: Insects Sensorimotor
Altimeter Ignores Ventral Angular Velocity
Scientific Question:How do insects controlflying altitude?
Background:
Flies were assumed to matchflying height to a preferred rate ofoptic flow on the ground.
Caltechs Velocity Clamp lab
enables experimental control ofdynamic optic flow, independent offlight velocity in a wind tunnel.
Discovery: Flies adjust height to match nearby horizontal features. They do not regulate optic flow rate for this purpose. They rely on optomotor and collision-avoidance reflexes.
AFOSR Young Investigator. A. Straw, Caltech. Current Biology 19 Aug. 2010.
Di R ti Fi ld S lf M ti
http://en.wikipedia.org/wiki/File:Caltech_logo.svghttp://en.wikipedia.org/wiki/File:Caltech_logo.svghttp://en.wikipedia.org/wiki/File:Caltech_logo.svg -
8/7/2019 4. Larkin Sensory
20/29
20
Maps of motion sensitivity in right visual field show a strong responseto yaw motion to the right and inhibition to forward translation
Discovery: Receptive Field Self-MotionTuning is Conserved across Species
Self-motion tuning is key to understanding gaze stabilization
Tuning organization is conserved across these 3 species.
Gaze stabilization is essential for flight control keeps visual
sensors aligned with the inertial reference frameH. Krapp, et al., AFOSR Report Nov. 2010. Imperial College, London
http://en.wikipedia.org/wiki/File:Imperial_College_London_crest.svghttp://en.wikipedia.org/wiki/File:Imperial_College_London_crest.svg -
8/7/2019 4. Larkin Sensory
21/29
21
3 Ocelli
(polarizationreceptors)
Lobula PlateVS Neuron
RotationTuning Curve
Primary RotationAxes of Ocellar
Neurons
Flight Control relies on Sensor Fusion inInsect Optical Processing
M. Parsons, et al., Current Biology 2010. Cambridge University.
Yaw
Rotation Fusion Site Discovered inLobula Plate Neurons
Slow Inputs from compoundeye combine with fast inputsfrom polarization receptors
(reflex latencies are 20 to 30ms versus 6 ms for ocelli.)
Both are tuned to specific,but different axes of rotation
Ocellar system has lowerangular precision than thecompound eye system.
Sensory information alignswith axes of natural modes of
insects flight instability
-
8/7/2019 4. Larkin Sensory
22/29
22
Insect Gaze Stabilization viaFeedback/Feedforward Sensor Fusion
HeadDeflection
HeadRoll
HALTERES
COMPOUND EYESNECK MOTOR
SYSTEM
THORAX
ROLL
Theory: Short latency mechanosensors firstdetect body rotation, then feed forwardto induce compensatory head roll via
neck motor system
Long latency visual system detectsresidual optic flow feedback fromincomplete compensation. Data fit preliminary linear model.
Compound eyes plus halteres and ocelli
Compound eyes plus halteresCompound eyes only
H. Krapp, et al., AFOSR Report Nov. 2010. Imperial College, London
-
8/7/2019 4. Larkin Sensory
23/29
23
Target trajectory
Tracking?
Interception?
OR
R. Olberg, Union College, A. Leonardo, Howard Hughes Medical Institute
Dragonfly Attacks Moving Target
21 sec. VIDEO
-
8/7/2019 4. Larkin Sensory
24/29
24
t = -28ms
Evidence for Interception
R. Olberg, Union College, Schenectady, NY
Dragonfly movesgaze prior to launch
Launch angleleads target
Launch angle depends ontarget velocity prior to
takeoff
Dragonfly plans launch direction while still on its perch
Dragonfly Predicts Prey Flight Path
http://en.wikipedia.org/wiki/File:UnionCollegeSeal.svghttp://en.wikipedia.org/wiki/File:UnionCollegeSeal.svg -
8/7/2019 4. Larkin Sensory
25/29
25Dragonfly perch
Dragonfly Predicts Prey Flight Path,Updates Interception Path in Real Time
Neural System Analysis:Relates pursuit time &
accuracy to small-targetselective descending (TSD)
neurons in ventral nerve cord.
~ 300 ms
Dragonfly Eyes
R. Olberg, Union College, Schenectady, NY
Dragonfly visual neurons code for
http://en.wikipedia.org/wiki/File:UnionCollegeSeal.svghttp://en.wikipedia.org/wiki/File:UnionCollegeSeal.svg -
8/7/2019 4. Larkin Sensory
26/29
26
Dragonfly visual neurons code for
both past and future target positions
Neurons tuned to targetlocations in the visual fieldalso convey informationabout past, current, orfuture target positions
Predictive CodingCoding Past Position
Receptive fieldInformation peaks 30 ms
after cell spikes
Receptive fieldInformation peaks 30 ms
before cell spikes
R. Olberg, Union College, Schenectady, NY
Neurons have nearlyorthogonal sensitivity
to direction of motion
Next Research Stage:
http://en.wikipedia.org/wiki/File:UnionCollegeSeal.svghttp://en.wikipedia.org/wiki/File:UnionCollegeSeal.svghttp://www.google.com/imgres?imgurl=http://www.globalcraftsb2b.com/catalog/images/dragonfly.jpg&imgrefurl=http://www.globalcraftsb2b.com/catalog/index.php%3FcPath%3D121_69&h=400&w=400&sz=28&tbnid=jDbdE_V39SNQcM:&tbnh=124&tbnw=124&prev=/images%3Fq%3Ddragonfly%2Bpictures&zoom=1&q=dragonfly+pictures&hl=en&usg=__IAK85ePRP-6nUwcU9VQXDGqHrck=&sa=X&ei=v5ULTaz2HYL6lweS-qGlDA&ved=0CCYQ9QEwAw -
8/7/2019 4. Larkin Sensory
27/29
27
Next Research Stage:Neural Telemetry in Free Flight
Flight Arena
Perch Platform
Cameras
Multi-contactprobe in
mesothoracicganglion
Telemetry chipmounts behind legs,recharges on perch
platform.
R. Olberg, Union College,R. Harrison, Univ. of Utah A. Leonardo, Howard Hughes Medical Institute
SUMMARY:
http://en.wikipedia.org/wiki/File:Hhmi_logo_small.pnghttp://en.wikipedia.org/wiki/File:Hhmi_logo_small.pnghttp://en.wikipedia.org/wiki/File:UofU_official_seal.pnghttp://en.wikipedia.org/wiki/File:UnionCollegeSeal.svg -
8/7/2019 4. Larkin Sensory
28/29
28
SUMMARY:Transformational Impacts & Opportunities
4
Advanced auditory modeling:
Hearing protection:
Optical processing:
Autonomous flight control:
Adaptive airfoils based upon bio-sensory mechanisms Guidance from neural systems, not from networks Discover sensorimotor basis of formation flight
Polarization vision and signaling adapted from biology Achromatic 1/4 wave optical retarders Emulating compound eye in new optical devices
Mathematics for coherent modulation analysis Neural-Inspired analyses to parse acoustic scenes
Massive improvements in high-noise attenuation.
http://images.google.com/imgres?imgurl=http://evolution.berkeley.edu/evosite/evo101/images/bat_echo.gif&imgrefurl=http://evolution.berkeley.edu/evosite/evo101/IIIE5cExaptations.shtml&usg=__xm5JuB-ltO06AUA5RIKytq8Co1g=&h=208&w=233&sz=7&hl=en&start=4&um=1&itbs=1&tbnid=J6qsPgdIc7X-aM:&tbnh=97&tbnw=109&prev=/images%3Fq%3Dbat%2Becholocation%26hl%3Den%26sa%3DX%26um%3D1 -
8/7/2019 4. Larkin Sensory
29/29
29
Questions?
Thank you for your attention
Willard Larkin, Program Manager, AFOSR/NL
703-696-7793
mailto:[email protected]:[email protected]