Using Data Compression to Increase the Bandwith of Existing Tactical Control System

Malcolm VantDeputy Director General at DREO

Chairman of NATO IST

Lots of bits in imagery

Not enough pipe to push imagery to the field

Visualisation quality is tied to compression

Problems

Information Management

Click to add sub-title

Margaret Varga

Sensors and Electronic Sectors

Telephone: +1684 895712Facsimile: +1684 894952Email: varga@signal.dera.gov.uk

Defence Evaluation & Research AgencyMalvernWorcestershireWR14 3PSUK

Background

Huge volumes of diverse data to be analysed

Existing systems are outpaced

Manual examination is still the main approach

Need for fast, cost effective and reliable semi-automatic or automatic approaches

To retrieve, screen, evaluate, correlate, disseminate, archive and present information at the right time, place and format

Information Deluge

An Overview of an Information Management Process

Raw Data

Target DataPre-processedData

TransformedData

Patterns

KnowledgeData Selection

Pre-processing

Transformation

Data Mining

Interpretation /Evaluation

Analysis

Intelligence

Visualisation

Data Collection

InformationData

Information Management Process(MoD)

Data Collection - Reports (digital/hardcopy), formatted, application oriented

Data Selection - Information Retrieval/Extraction/Routing

Pre-Processing - Elimination/reduction of outliers

Transformation - Conversion into appropriate format for processing

Pattern Finding - Data mining

Interpretation/Evaluation of Patterns - Knowledge discovery

Analysis of Knowledge - Intelligence

Summarisation

Information visualisation

Information Management Principles

Relevance

Effectiveness and Efficiency

Responsiveness

Accessibility

Custody

Preservation

Accountability

Open Source Textual Information Exploitation(Military Relevance)

Timely access to relevant, accurate, precise, comprehensible and credible information (e.g. for decision making)

Ability to track what information is available (covert and overt)

Ability to define rapidly changing topics of interest from diverse users

Zoom or expand over levels of specialised knowledge (i.e. summary or detailed analysis)

Limitations of Existing Information Retrieval Approaches

Commercial Off The Shelf (COTS) products are only effective in a few domain specific applications

COTS based on user-specified ad hoc keywords, phrases or topics:– subject scope can become too general and unfamiliar– costly (time & money), depends on predefined search definition– irrelevant reports are retrieved and relevant reports may remain hidden

Full reports are presented

Duplicate information from same and different sources is presented

Retrieved reports are ranked by existence and frequency of keywords- not necessary equivalent to relevancy

Open Source Textual Information Exploitation Research Programmes

VisualisationPlumber, TextScape,Delos

Delos - Textual (and speech) information sourcesDelos - Textual (and speech) information sources

DERA-OKAPI Demonstrator

RedundancyElimination

Accurate KeyGeneration &

Topic spotting

RelevanceRanking

Summarisation

Indexation, Profiling,Archival & intelligence Analysis

DERA-OKAPI Information Retrieval Demonstrator

DERA-OKAPI Information Retrieval Demonstrator

DERA-OKAPI Information Retrieval Demonstrator

User Interest Oriented Text Summarisation

Accurate ‘keys’ are used as salient summary indicators of

the news feeds

Extracted sentences with the occurrence of the accurate keys

Extract the leading paragraph of the news item as a ‘rough’

summary of the reporting event

Extract the sentence before and after the ‘key’ sentence -

reduce dangling problems

Paragraphs with more than 3 ‘keys’ or more than 70%

relevant sentences

Text Summarization

Keywords/phrases

First paragraph

Sentences with ‘keys’

Paragraphs with more than 70% of sentences deemed relevant or > 3 keys

News Items

1 - 3%

5 - 10 %

15 - 20%

25 - 30 %

30 - 35%

100%

Amount of textAmount of text

Sentences before and after the sentence with ‘keys’

Salient Keys

1. I-FOR2. CONVINCED3. DAYTON4. CIVILIAN5. CRIMES6. MISSION7. SERIOUS8. INVESTIGATIONS9. SITUATION10. LIFE11. SUCCESS12. Nato13. WAR14. POLICE15. STATEMENTS16. PROCESS17. PROBLEMS18. keeping19. peace20 FORCE

A.1.1 Original News

Record 0 weight 1000 Length: 2/~1 pages

<Subject:> BOSNIA/SMITH <L-ONLY> BY NICK SIMEONE /WASHINGTON<ORGANISATION:> Defence Research AgencyDATE=/4/2/96TYPE=CORRESPONDENT REPORTNUMBER=2-195223TITLE=BOSNIA/SMITH (L-ONLY)BYLINE=NICK SIMEONEDATELINE=WASHINGTONCONTENT=VOICED AT:

INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRAL LEIGHTON SMITH, SAYS IT IS TOOEARLY FOR MILITARY COMMANDERS TO RECOMMEND WHETHER U-S PEACE-KEEPING TROOPSSHOULD REMAIN IN BOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER. THAT’S ADECISION, HE SAYS , THAT WILL HAVE TO BE MADE LATER THIS YEAR BUT NEARLY FIVE MONTHSINTO THE PEACE-KEEPING OPERATION, ADMIRAL SMITH IS RAISING CONCERNS ABOUT CIVILIANASPECTS OF THE BOSNIA PEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICE TOACTUALLY ENFORCE LAW. WE HAVE THIS REPORT FROM V-O-A’S NICK SIMEONE.

TEXT: WITH WAR CRIMES INVESTIGATORS STARTING TO LOOK INTO MASS GRAVES, ADMIRALSMITH SAYS ENORMOUSLY DIFFICULT TASKS STILL LIE AHEAD AS THE WORLD WATCHES TO SEEWHETHER BOSNIA’S FORMER WARRING PARTIES WILL BE ABLE TO FULFILL TERMS OF THE PEACEACCORD THEY SIGNED TO END THE WAR. // OPT // HE MADE THE STATEMENTS TUESDAY AT APENTAGON BRIEFING. //

LATELY, SOME POLITICAL LEADERS IN EUROPE HAVE BEEN PRIVATELY SUGGESTING THE PEACEPROCESS IN THE FORMER YUGLOSLAVIA IS TOO DELICATE AND THE COUNTRY IS TOO LIKELY TOSLIP BACK INTO WAR FOR NATO FORCES - PARTICULARLY U-S TROOPS ----- TO CONSIDER PULLINGOUT AT THE END OF THE YEAR. PRESIDENT CLINTON HAS REPEATEDLY SAID THAT U-SPARTICIPATION IN THE PEAC-KEEPING MISSION IN BOSNIA WIIL ONLY BE FOR ONE YEAR. NATOCOMMANDER SMITH SAYS THE MILITARY, AT THIS STAGE AT LEAST, IS NOT MAKING PREDICTIONSEITHER WAY.

// SMITH ACT //

THERE ARE SOME PEOPLE IN EUROPE WHO HAVE EXPRESSED ANOPINION BOTH FOR AND AGAINST I-FOR REMAINING AFTERDECEMBER. I WILL JUST TELL YOU THAT THE SMART MAN WOULDGOO LOOK AT THE SITUATION, JUDGE IT AT THE TIME, MAKE ADECISION AND THEN FROM THAT POINT ON BE GUIDEDACCORDINGLY.

A.1.1 Leading Paragraph Summarisation

INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRALLEIGHTON SMITH, SAYS IT IS TOO EARLY FOR MILITARY COMMANDERS TORECOMMEND WHETHER U-S PEACE-KEEPING TROOPS SHOULD REMAIN INBOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER. THAT’S ADECISION, HE SAYS , THAT WILL HAVE TO BE MADE LATER THIS YEAR BUTNEARLY FIVE MONTHS INTO THE PEACE-KEEPING OPERATION, ADMIRALSMITH IS RAISING CONCERNS ABOUT CIVILIAN ASPECTS OF THE BOSNIAPEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICE TOACTUALLY ENFORCE LAW. WE HAVE THIS REPORT FROM V-O-A’S NICKSIMEONE.

INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRALLEIGHTON SMITH, SAYS IT IS TOO EARLY FOR MILITARY COMMANDERS TORECOMMEND WHETHER U-S PEACE-KEEPING TROOPS SHOULD REMAIN INBOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER. THAT’S ADECISION, HE SAYS , THAT WILL HAVE TO BE MADE LATER THIS YEAR BUTNEARLY FIVE MONTHS INTO THE PEACE-KEEPING OPERATION, ADMIRALSMITH IS RAISING CONCERNS ABOUT CIVILIAN ASPECTS OF THE BOSNIAPEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICE TOACTUALLY ENFORCE LAW. WE HAVE THIS REPORT FROM V-O-A’S NICKSIMEONE

LATELY, SOME POLITICAL LEADERS IN EUROPE HAVE BEEN PRIVATELYSUGGESTING THE PEACE PROCESS IN THE FORMER YUGLOSLAVIA IS TOODELICATE AND THE COUNTRY IS TOO LIKELY TO SLIP BACK INTO WAR FORNATO FORCES - PARTICULARLY U-S TROOPS ----- TO CONSIDER PULLINGOUT AT THE END OF THE YEAR. PRESIDENT CLINTON HAS REPEATEDLYSAID THAT U-S PARTICIPATION IN THE PEAC-KEEPING MISSION IN BOSNIAWIIL ONLY BE FOR ONE YEAR. NATO COMMANDER SMITH SAYS THEMILITARY, AT THIS STAGE AT LEAST, IS NOT MAKING PREDICTIONS EITHERWAY.

BUT IN EITHER CASE, ADMIRAL SMITH SAID THAT THOUGHT IS BEINGGIVEN TO SCALING BACK THE NATO FORCE, WHICH NOW STANDS ATABOUT 60-THOUSAND TROOPS, AS THE OPERATION HEADS INTO THESUMMER MONTHS. SO FAR NATO COMMANDERS SEE NO EVIDENCE THATTHE WARRING PARTIES ARE PREPARING FOR WHAT IN YEARS PAST HASBEEN THE START OF A WARM WEATHER, SPRING OFFENSIVE.

HE SAYS A KEY TO THE SUCCESS OF THE BOSNIA PEACE PLAN WILL BETHE BOSNIAN ELECTIONS, WHICH, ACCORDING TO THE PEACE PLAN, ARETO BE HELD IN SEPTEMBER, BY THEN NATO COMMANDERS, AS WELL ASTHE SPONSERS OF THE DAYTON PEACE ACCORD, WILL LIKELY HAVE AGOOD IDEA WHETHER BOSNIA’S POST-WAR ATMOSPHERE IS FREE OFTHREATS AND IF, IN ADMIRAL SMITH’S WORDS, POLITICIANS CAN GIVESPEECHES WITHOUT BEING SHOT. SUCH INDICATIONS COULD INFLUENCETHE DECISION ON WHETHER COME DECEMBER, U-S TROOPS STAY ORLEAVE.(SIGNED)

Salient key-based summary

A.1.1 Simple Summarisation Based on User’s Ad-hoc Keywords(Nato/peace/keeping)

INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRALLEIGHTON SMITH, SAYS IT IS TOO EARLY FOR MILITARY COMMANDERS TORECOMMEND WHETHER U-S PEACE-KEEPING TROOPS SHOULD REMAIN INBOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER.

BUT NEARLY FIVE MONTHS INTO THE PEACE-KEEPING OPERATION,ADMIRAL SMITH IS RAISING CONCERNS ABOUT CIVILIAN ASPECTS OF THEBOSNIA PEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICETOACTUALLY ENFORCE LAW.

TEXT: WITH WAR CRIMES INVESTIGATORS STARTING TO LOOK INTO MASSGRAVES, ADMIRAL SMITH SAYS ENORMOUSLY DIFFICULT TASKS STILLLIE AHEAD AS THE WORLD WATCHES TO SEE WHETHER BOSNIA’S FORMERWARRING PARTIES WILL BE ABLE TO FULFILL TERMS OF THE PEACEACCORD THEY SIGNED TO END THE WAR.

LATELY, SOME POLITICAL LEADERS IN EUROPE HAVE BEEN PRIVATELYSUGGESTING THE PEACE PROCESS IN THE FORMER YUGLOSLAVIA IS TOODELICATE AND THE COUNTRY IS TOOL LIKELY TO SLIP BACK INTO WARFOR NATO FORCES - PARTICULARLY U-S TROOPS - TO CONSIDER PULLINGOUT AT THE END OF THE YEAR. PRESIDENT CLINTON HAS REPEATEDLYSAID THAT U-S PARTICIPATION IN THE PEACE-KEEPING MISSION IN BOSNIAWILL ONLY BE FOR ONE YEAR.

NATO COMMANDER SMITH SAYS THE MILITARY, AT THIS STAGE AT LEAST,IS NOT MAKING PREDICTIONS EITHER WAY.

BUT IN EITHER CASE, ADMIRAL SMITH SAID THAT THOUGHT IS BEINGGIVEN TO SCALING BACK THE NATO FORCE, WHICH NOW STANDS ATABOUT 60-THOUSAND TROOPS, AS THE OPERATION HEADS INTO THESUMMER MONTHS.

SO FAR NATO COMMANDERS SEE NO EVIDENCE THAT THE WARRINGPARTIES ARE PREPARING FOR WHAT IN YEARS PAST HAS BEEN THE STARTOF A WARM WEATHER, SPRING OFFENSIVE.

WHAT WE WANT TO BE ABLE TO DO IS TO KEEP THE MILITARY APART ANDIF WE CAN DO THAT WITHOUT A LARGE FORCE OR WITHOUT A FORCE,THAT REALLY THE GOAL.

Information Visualisation

Represent both the structure & content of the retrieved information

Readily comprehensive manners

Facilitate high level tasks

– Identification of similar/duplicate information– Identification of topic trends– Identification of anomalies

Provides access to different levels of summary

DERA TextScape

DERA TextScape

DERA TextScape

DERA TextScape

DERA TextScape

DERA TextScape

Evaluation Issues

Retrieval effectiveness: Dynamic data sources User’s oriented/task oriented Key word saliency

Summarisation Summaries first Followed by full report

Visualisation Effective in showing the

– Identification of similar/duplicate information– Identification of topic trends– Identification of anomalies

Inter-relationship? Temporal ?

Event Stream Analysis

Analysis of large datasets generated during intensive man/machine interaction

The analysis of the large log file generated by these interaction is assisted by DERA’s Plumber

Plumber - a visual programming tool which enables a new visual environment for data processing

Part of a commercial package of DP+MS

Plumber

Main canvas

Palette ofprocessing

widgets

Libraries ofpre-written

widgets

Built-invisualisers such

as bar charts

Access to externalprocessing tools

such as Excel andMatlab

Store resultsto disk

Java version