recorded future: detecting and analyzing changes through space and time

7/29/2019 Recorded Future: Detecting and analyzing changes through space and time

1/10

August 2012

Recorded Future:Detecting and analyzing changes

through space and time

CTOlabs.com

Inside:

Background on Recorded Future

Special Operations Command (SOCOM) use cases

An Open Source Example

An Introduction to Recorded Future for the national security community


2/10

CTOlabs.com

Recorded Future: Detecting and Analyzing ChangesThrough Space and Time

Recorded Future is a company providing new ways of exploring what is known and projected about

coming events. The analytic tools provided by Recorded Future extract meaning and relevance of

information and apply this knowledge to organizational missions. This paper gives insights into how.

The public cloud-based capabilities of Recorded Future are already serving commercial clients with

predictive capabilities. Current use cases span from media analytics to market assessment to nancialforecasts. Government missions are also being served with public cloud oerings through Recorded

Futures ability to analyze and provide information on open source information. Recorded Future has a

proven ability to analyze web-based information to detect changes through space and time.

The Need For Analysis Over Open Source Plus Classifed

Most enterprises need more than analysis over open source. Enterprises have vast stores of internal,

more private information that could also benet from the capabilities of Recorded Future. Most ofthose who have seen the powerful visualizations and analytic capabilities provided by Recorded

Future come away wanting these capabilities inside their enterprise where protected documents can

be analyzed with these predictive tools. In most cases, the best possible solution would be a means

of taking all the predictive insights harvested from the collective resources of the Internet into private

enclaves or protected (even classied) networks where sensitive internal documents could also be

added to the analytic process.

Recorded Future now has an oering that does just that. Their platform, called Foresite, is designed to

run in both private clouds and classied networks. Foresite can extract temporal signals from internal

documents, third-party content and sensitive data sources. What can this mean for operational military

users? Consider a Special Operations Command use-case.

1


3/10

A White Paper or the Government IT Community

SOCOM Mission Needs: Use cases o strategic importance

Organizations like Special Operations Command (SOCOM) frequently articulate needs for capabilities

to detect and analyze changes through space and time. For example, documents such as SOCOMs

2013 Budget Highlights Document and the stated technology needs on the SOCOM.mil website call

for capabilities that can help meet the following mission needs:

Ability to process, display, disseminate and exploit diverse information sources and databases

through the use of multi-level security systems that employ advanced data mining and data

warehousing techniques.

Grouping nodes, identifying local patterns, comparing and contrasting networks, groups,

and individuals. Analysis of network changes through space and time with change detection

techniques.

Analytical technologies showing socio-cultural, economic and demographic factors.

Defendable and repeatable processes, models, and measurement technologies that allow for the

ability to detect changes in behavior or belief over time along with the associated factors that

caused the changes. Including reporting of shifts in reaction to stimulus.

An Open Source, Sel-Generated Proo O Concept

Recorded Future and their Foresite capability show great promise in addressing mission needs like

SOCOMs. For example, consider the fast open analysis their system enabled over ocially released

documents provided the research community by the Combating Terrorism Center (CTC). The released

documents were a collection of 175 pages of Osama Bin Ladens letters discovered at his Abbottabad

compound. 17 total letters were analyzed using Recorded Futures temporal analytic technology and

intelligence analysis tools.

The letters were treated like any other source in the Recorded Future system. Linguistic algorithms

extracted a variety of data points available in the text, and then they were visualized in the Recorded

Future user interface.

2


4/10

CTOlabs.com

3

Some patterns and insights became immediately apparent. For example, a network graph generatedfrom connections found in the letters show clear focal points around topics of God, Yemen and

Afghanistan:

Tools also show, rapidly, locations mentioned the most via visualizations over terrain and in text as

desired:


5/10

A White Paper For The Federal IT Community

Multiple views are possible into the same dataset, and other views can extract meaning of importanceto decision-makers seeking information on human relationships. For example, this view is of those

individuals associated with Iran in the collected letters:

Temporal analysis over timelines is a particular strength of the Recorded Future system. This system is

built to scale to the size of the Internet, but works great over smaller sets like these 175 pages as well.

Looking at all events in time in these letters produces a display like the following:

4


6/10

CTOlabs.com

5

This data can be interacted with in multiple ways. Analysts are empowered with tools that let themdive into and interact with data and display correlations. What If scenarios can also be examined.

The image below shows a deeper look at the years from which quite a bit of data is collected.

As you can imagine in a well architected modern system, the Recorded Future system enables data to

be interlinked and cross referenced easy. Clicking on any point reveals what is known about the point

and can lead to source text that analysts will want before making assessments.

Information can also be extracted for use in other systems. This system was designed to be

interoperable and work with existing technologies and it does that very well. Information can be

exported many ways, including automated machine to machine connections or via export or via directpublication to documents and reports. This is a very easy system to export data from.


7/10


Also of importance to todays analyst is an ability to nd insights into the future. Where documentsmake reference to future events these are plotted using easy to navigate and explore visualizations.

From this particular set of documents, one future reference emerged related to planning the

foundation of a Muslim state.

A Sampling o Capabilities

The graphics above were produced with a small sampling of public data. Imagine the results if this was

operating over all your data, plus all the data of the Internet, in ways your analysts can interact with

and extract knowledge from. You can leverage a powerful temporal analytics engine designed to scale

to Internet size and empower your analysts and decision makers with this predictive power.

Recorded Future Foresite Inside Your Networks

You can have this capability running inside your networks on your servers. Most small deployments

of the Recorded Future system take ve Virtual Machines, and these can easily scale up to meet any

mission need. This architecture scales to billions and billions of records, but is simple to express and

understand. Of the ve Virtual Machines, two are for a data store (leveraging MongoDB), one VM is for

temporal processing, clustering and scoring, and one VM is dedicated to the analytics key-value store

for the UI. Another VM is for the front end.

6


8/10

CTOlabs.com

All common enterprise tools for data integration and ETL can be leveraged, and many are available aspart of the Recorded Future deployment if desired.

The Result

With Recorded Future, some of the most awe-inspiring analytical tools and visualizations in

available to humanity can now run inside your enterprise.

Analysts can take advantage of the information extraction, analysis and visualization capabilities

of Recorded Future to do things like search for inuencers in terror networks or nd the primary

money laundering points of interest or extract evidence of fraud from large data stores or detect

fraudulent visa applications. The architecture is modular and can be used in existing document enrichment pipelines.

Data can just as easily ow in and back out using APIs. Many analytic tools are a roach motel for

data it goes in but doesnt ow out in a nice way. Too many tools want to be the central data

store for the organization. Not so with Recorded Future Foresite. You can use Foresite that way if

you desire, but it can also participate well in your existing architecture.

Analysts can also take advantage of enhanced predictive power of who will be where when so

assessments and analysis can benet from this knowledge.

All the above can be done with all sources, not just open sources.

Concluding Thoughts

The analytic tools provided by Recorded Future helps extract the meaning and relevance of

information and apply this knowledge to organizational missions.

Recommendation

A proof of concept can be up and running in your enterprise in a matter of days. Your analysts can be

interacting with your data in new ways on very short order and you can decide the value of scaling this

system up to address more of your mission needs.

Recorded Future stands ready to support your mission with demos and a working prototype. Contact

[email protected] to schedule your proof of concept.

7


9/10


About The Author

Bob Gourley

The assessments here are from the perspective of Bob Gourley, an

intelligence professional with direct and personal experience applying

technology to the evaluation of our nations adversaries in operational

situations. As a Naval Intelligence ocer he was the rst director of

intelligence for the Department of Defenses Joint Task Force for Computer

Network Defense, where he worked with every element of the intelligence

community to enhance operational support to this emerging mission. Bob

remains a student of the cyber threat.

Following retirement from the Navy, Mr. Gourley was a senior executive with

TRW and Northrop Grumman, and then returned to government service as the

Chief Technology Ocer (CTO) of the Defense Intelligence Agency (DIA).

Mr. Gourley was named one of the top 25 most inuential CTOs in the globe by Infoworldin 2007, and

selected for AFCEAs award for meritorious service to the intelligence community in 2008. He was named

byWashingtonianmagazine as one of DCs Tech Titans in 2009; and one of the Top 25 Most Fascinating

Communicators in Government IT by the Gov2.0 community GovFresh. Forbes named him one of the most

inuential in Big Data in 2012.


10/10

CTOlabs.com

For More Inormation

If you have questions or would like to discuss this report, please contact me. As an advocate for better

IT in government, I am committed to keeping the dialogue open on technologies, processes and best

practices that will keep us moving forward.

Contact:Bob Gourley

[email protected]

703-994-0549

All information/data 2012 CTOLabs.com.

recorded future: detecting and analyzing changes through space and time

Documents