FINALREPORT
OCTOBER2015
SubmittedasMilestone2.10under
ANLECProject0128
Sub-Project2
Preparedby
DrAdrianSheppard
DrDanielPalamara
DrOlafDelgado-Freidrichs
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page2
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page3
TableofContents
1.Background 4
1.1ProjectHistory 5
2.Outputs 6
2.1TheDDCA 6
2.2TheData 7
2.3TheUserGuide 11
2.4AUserWorkshop 11
2.5FutureWork 11
3.AppendixA.UserGuide 12
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page4
1. Background
ThisdocumentreportsonMilestone2.10ofANLECProject128:Maximisingthevalueof
digitalcoreanalysisforcarbonsequestrationsiteassessment.Thedocumentmilestone
objectivesare:
2.10Fullinternetaccessdigitaldatabasewithallconventionalanddigitalcoreanalysisdata,
userguide,andworkshoponsoftwareforallusers.
TheDigitaldatabaseisamajorcomponentofsub-project2andhaslinkstotheoutputof
othersub-projectsinANLECProject128:
Sub-Project1.Toderiveafullsuiteofspecialcoreanalysis(capillarypressure,supercritical
CO2:brinerelativepermeability)datasetsonplugsfromSuratbasinreservoirandsealrock
samples;
Sub-Project2.Toobtainhigh-resolution3Ddigitalimagesofthesameplugsamples,derive
digitalpetrophysicalandSCALdata,andperformpetrographicanalysesbyautomated
quantitativemineralmapping.Also,buildadigitaldatabaseofconventionalandDCA-
deriveddata,rocktypes,3Dimages,andstorageandsealpropertiesofcorematerialfrom
theSuratbasin;
Sub-Project3.Perform3DimagingofinsitusupercriticalCO2saturationattheporescale.
Correlateobservedsaturationrelationshipswithporositydistributionsandrocktypes;
Sub-Project4.Derivefasterandcheapermodel–basedanalysesofflow,storageand
CO2:brinedisplacementsindifferentrocktypesviareliableimage–basedmodelling;
Sub-Project5.Undertaketimeseries(4D)imagingandconventionalexperimentalstudiesto
measurethegeochemicalreactivityanddissolutiontrappingcapacityofcorematerialusing
supercriticalCO2;and
Sub-Project6.Integrateinformationobtainedatlaboratoryscaletowirelinelogdata.
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page5
Considerableachievementshavebeenmadeonthedigitaldatabase(hereafterreferredto
astheDDCA–DatabaseforDigitalCoreAnalysis);thisdocumentliststheoutputsofthis
projectwithinthecontextoftheMilestonedeliverables.
1.1 Project History
Amilestonereportfordeliverable2.4,entitled“Finalspecificationdocumentwithan
overviewoffunctionalityandarchitecturefordigitaldatabase”,wassubmittedtoANLECin
June2013.
Additionally,twoeducationsessionswereconducted.Afirsteducationsessionwas
providedforANLECstaffon6May2014.Thismeetinginvolvedalivedemonstrationofthe
DDCA,specificallybrowsingandenteringdata.Anoverviewofthestructurewaspresented
aswellasaschemaforhowthesystemwouldbeimplemented,whichincludeddatafrom
theSuratBasin.Theattendees,SandeepSharma,programmanagerfortheSouth-WestHub
projectandRickCausebrookandMelodyXiuhuiLifromANLECR&Dprovidedfeedbackon
variousaspectsofthedatabaseimplementation.
Asecondeducationsessionwasprovidedtoalargergroupon5June2014aspartofthe
2014ANLECR&DEastCoastReview.TheaudienceincludedANLECR&Dstaff(Noel
Simiento,RickCausebrookandMelodyLi),CTSCoTechnicalProgramManagerRobHeath,
formerANLECR&DResearchManagerJimUndershultz,aswellasindividualsfromother
relatedANLECprojects.ThescopeandpurposeoftheDDCAwasdiscussedatthismeeting,
particularlyhowitwouldbeimplemented,howstakeholderswouldaccessdata,whatits
currentandplannedcapabilitiesare(includingitssearchcapabilities)andwhatiswithinthe
scopeoftheDDCAproject.
AfurtherreportwassubmittedtoANLECfordeliverable2.7,entitled“DraftUserGuide”in
June2014.Thatreportoutlinedtheimplementationofthemajorityofthecomponents
describedinthespecificationdocumentfrom2013andalsodescribedthedatastructure
andvisualisationworkflowfortheimagedatathataccompaniesthemetadatainthe
database.Itprovidedacomprehensiveoverviewoftheconceptualandtechnicalframework
ofthedatabase.Assuch,thosedetailshavenotbeenrepeatedinthismilestonereport.
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page6
2. Outputs
2.1 The DDCA
Thefirstandmostimportantoutputisthedatabaseitself.ThisiscurrentlyhostedatNCI,
theNationalComputationalInfrastructure,andisaccessedwiththefollowingaddress:
https://ddca-staging.anu.edu.au
Thereisnolandingpageprovided,soonlyauthorisedusersareabletoaccessanypartof
thedatabase.AsmentionedtoANLECinface-to-facediscussions,thereisscopeforANLEC
orotherpartiestodeploytheirownversionofthedatabaseandhostthedataontheirown
server.ThiswouldmakeitmoredifficulttoprovidetheongoingsupportthattheANUand
NCIarepreparedtoprovideinordertokeeptheDDCAsiterunningforsomeyears.
AnotheroptionisforthesitetocontinuetobehostedatNCI,butaccessedthroughan
anlecrd.com.auwebaddress.
Asoutlinedinpreviousdocumentsandproposals,thedatabasefeatures:
(1) Thecapabilitytoautomaticallyharvesttomogramdataandotherproductsrelatedto
thedigitalcoreanalysisprocess.Thisisapowerfulanduniquecomponentofthe
database,andanareawheremanyfuturerefinementsarepossibleinordertomore
closelycoupletheuser-enteredmetadatawiththeautomatically-harvestedimage
data.
(2) AuthenticationprovidedviaGoogle'sgmailauthenticationprocess,whichmandates
thatallusershaveaGmailaccountforaccess.Asof2015thisappearstobethemost
secureexternalauthenticationsystemavailable.Asmentionedinthemilestone2.4
report,itwouldbeimpracticalandriskytodevelopaninternalauthentication
systemfortheDDCA.Accesstoindividualelementsisdeterminedbygroup
ownershipofeachelementandwhethertheuserisamemberofthatgroup.
(3) Comprehensivesearchandexportfunctionality,allowingusersandanalyststo
searchacrossalldataelementtypeswithinaprojectandexportdatawithuser-
selectedattributes.Thereisalsoscopeforthedata,exportedinCSV(orinJSON,that
is–JavaScriptObjectNotation)tobeamendedoutsideoftheDDCAandreimported.
Customsearchescanalsobedefinedandsavedforfutureuse.
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page7
(4) Intuitiveandflexiblebrowsingcapabilities,includingtheabilitytoquicklynavigateto
parentorchildelementsaswellassiblingelements,ortorapidlynavigateeven
furtherthroughthechain,toancestorordescendent.
(5) Qualitycontrolandcurationfunctionality,includingtheabilitytomodifyanddelete
elements,attachcustomdata,andmoveelementswithinthedatahierarchy.All
operationsarereversible,withnodataeverbeingpermanentlydeletedfromthe
DDCA.
Assuch,theDDCAnowsatisfiesallofthekeyfeaturesindicatedunderthe“Summaryof
Requirements”frompoint1.2inANLECMilestoneReport2.4(page4),whicharelisted
below:
ü Theabilitytostoreallrelevantdatatypesandtheirinter-relationships
ü Asecuritymodelthatincludesmixed-permissions
ü Datauploadcapabilities(automaticandmanual)
ü Extensibility(suchastheabilitytoincludecustomdata)
ü Dataqualitycontrolandcurationcapabilities
ü BrowsingandSearching
ü Inspectingandanalysingdataelements
2.2 The Data
ThenextcriticalcomponentoftheDDCAdeliverableisthecontent,whichcentresonthe
conventionalanddigitalcoreanalysisdatafromthevarioussub-projectsofANLEC0128,as
showninFigure1.Thesediversedatahavebeenuploadedintothedatabaseinvarious
forms,andaccompanythelargestoreoftomogramandotherimagedatathatare
generatedbythedigitalcoreanalysisworkflowthathasbeenappliedtothevarious
samples.
Mostofthedataofinterestaredirectlyattachedtotherelevantsub-plugs(informationon
searchingandnavigatingtosub-plugsisincludedintheDataGuide)asshowninFigure2,
andincludeimagesofthecoreplugsandscopingscansaswellasX-andZ-slicesofthesub-
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page8
plugsshowingmicroporosity,grainsizedistributions,mineralsegmentation,saturatedand
nativestatetomograms,porosityprofilesandmore.
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page9
Figure1.AnalysesperformedunderANLEC0128
3 4 6
SEM
+ QE
MSCA
N R
CA SC
AL W
hole-
Core μC
T Sc
oping
Scan
μCT
Pore
-scale
μCT
+ Seg
. W
hole-
Core μC
T Sc
oping
Scan
μCT
QEM
SCAN
+ Ge
o.
SEM
+ QE
MSCA
N R
CA SC
AL W
hole-
Core μC
T Sc
oping
Scan
μCT
Pore
-scale
μCT
+ Seg
. C
ore-flo
oding
Dyna
mic M
odelli
ng W
hole-
Core μC
T Sc
oping
Scan
μCT
QEM
SCAN
+ Ge
o. U
pscali
ng
H WC15 800.8 P2 1168.0P3 1170.0
E1 946.7 P5 1174.0E2 954.4 P6 1176.0 E3 961.9 P7 1178.2 E4 987.3 P8 1180.4 E5 988.7 P9 1182.0E8 1002.6 P10 1184.0 E9 1006.0 P11 1186.0 E10 P12 1188.0 E11 1032.1 P13 1190.2 E12 1035.0 P14 1192.0 E13 1038.0 P15 1194.0 E14 1039.9 P17 1200.0 E15 1040.7 P19 1204.0 E16 1043.7 P20 1206.0 E17 1048.7 P21 1207.9E18 1056.2 P22 1210.0 E19 1152.5 P23 1212.0E20 1155.4 P24 1214.0 E22 1161.8 P25 1219.0E23 954.0 P26 1220.9E24 966.0 P28 1224.9 E25 997.0 P29 1227.0 E26 1015.0 P30 1229.1 E27 1026.0 P31 1231.0 E28 1030.0 P32 1233.1 E29 1151.0 P33 1235.0 E30 1152.0 PU1 1163.2E32 1161.0 PU2 1163.9 E33 1163.0 PU3 1164.3 WC3 981.2 PU4 1164.6WC5 992.4 PU5 1174.4 WC8 1043.7 PU6 1174.5 WC9 1056.1 PU7 1195.4
PU8 1195.5 Notes: H = Hutton Sandstone PU9 1216.1
Seg. = Segmentation PU10 1216.3 Geo. = Geochemical PU11 1216.6
PU12 1216.8 PU13 1217.1 PU14 1217.3 PU15 1217.6 PU16 1217.8 WC11 1165.4WC14 1217.5
Sub-Project
Precip
ice Sa
ndsto
neEvergr
een F
ormati
on
Sub-Project
IDDepth(m)
1 2 5
IDDepth(m)
1 2 5
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page10
Figure2.SnapshotoftheDDCAshowingsomeofthedatatypicallyenteredforasub-plug(PrecipicesampleP22).Notethepresenceofcalculateddataenteredascustomdata(aplotofformationfactor)aswellasdescendantdatarelatingtoimageslicesandmetadatafromtheautomaticallyingestedtomograms.
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page11
2.3 The User Guide
Overall,theDDCAinterfaceisintuitiveand–becauseofthehierarchicalnatureofthedata–
straightforwardtonavigate.
Tofacilitatefamiliarisationwiththewebsite,auserguidehasalsobeenprepared–itis
presentedinAppendixA.Itfeaturesaquick-startguidetoallowuserstoquicklyaccessthe
searchandbrowsingfunctionalityoftheDDCA.
ItisintendedthattheUserGuidewillevolveandadaptwithincreasingclientuseofthe
DDCA.Itwill,overtimeandwithuserfeedback,bereplacedwithcontext-sensitiveinline
helpbuiltdirectlyintotheDigitalDatabaseforCoreAnalysis.Inthemeantime,itspurposeis
toprovidedirection(andinstructionsforeasyaccess)onthemainfeaturesoftheDDCA–
browsingandsearchingmetadatarelatedtodigitalcoreanalyses.Foramoredetailed
understandingofthephilosophyofthedatabaseanditstechnicalframework,usersshould
refertotheMilestone2.7report.
2.4 A User Workshop
WiththeDDCAfullyfunctionalandconsiderableamountsofdatauploaded,thedatabaseis
readytobedemonstratedinauserworkshop.Onewebinar,presentedbyAdrianSheppard,
wasorganisedbyANLECR&Dheldon6thNovember2016.Furtheruserconsultationwill
yieldimportantinsightsintouserexpectationsandareasinwhichthedatabasecanbemade
moreuser-friendlyandmorefunctional.AwebcastoftheDDCAinoperationwillbe
producedinearly2016.
2.5 Future Work
TheDDCAtodayrepresentsasecuredatabasethatisabletostoreandpresentdiversedata.
Itisbuiltonasophisticatedback-endarchitecture.Itwillenablethedatageneratedinthis
largeprojecttodelivervaluetotheend-usersoftheSuratbasinflagshipproject.Withthe
essentialframeworkinplace,theDDCAisnowatapointwherearelativelysmalladditional
effortcouldyieldmajorimprovementsindatapresentationandtheuser-friendliness,
pendingsufficientuserfeedback.Itisthereforerecommendedthat,ifpossible,some
additionalresourcesbeallocatedtotheongoingdevelopmentofthedatabaseinresponse
touserfeedback.
“maximisingthevalueofdigitalcoreanalysisforcarbonsequestrationsiteassessment”
©ANU2015|Page12
3. Appendix A. User Guide
Seefollowingpages
DatabaseforDigitalCoreAnalysis
(DDCA)USERGUIDE
VERS I ON 1 . 0
It is intended that this User Guide will evolve and adapt with increasing client use of the DDCA. It will, over time, be replaced with context-sensitive inline help built directly into the Digital Database for Core Analysis. In the meantime, its purpose is to provide direction and instructions for easy access on the main features of the DDCA – browsing and searching metadata related to digital core analyses.
January 2016
DDCA U s e r G u i d e 0 . 9
User Guide / Page 2
QuickStartGuideThe DDCA (Database for Digital Core Analysis) stores metadata from the digital and physical core
analysis workflows. Each entity in the hierarchy is referred to as a data element and represents a
conceptual and sometimes physical step in the analytical process. Initially, the parent element is the
project. Subsequent data elements will then converge on core plug elements, which represent the
physical entities created in the laboratory and subsequently examined using microCT. The core plug
data elements that are used for quantitative microCT petrophysical analyses will typically correspond
to real world-entities known as sub plugs, which consist of 3 mm to 8 mm cores taken from physical
core plugs, commonly 25 mm in diameter. It is from these sub plugs (recorded in the database as core
plugs, often with larger-diameter core plugs as parent entities) that the automatically-harvested
metadata and image previews (of the tomographic images upon which the digital core analyses are
performed) are often attached.
A screenshot of the top-level page of the online database is shown below. Access to the data is
through either the search interface or the browsing interface; the mechanism of each approach is
elaborated in later sections of this guide.
Use this lin
k to access th
e
search interface U
se this link to access the
browsing interface
Data Elements are displayed as clickable hyperlinks to allow browsing
and editing in all contexts
DDCA U s e r G u i d e 0 . 9
User Guide / Page 3
ContentsOverview ..................................................................................................................................................... 4
Understanding DDCA – A Conceptual Framework ..................................................................................... 5
The Interface ............................................................................................................................................... 8
Browsing .................................................................................................................................................. 8
Searching ............................................................................................................................................... 12
Data Dictionary ......................................................................................................................................... 28
3D Visualisation ......................................................................................................................................... 29
Drishti ................................................................................................................................................ 32
Voluminous ....................................................................................................................................... 41
DDCA U s e r G u i d e 0 . 9
User Guide / Page 4
OverviewThis DDCA User Guide is developed with the expectation that there are three ways in which users
interact with the database (input, access, export) and two main types of users (analysts and
c l ients):
1. Analysts - typically engineers or laboratory staff associated with the ANU microCT laboratory or
FEI Lithicon - will primarily use the database to input information related to workflows as well as the
results of laboratory analyses or digital core analyses. Analysts might also exploit the hierarchical
and structured nature of the data – and the fact that the image metadata are automatically ingested
to the DDCA database – to enable easy access to analysis results.
2. C l ients – such as reservoir engineers, modellers, geologists and project managers – will access
the database for online viewing. Some will also export data (for example, tables of derived
properties such as porosity or permeability) for analysis or visualisation in other systems. This user
guide is aimed primarily at c l ients.
This guide covers three aspects of the database:
(1) The first section provides a general overview of the conceptual framework for DDCA.
(2) This is followed by an explanation of the interface, including the search functionality, as this is
the medium through which users will access the metadata within DDCA.
(3) The user guide then presents a data dictionary, which a user should understand in order to
successfully navigate the data within DDCA and modify it if necessary.
Further items, such as advice for 3D visualisation of tomographic data using external products such as
Drishti, are included in Appendix A.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 5
UnderstandingtheDDCAThe DDCA (Database for Digita l Core Analys is) is designed to provide an integrated
environment for the access and management of the 3D datasets from microCT
analyses as wel l as the accompanying workf lows (for more detai l see Project 128
Milestone 2.9 Report) .
Figure 1: Overview of a typical workflow applied to core material
The purpose of the DDCA is to assist analysts and end-users to manage and understand the workflow,
that has been applied to core materials, of which an example is shown in Figure 1. The workflow
aspect of the database is facilitated by having a parent-child relationship between data elements.
Each element has a parent, and the parent chain will eventually lead to the parent project. Children of
each data element will generally represent subsequent steps in the digital and physical core analysis
workflow.
The data elements, shown in Figure 2, can be conceptually divided into two categories. The first
category covers the user-entered data elements that provide the contextual framework for the digital
core analysis as well as representing part of the physical workflow. That is, the first category
comprises the data elements (see the later section on the data structure of the database) that relate
DDCA U s e r G u i d e 0 . 9
User Guide / Page 6
to the origin of the core plug or sub plug that is eventually used in the microCT imaging process and
the digital core analysis. These data elements will typically include the PROJECT, the source of the
geological material (a WELL or OUTCROP), optionally a WHOLE CORE section and other information
relating to the WELL such as GEOLOGIC UNITS and WELL LOGS, and the CORE PLUG extracted from
the WHOLE CORE. The critical data element here is the CORE PLUG – and often there will be a chain
of CORE PLUGS representing cases where a sub plug has been extracted from a plug by physical
coring.
Figure 2: An overview of the DDCA Data Element Hierarchy
There is also a second suite of data elements relating to data that are automatically ingested
(harvested from archival data storage). This second category captures all of the data elements that
are not manually created by a user. Instead, they are generated through analytical processes and
form part of a rigid workflow in which various types of 3D images and derived entities (for example,
tomographic images, segmented images and pore network models) are obtained from parent image
acquisitions such as tomograms through an image analysis and computational pipeline. It is these data
elements that are of greatest interest to analysts and clients and it is from these data elements that
the derived properties such as porosity and permeability are calculated.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 7
Project
Well
Well Log
Core Plug
Measured Properties
Whole Core Tomograms Segmented Images Pore-Networks SEM Images
Geological Unit Derived Properties
Outcrop Core Plug
F igure 3: A potential hierarchy of user-entered and automatical ly- ingested data, connected at the core-plug level and showing the ful l workflow for digital core analysis.
Coupling the user-entered data from the first category of elements to the automatically-ingested data
in the second suite of data elements (that is, the products of the imaging process) completes the
workflow chain, shown in Figure 3. This coupling generally occurs the CORE PLUG level since the core
plug is the physical entity to which the digital workflow is applied (keeping in mind that in the case of
sub plugs, CORE PLUGS can be entered as children of other CORE PLUGS ad infinitum), as shown in
the figure above.
Thus, the database allows digital core analysts to easily link the physical core specimens with the
imaging workflow they have undergone, and also to link analyses (for example, laboratory-based
measurements such as MICP) to the output of the micro-CT imaging process (this output is typically
the segmented data image data). End-users are therefore able to correlate petrophysical properties
calculated during digital core analysis with laboratory measurements and with representations of the
3D image data such as orthogonal slice images and 3D sub-sampled images.
Automatically-ingested data User-entered data
DDCA U s e r G u i d e 0 . 9
User Guide / Page 8
TheInterfaceThere are two approaches to accessing data within the DDCA Digital Database for Core Analysis:
browsing and searching. As shown in the Quick Start Guide, The browsing interface can be
accessed by clicking on the Projects link at the top right of the screen, inside the top panel that
features the DDCA logo. The search interface is located alongside and is labelled Search.
BrowsingAt all times, each data element can be
investigated by clicking on its name. This
reveals an interfaces that features the
type and name of the chosen element
and, under various panels, its re lat ions:
• Ancestors – that is, all of the
data elements in the hierarchy
above;
• Parent – the direct parent of
each elements;
• Sibl ings – elements of the same
type with the same parent;
• Children – data elements that
sit directly below the chosen
element;
• Descendants – all data elements that sit below the chosen element.
The relation panel (shown in Figure 4) allows a user to easily traverse the digital core analysis
workflow – each data element listed within is a clickable hyperlink which will take the user to a new
relation panel that focuses on the chosen element and lists the aforementioned relationships. A user
can therefore readily traverse the workflow chain simply by clicking on subsequent relations of a
chosen data element.
Below the relation panel, the DDCA interface displays an attr ibutes panel. This lists the details of the
chosen attribute – further information on what attributes correspond to what data type are shown in
Figure 4 screen shot showing par t of the relatio n panel
DDCA U s e r G u i d e 0 . 9
User Guide / Page 9
the following section, which describes the data dictionary. If the user has sufficient access privileges,
this panel features an EDIT button that allows the user to update or change the attribute data.
Finally, the interface features a custom data panel. Custom data may be of four possible types:
• string (text),
• number,
• Boolean (true/false),
• an attachment (a link to a file and a text description describing the file).
This allows for any data element to have supplementary information attached by users with edit
access rights. Importantly, the custom data follows a key-value system. For each entry of custom data
the user should specify the “key” for the chosen value. The key can be considered a label; consistent
use of labelling will facilitate searching and data export. For example, a measured property could be
attached to a core plug element by choosing the edit option under custom data, entering a key value
of Framework Grain Content %, a type of number and a value of 69.2, as shown in Figure 5.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 10
Figure 5: Screen shot showing the attachment of a measured property to a core plug element after choosing the edit option under custom data
DDCA U s e r G u i d e 0 . 9
User Guide / Page 11
Users with the appropriate level of access can modify the structure of the data. As seen in Figure 6
they can delete the element being shown in the
browsing interface, create a child element (they will
be prompted to choose the type of the new child
element from a list of legitimate options), or adopt a
child.
Creating a new child element is straightforward as
seen in Figure 7 – the user is given the option of
specifying the TYPE of the new data element, and its NAME. The user can then follow the newly-
created hyperlink and, by using the EDIT button under the attributes subheading, populate the
relevant attributes.
Figure 7: Example of edit ing: the dialog box for adding a new chi ld data element
Figure 6 Screensho t sho wing how users with ‘edit ’ access can change the structure and con tent of the data
No data is ever truly deleted from the
DDCA and all operations are in principle
reversible. No user interface is yet
provided for undoing changes made in
error, this currently requires database
administrator assistance.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 12
Adopting a child element allows the user to attach an element – and all its descendants – to a chosen
element as its child. The element being attached could be an orphan (not have a parent – typically
data that have been harvested before the matching user-generated data have been entered) or may
already be part of a chain of data elements. If so, it will be moved from its original chain – ALONG
WITH ALL OF ITS DESCENDANTS – and will be given the current element as its new parent.
The process of adopting an existing element (shown below) only requires one piece of information -
either the ID or URL of the child element which is to be adopted. The URL is usally readily obtained by
(in a Windows or Linux environment) right-clicking on the target child data element’s hyperlink within
the DDCA. The ID is the alpha-numeric sequence that forms the last part of the URL and it is generally
easier to supply the entire URL for this process.
Figure 8 Dialog for attaching an exist ing element to a new parent ( i .e. ‘adopting’ it)
All three operations – deleting an element, creating a child, adopting a child, manifest in the database
immediately.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 13
SearchingUsers can use the search interface to find
specific data (based on its type, attributes
and more) and to display lists of data
elements with selected properties. The
search interface is accessed through the
search hyperlink at the right of the
topmost panel (featuring the DDCA logo;
see the Quick Start Guide at the beginning
of this document).
This section describes the search interface
(as shown in Figure 9) both in a general
sense and through the example of the
some of the pre-defined searches that are
stored within the DDCA (these are listed
below the search interface as hyperlinks).
Note, the results of a search process are
presented in a separate list (as shown in
the following sections). This list sits to the right of the search interface, except in cases where the
browser window is narrow. In these cases the search interface will reside in a query tab and the
results in a separate tab. Within the results list, each data element presents as a clickable hyperlink
that browses directly to the chosen data element in the same manner as the browse interface.
Figure 9 The toplevel searching dialo g, that expands as search terms are added
DDCA U s e r G u i d e 0 . 9
User Guide / Page 14
TheSearchElementsThere are five elements to the search interface:
1. the scope limits the search to all elements or makes the search relative to a chosen element
(specified by its URL), which allows the user to search the element’s ancestors or descendants
or children (recall, children are the direct descendant data elements, whereas descendants
extend all the way to the end of the data chain).
2. the data type option allows the user to limit the search results to a chosen data type (for
example, only core plugs). The user should use the option “imported-dataset” here if they
wish to only search among the automatically-ingested image data such as tomograms.
3. the condit ion panel provides limits on the search results,
4. the display panel allows the user to specify additional attributes to show in the results, and
5. Optionally, the user can control how many search results will be shown. The default is 50
records, and it is useful to maintain this value while developing a search, increasingly it only
when results are satisfactory. Showing many records (the options are 20, 50, 100, 200, 500
and 1000) may make the database timeout if the search query was poorly structured.
These search elements are discussed further below.
Scope:
This specifies whether a search extends through all data elements that the user can access (“entire
database”), or through the relations (ancestors, descendants, children) of a specific data element, a
specific data type, or elements with specified properties.
For example, to search only through the descendants of the acquisition dataset WW1_P22, one
would select "descendants of" and “base_element”, paste in the URL for WW1_P22 (https://ddca-
staging.anu.edu.au/entity/902c3ec8-7b39-4dfb-b6c1-fdc78718f8e9 - this can be right-clicking on the
link and selecting Copy Link Address in Chrome or Copy Link Location in Firefox) in the “ID or URL”
field. The "base element" itself is not included in the search. The results are shown in Figure 10.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 15
Figure 10 Example of the results obtained from a search for al l descendents of a data element
The URL for
WW
1_P22 goes here
DDCA U s e r G u i d e 0 . 9
User Guide / Page 16
The scope functionality can also be used to easily search for relations of a particular data element –
for example, children of whole_core_section[s] shown in Figure 11.
Figure 11 Search results showing al l chi ldren of whole-core elements – showing the PU ‘upscal ing’ series
DDCA U s e r G u i d e 0 . 9
User Guide / Page 17
The scope functionality can also limit the search to relations of data elements with certain
characteristics, as in the case of the predefined search “SEM Registrations”. In this case the search is
limited in scope to descendants of elements where [attribute] name begins with “tomoSEM”, shown
in Figure 12.
Figure 12: Search example showing how to l imit the search results by adding Condit ions, in this case a constraint on the name
DDCA U s e r G u i d e 0 . 9
User Guide / Page 18
Data type:
Allows the user to indicate that the results should only show elements of the specified type. If "- any
data type –" is selected then all elements will be shown regardless of type. The effect of the data
type element is evident in the predefined search, “Well Search” shown in Figure 13
Figure 13 Search example: f ind al l data elements of a particular type (Well)
DDCA U s e r G u i d e 0 . 9
User Guide / Page 19
Note, all tomogram data are collectively harvested as imported_dataset types. So to view all
tomograms for, say, whole core sample CT1, one would search as shown in Figure 14 (note, the
search results have been truncated in the image shown):
Figure 14 Search example: f ind al l descendants of a particular element, identif ied by its URL (address)
The URL for CT1 goes
here
DDCA U s e r G u i d e 0 . 9
User Guide / Page 20
Condit ions:
This interface allows the user to specify additional conditions
for the search; only data elements that match the chosen
conditions will be returned by the search. The attribute field
auto-completes with a list of possible properties, so the user
does not need to know the exact spelling or data element
structure in order to complete a search.
For example, to look for an acquisition dataset with a sample
name that includes, say, the string "E23", one might type
"spec.sample.sample_name" under "attribute", then select "contains" from the drop-down menu and
finally type "E23" in the text box beneath it. Here, "spec.sample.sample_name" means something like
"take the value of the spec attribute, within that the value of the sample attribute, and within that,
the value of the sample_name attribute".
There are also limited wildcards for structured attributes. An asterisk (*.) at the beginning can stand
for any initial sequence of attribute names of dots. For example, "*.sample_name" would find
"spec.sample.sample_name", but also, say, something like "parameters.acquisition. sample_name" if
that existed. The asterisk only works at the beginning.
If the browser width is
insufficient the search interface
is automatically partitioned into
two panels, one featuring the
query and the other the results.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 21
The example shown in Figure 15 returns core plugs between depths of 1000 and 1050 m and also
displays the depth attribute (the display option is described in the following section).
The condit ions search element is commonly used to pinpoint desired data from the database. See,
for example, the structure of the predefined “Acquisition Search” shown in Figure 16, which limits the
results to data of type “imported_dataset” with the condition that the process equals “Acquisition”
(note, this field is not case-sensitive):
Figure 15: search example: f ind a l l e lements o f a p art icular type wi th an attr ibute in a numer ical ran ge
DDCA U s e r G u i d e 0 . 9
User Guide / Page 22
Figure 16 Search example: (predefined “acquisit ion search”) f ind al l data elements where the ‘process’ that created them is named ‘acquisit ion’ – in this case the results are sets of projection data prior to tomographic reconstruction
Note, in order to explore potential options for an attribute condition, the user can include the desired
attribute as a display attribute from the matched element (the display search element is described
in the next section). An example of how that can be used to explore, for example, the range of values
for the “process” attribute is shown in Figure 17. From the results (not shown) it is evident that there
are numerous process types for imported datasets, including Acquisition, Clusters_Region_Merging,
Multi_Subset, Subset, Network_From_Labels, Registration_Resample and so on.
The condit ions element features eight states – equals, begins with, contains, is in range, does not
equal, does not begin with, does not contain, is not in range.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 23
Figure 17 Search example: an attr ibute ( in this case ‘process’) can be included in the display f ield in order to get a feel for potential values, which can then be used in the condit ions f ield when searching.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 24
D isplay:
The display options provide functionality for the user to specify which properties attributes should be
shown in the results table. Specifying attributes in this field allows the user to create a custom table
showing specific properties for the data elements listed in the search results. For example, Figure 18
shows a search process that returns core plugs and shows the diameter attribute in the results. As is
the case with the condition panel, the attribute field in the display panel auto-populates based on
possible values, so the user does not need to know the exact spelling or data element structure in
order to complete a search.
Figure 18 search example: the search interface showing a display attr ibute (diameter).
DDCA U s e r G u i d e 0 . 9
User Guide / Page 25
The display functionality is quite powerful – it allows for the results to be shown as a table or a
scatterplot (if the format of the resulting data allow it) and can show user-determined attributes from
either the “matched element” (that is, the elements found when the conditions in the scope, data
type and condit ions panels are met) as well as attributes from relations (ancestors, descendants,
children). These relation attributes can also refined based on the conditions of the chosen attribute in
the relation data element (possible conditional tests include equals, begins with, contains and so on –
the same eight as in the conditions panel).
Examples of the use of the display functionality are shown in the following section, which describes
a few of the predefined searches.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 26
SearchingforSub-Plugs
One common search purpose would be to find sub-plugs. In the database, sub-plugs are stored as a
core-plug data element type (see the Data Dictionary in the following section), but will have core-
plugs as parent elements.
Parent attributes are not accessible in a search through the CONDITIONS field, however. To extract a
list of sub-plugs the user should modify the SCOPE of the search to include children of core plugs and
the returned data type as core plugs, as shown in Figure 19.
Figure 19 Searching for sub -plugs.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 27
Exportingdata
The database provides an option to download
the search results in JSON or CSV format; the
DOWNLOAD option can be accessed below the
search panel as shown in Figure 20.
There is an additional CSV Template type for
users wishing to modify the data elements
returned in the search results. For example,
changing attribute values on all the returned
data elements, or perhaps creating child
elements from external data for the returned
data elements. For the CSV Template
download format the user is able to choose
which attributes are included in the
downloaded table. To actually modify the
DDCA data elements the user needs to convert
the CSV table into JSON data using a script
(external to the DDCA), before uploading the
JSON data at a separate page on the DDCA.
Figure 20: The search panel; no te the ‘Download Results ’ button
DDCA U s e r G u i d e 0 . 9
User Guide / Page 28
DataDictionaryThe DDCA interface automatically populates attributes field during searching, and lists the full suite of
properties during element editing. Nevertheless, it is useful for the user to have some understanding
of the data elements and their properties/attributes, as shown below for the user-generated data.
Project Outcrop WellLogName Elevationinm NameOrganisation Datum LogtypeProjectLeaders Surfacelocationin ResolutionNotes Latitude Units Longitude ScalePossibleChildren: Samplecollectedby ImageofLog-CaptionWell Sampleprovidedby ImageofLog-FileOutcrop Datesampled Env.Anddepthcorrection
Datesupplied Logdata-DepthWell Notes Logdata-ValueName NotesState PossibleChildren: Nameofbasin WholeCore CorePlugorSub-PlugField CorePlug NamePermit GeologicUnitorSub-Unit DiameterinmmENO/GAID LengthinmmPPDMID GeologicUnitorSub-Unit DepthinmPressureDBID Name PlugorientationStatus- GAStratigraphicunitID SedimentologicaldescriptionSpuddate DepthRange-From BoxnumberDatereachedTD DepthRange-To PlugPhoto-CaptionDaterigreleased Notes PlugPhoto-FileTotalDepth(driller’s) NotesDriller'sdatum PossibleChildren: TotalDepth(logger’s) GeologicUnitorSub-Unit PossibleChildren:Logger'sdatum MeasuredPropertyofPlugLocation WholeCoreSection CalculatedPropertyElevationinm Name ImportedDatasetDatum UnitID SEM-EDSImageSurfacelocationin Sub-UnitID SEMImageLatitude Diameterinmm Longitude Lengthinmm MeasuredPropertyofPlugCompany Depthinm NameTypeofwell Sedimentologicaldescription PhysicalpropertyBitsizeinmm Intervalinm-Top UnitsPreservedcore Intervalinm-Bottom ValueNotes CorePhoto-Caption LabTechnique…Preservedcore CorePhoto-File LabIDNotes Notes Date NotesPossibleChildren: PossibleChildren: WellLog CorePlug WholeCore GeologicalUnitorSub-Unit CorePlug KeytoPropertyFormats:TextNumberDate
DDCA U s e r G u i d e 0 . 9
User Guide / Page 29
3DVisualisationThe DDCA is primarily a system for managing metadata, curating data and representing workflows.
High-performance visualisation of the data, particularly in 3D, is beyond the scope of the database
itself.
There is, however, potential for industry-standard visualisation of data stored in the DDCA using freely
available tools developed at the Australian National University, both in the Department of Applied
Mathematics and in the National Computational Infrastructure (NCI) VizLab. This software-
development work has occurred in concert with developments at the ANU x-ray micro-CT imaging
facility.
These tools are Drishti and Voluminous. As well as the 3D visualisation that these tools offer, users
can interact with large 3D datasets in a 2D slice-wise manner, or overlay individual slices, using the
NCViewer slice viewer (developed by the Australian National University and to be made available to
users of the DDCA). This section provides an overview of each tool and provides a more in-depth
explanation of the use of Drishti, which is routinely used for 3D visualisation in a commercial
production environment. By using Drishti, DDCA users will therefore have access to industry-standard
visualisation capabilities.
Down-scaledvolumesforvisualisation
Volumetric data require considerable space for storage and processing power for visualisation. The
DDCA therefore provides access to downscaled raw tomograms and segmented images for use in 3D
visualisation, as well as 2D slices for use when the user is not able to use 3D methods.
Voxels in volumes from µCT are often stored as 16-bit integer values, with the number of voxels in
each dimension being a function of the sample size (diameter and height) and the detector
configuration and capabilities. Within a volume, each voxel requires two bytes to store and so file
sizes are slightly more than the product of the X, Y, Z dimensions multiplied by two (the metadata
being the reason that the file sizes are “slightly more” than size of the raw image data). A typical
image, of 1760 x 1760 x 3500 voxels, therefore measures around 20 GB, which is not amenable to
visualisation on anything but the highest performance computer hardware. Nor can such volumes be
downloaded over the internet within any reasonable period. In addition, such images of complex
geomaterials usually contain too much detailed information to be visualised in three dimensions.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 30
Therefore, the DDCA provides downscaled versions of volumes – in the form of raw µCT and
segmented images – for user visualisation. The volumes are downscaled four-to-one in each
dimension, usually resulting in 480 voxels in the X and Y dimensions (with the Z dimension remaining
a function of the sub-plug height). These volumes will be stored in their native netCDF file format,
which can be readily imported into Drishti, and NCViewer. For import into Voluminous the user will
need to use the Drishti import tool to convert the netCDF file into an MHD (meta-image) file set.
The system will also provide tiff (tagged image file format) slices for volumes, which can be viewed in
NCViewer, but are generally not as useful for interpretation as the 3D volumes viewed in either
NCViewer or Drishti.
NCViewer
NCViewer provides interactive visualisation of 3D volumes stored in netCDF and is routinely used in a
production environment to explore 3D dataset, choose values for processing parameters (for
example, thresholds for segmentation) and investigating the outcomes of processing operations.
NCViewer is freely available on request from ANU.
The software allows a user to traverse through stacked layers of a 3D tomogram or segmented image.
NCViewer only provides 2D representations of the data, though it is important not to underestimate
the importance of this tool for quantitative analysis of images. Firstly, the mapping between voxel
values intensity is well defined, so that one can see the actual data values. Secondly, one can overlay
images upon one another, allowing comparison between original and processed images. Thirdly, one
can zoom in until the values of individual voxels are evident. Finally, one can test the results of
applying thresholds or conduct measurements of feature size or contrast and image noise levels. For
these reasons, NCViewer, rather than 3D visualisation is an integral part of the quantitative analysis
workflow in a production environment. In particular, NCViewer allows the user to determine the
thresholds and gradients that will be used for image filtering and segmentation; critical steps in
creating binarised data from which calculated properties (for example, porosity, permeability, pore
networks) are derived.
For example, as Figure 21 shows, one can interactively change the thresholds, and the colours
assigned to them, on the slice histogram. It also allows users to manually paint tomograms, calculate
statistics on particular regions of the tomogram, measure distances, and export the tomogram as a
series of images.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 31
As with visualisation via Drishti, a user can export the downscaled netCDF data from the DDCA and
readily import it into NCViewer for exploration, as shown in Figure 21.
Figure 21 The NCViewer interface, shown h ere depict ing layer 61 from a tomo gram im age block .
DDCA U s e r G u i d e 0 . 9
User Guide / Page 32
Drishti
Drishti is cross-platform desktop software for volume rendering developed by Ajay Limaye at the NCI
Vizlab and released under the GNU GPL v3 open software licence. Drishti, which is freely available via
download from google code, has been in development for over 12 years and is a stable, mature
product with an active user base of over 1,000 users. It is routinely used in a commercial production
environment for the visualisation of µCT tomograms and segmented (Mango-processed) images.
Designed for the exploration of volumetric datasets, it allows users to import image or volume files,
downscale them as required and then render up to four volumes up to 1024 x 1024 x 1024,
depending on the graphics hardware on the host computer. The user is able to interactively
manipulate the volume, rotating, zooming and displacing it, altering its colour, shading and
transparency, and taking slices in any axis and on any angle as well as exporting all views as static
images or movies.
CT or MANGO image is uploaded into DDCA
DDCA creates downscaled
version (4-to-1 in each
dimention)
User downloadeds downscaled
version (netCDF) to
desktop
User imports netCDF into Drishti (or
NCViewer) for visusalisation
Figure 22 Gener ic workf low for 3D vi sua lisat ion of data from th e DDCA
DDCA U s e r G u i d e 0 . 9
User Guide / Page 33
Figure 23 3D Visual isation of a downscaled tomogram and segmented volume using Drishti software.
Figure 24 The Drisht i import package a l lows a netCDF f i le from the DDCA to be adjusted b efore 3D vi sua lisat ion.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 34
UsingDristhi2.4forvisualisationofDDCAdata
Interested users can gain a general understanding of the use of Drishti through the online user guide,
help videos and Drishti User Group, all of which are accessible from the Google Code page on which
Drishti is hosted: https://code.google.com/p/drishti-2/
This section describes the basic steps involved in visualising a downscaled tomogram from the DDCA
into Drishti, paying particular attention to the steps that are unique to data from digital core analyses.
Specifically, it covers the visualisation of segmented images, which have discrete, rather than
continuous, histograms and therefore require a different approach for shading than regular
tomograms. For example, Figure 25 Figure 26shows a raw µCT tomogram image – note the
continuous histogram (highlighted in red) – whereas the histogram for a segmented image shows
discrete intervals and requires a separate transfer function for each one (Figure 26).
The general process for visualisation a downscaled volume from the DDCA in Dhristi is:
(1) download the downscaled netCDF volume (either a raw tomogram or a segmented volume)
from the DDCA. Typically, the segmented image is shown together with the raw tomogram,
so both volumes should be opened in turn.
(2) open the first volume (either the raw tomogram or segmented image, the order is not
important) using the Drishti Import program:
DDCA U s e r G u i d e 0 . 9
User Guide / Page 35
(3) Adjust the histogram to achieve maximum contrast in the image – this step is critical as it
will change the values in the exported volume:
DDCA U s e r G u i d e 0 . 9
User Guide / Page 36
(4) Save the processed volume as a Drishti image (*.pvl.nc) – access all options that arise during
the export process in their default state.
(5) Repeat the process for the second volume (to ensure that both the raw tomogram and the
segmented image have been ‘imported’).
(6) Close Drishti Import and open Drishti.
(7) Load both volumes together, as shown below.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 37
(8) The results are shown below:
(9) Now create more transfer functions (the Transfer Function Editor is located in the top right
of the image above) to capture the discrete distribution of the segmented image – one
transfer function is needed for each ‘peak’.
(10) Ensure that the tick box for the new transfer functions are in the column that corresponds
to the segmented image (depending on what order it was loaded).
DDCA U s e r G u i d e 0 . 9
User Guide / Page 38
(11) Adjust the histograms for the raw tomogram and the segmented image – the segmented
image should have one transfer function on each peak in the histogram, as shown below
(note the green transfer function, which coincides with the first peak in intensity in the
segmented volume):
DDCA U s e r G u i d e 0 . 9
User Guide / Page 39
Figure 25 3D Visua lisat ion of a down sca led tomo gram for W est W andoan sam ple P12. TF (Transfer Fun ctio n) 0 corresponds to the raw tom ogram. TF 1, 2 , and 3 co rrespond to the segmented image and a shown inFigure 6. Note the d iscrete distribut ion of the segmented image histograms.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 40
Figure 26 3D Visua lisat ion of a downscaled segmented volum e for West Wandoan sample P12. Th ree transfer fu nctio ns have been created to capture th e three discrete sect io ns of m ateria l in the image histogram; transfer funct ion 1 is vis ibly act ive and covers the fi rst sectio n in the histogram; transfer funct ions 2 and 3 (n ot vi sible) have been ass igned di fferent colou rs and cover the next two discrete gro upings in the h istogram (marked by arrows).
DDCA U s e r G u i d e 0 . 9
User Guide / Page 41
Voluminous
The Voluminous 3D visualisation software is being developed by the National Computational
Infrastructure (NCI) VizLab at the Australian National University. Voluminous may one day have the
capability to accommodate the visualisation of full-scale netCDF data within the DDCA, given
sufficient rendering hardware on the DDCA web server.
Voluminous is remote volume-rendering software deployed as a cloud service. Data are stored and
visualisations rendered on a remote server with high-performance graphics capabilities. Users can
access and manipulate the visualisations in real time via their web browser (Figure 27).
Cloud-based storage and visualisation presents potential difficulties related to security and bandwidth
as files currently need to be uploaded to the remote Voluminous server. Security concerns are
ameliorated by the fact that the data from Mango will already reside on ANU secure servers. Ideally,
Voluminous will be deployed on the same server as the DDCA, so that the large data sets would not
need to be moved. In this configuration, users would identify a dataset for visualisation from the
DDCA environment, then be transferred to Voluminous, most likely by opening a new tab in their
browser. The data would immediately be available for visualisation with Voluminous. While this
behaviour is part of the long-term roadmap for Voluminous, it is not available today nor part of short-
term plans. In the meantime, users will need to export data from the DDCA then upload it for
visualisation with Voluminous.
DDCA U s e r G u i d e 0 . 9
User Guide / Page 42
Figure 27 Visua lisat ion of data from the DDCA us ing the cloud-based system, Vo luminous