![Page 1: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/1.jpg)
Services for Sensitive Research Data
Iozzi Maria Francesca, Group Leader & Nihal D. Perera , Senior Engineer
Research Support Services Group”Services for Sensitive Data” University Center for Information Technology (USIT)University of Oslo
![Page 2: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/2.jpg)
Outline
Part I• What is “Services for Sensitive Data”- TSD• Project background- “ How TSD was established”• Prerequisites for getting access to TSD• System outline• Access control • What type of services a project can get?
Part II • Demo & How-to
Gard Thomassen,TSD 2.0
![Page 3: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/3.jpg)
Services for Sensitive Data - TSD
«Services for Sensitive Data» is an e-Infrastructure which provides a set of services to collect/register, to store and to analyze sensitive-data, in a highly secured enviroment.
Our services are recognized by : Norwegian Data protection Authority (DT) Regional Ethical Committee (REK) Norwegian Social Science Data Services (NSD)
Our services are designed to serve all the universities, high schools and other public research institutions in Norway.
![Page 4: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/4.jpg)
Project background
Pilot project from 2009
Full scale project from 2012
In production from may 2014
Expanding and changing ...
![Page 5: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/5.jpg)
Services TSD provides to researchers Information security, isolation and access control in compliance
with the directive on privacy and electronic communication
Large storage capacity (> 1 Petrabytes of Disk space)
High performance computing (HPC) resource (> 1500 cores)
High bandwidth
Accessible from anywhere in the world through proper mechanisms
A variety of software and databases
Data collection services (Nettskjema)
![Page 6: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/6.jpg)
What are the prerequisites for getting access to TSD services ?
Data should be classified as sensistive-data
An formal approval from one of the following institutions:
The Norwegian Data protection Authoruty
Regional Committee for Medical and Health Research Ethics(REK)
National Social Science Data Service (NSD)
![Page 7: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/7.jpg)
What is sensitive data?
![Page 8: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/8.jpg)
Contracting & Pricing
Contracting means having a Data Handler Agreement and a TSD Usage Agreement between the data owner (the institution your project belongs to) and the service provider (UiO/USIT).
Prices are defined within the TSD Usage Agreement. The contracting depends on the institution the project belongs to.
Link: http://www.uio.no/tjenester/it/forskning/sensitiv/ta-i-bruk/index.html
![Page 9: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/9.jpg)
Applying for a project (Register your project)
Before you can register your project, you should have obtained the formal approval from one of the above mentioned authorities.
Link: http://www.uio.no/tjenester/it/forskning/sensitiv/ta-i-bruk/index.html#toc5
![Page 10: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/10.jpg)
Access control - two –factor authentication
Smartphones or programmable hardware tokens
OATH-TOTP 2-factor authentication
![Page 11: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/11.jpg)
System outline
16
Gateway
HPC - ColossusVM-server
Storage
Internet
Secure encrypted network to special high volume data production sites
1 (project)
1 (storage area)
n 1
Gard Thomassen,TSD 2.0
![Page 12: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/12.jpg)
Using TSD
VM U1 S1
S1
TSD disk
VM U2 S1
GWUser1 Study1
Colossus disk
Colossus
Front endColossusUser2 Study1
TSDS1 DB
![Page 13: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/13.jpg)
Type of machines & services a project can get
A project can have Windows 2012 Server VM and/or a Linux Server VM
The VMs comes with a portfolio of software installed.
If requested each project get access to our HPC cluster – Colossus.
Each project gets their own virtual working enviroment in a dedicated VLAN/subnett.
A project can have many users
![Page 14: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/14.jpg)
Data import & export facility in TSD
“Sluice HD”
“Sluice –server”
“Project–server”
Virtual “Sluice –
server”
Project HD
Data copied here by ssh + scp (2-factor authentication)Encrypted data if sensitive
1
2
3
4
TSD 2.0
![Page 15: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/15.jpg)
![Page 16: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/16.jpg)
Demo
Step1:Connect your laptop to the uio-guest network. Open a browser window and order a UiO-guest account (you need your mobil!)
Step2:Are you a linux user? https://login.tl.tsd.usit.no/
Are you a windows user? https://view.tsd.usit.no (you have to select the “Install VMWare Horizon Client”)
![Page 17: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/17.jpg)
Demo
Step3:Download on your mobil the app: Google Authenticator (or FreeOTP for Android).
Open the App, select “configure” and then “scan barcode”The barcode is on the paper you received!
Step4:Use your username, password and OTP code to connect to TSD! Enjoy!
![Page 18: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/18.jpg)
Outlook
Filesystem and directories structure
How to change the password
How to import/export file
How to set up nettskjema in TSD
How to submit jobs on Colossus Cluster
I have done a mess! What shall I do?
![Page 19: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/19.jpg)
Filesystem and directories structure
Given that your project is pXX (here p77) there are:
Directories that visible to all the pXX-users pXX/data/durable (important stuff!) pXX/data/no-backup (not so important stuff!) pXX/data/colossus (hpc)
Private directories ( single user) pXX/home (exported to hpc)
Import and Export folders (all pXX-users) pXX/fx/import pXX/fx/export pXX/fx/15MC56NAUKWPN629/60044
(nettskjema) /shared/ read only to all! (data useful for everyone)
![Page 20: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/20.jpg)
Filesystem and directories structure
Panic! Where are my directories????
![Page 21: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/21.jpg)
How to change password?
Open a browser in TSD and type:
https://brukerinfo.tsd.usit.no
![Page 22: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/22.jpg)
How to import/export files in/out TSD?
NB: you need to have file-transfer protocol on your local machine (either sftp or FileZilla or winSCP) to connect to the filelock: tsd-fx01.tsd.usit.no
Export:1) in TSD drop your file in the /tsd/pxx/fx/export folder 2) on your local machine, login to the filelock and pick up the file!
Import:1) on your local machine, login to the filelock and drop the file 2) in TSD pick up the file from /tsd/pxx/fx/import folder
Demo live of the file import.
![Page 23: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/23.jpg)
How to import/export files in/out TSD?
Note of caution:
No files bigger then 1TB
No 100 files at the time. Better one tar-ball (or zip or 7zip)!
No special character in the file-names (norwegian characters etc)
Remove the file from the filelock once you have copied it!
Encrypt before transferring!
![Page 24: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/24.jpg)
How to set up nettskjema in TSD?
You can use nettskjema to run web-based questionnaire. The sensitive answers will pop up magically in TSD!
Create your Nettskjema form (https://nettskjema.uio.no)
Get the Form ID
Inform us:
![Page 25: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/25.jpg)
How to set up nettskjema in TSD?
... we need to do some magic to create for you the secret and public key pairs. Once we are ready:
Your nettskjema answers will appear in here:/tsd/pxx/fx/import/sns/SBHA5SJDKS8KW8/<FORM ID>/
The answers are encrypted. Use either Kleopatra or GPA (windows) or gpg (linux) to decrypt them.
The keys are stored in:/tsd/pxx/data/durable/pxxGPG/
With Kleopatra or GPA, you need to import the secret key at the first use:
![Page 26: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/26.jpg)
How to set up nettskjema in TSD?
![Page 27: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/27.jpg)
How to use Colossus (HPC)?
Your project needs to be configured in order to use Colossus resources (HPC) and must have a linux VM server
Connect to the linux server (https://login.tl.tsd.usit.no)
Open a terminal and write your sbatch script
Copy your data you want to compute on /tsd/pxx/data/colossus or /tsd/pxx/home/
Submit you script
![Page 28: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/28.jpg)
How to use Colossus (HPC)?
Software on Colossus are synced with the Abel software portfolio
Max run on colossus: 30 days!
Colossus has hugemem nodes (16GB node)
Attention: you can submit a job 30 day 4 hugemem nodes but it will cost several thousand NOK!!!
You can increase your memory, but if max-mem > mem-per-cpu then you consume more CPUs (because you are allocating more CPUs)
Be smart! Tune properly your job scripts.
![Page 29: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/29.jpg)
I have done a mess! What shall I do?
Have you deleted by mistake the entire data of the PhD thesis to be presented in one month?
Breath deeply and calm down....
Linux: cd /tsd/pxx/.snapshot/ here you find everything from teh last night!
Windows: ask us! We have the snapshot for you.
![Page 30: Services for Sensitive Research Data - Universitetet i oslo · Services for Sensitive Data - TSD «Services for Sensitive Data» is an e-Infrastructure which provides a set of services](https://reader033.vdocuments.us/reader033/viewer/2022051914/600599067d33c045317fa281/html5/thumbnails/30.jpg)
Get in contact with TSD
User Support: [email protected]
Contracting Support: [email protected]
…or in case of fire alarm:
Iozzi Maria Francesca, Group LeaderResearch Support Services Group”Services for Sensitive Data” [email protected]
Enjoy TSD!