Activities of Computer Division, BARC

B. S. JAGADEESHComputer DivisionBARC, Trombay, MumbaiJag@barc.gov.in

Presentation outline

• What have we done so far?

• What are we currently doing?

• Typical Anupam Applications (in-house)

• What do we propose to do in near Future? The issues?

High Performance Computing – The ANUPAM Project

• To Achieve Supercomputing Speeds– At least 10 times faster than the available sequential machines

in BARC

• To build a General Purpose Parallel Computer– Catering to wide variety of problems– General Purpose compute nodes and interconnection network

• To keep development cycle short– Use readily available, off-the-shelf components

ANUPAM-860 Block Diagram

•Intel i860 processors

•Intel MultiBus-II backplane

•Upto 8 nodes in a MB-II•Bus based architecturebackplane

•Upto 16 nodes of i860XPprocessors

ANUPAM - Alpha

• Each node is a complete Alpha workstation with 21164 cpu, 256 MB memory, Digital UNIX OS etc.

• Interconnection thru ATM switch with fiber optic links @ 155 Mbps

NODE-01 NODE-02 NODE-03 NODE-04 NODE-05 NODE-06 NODE-07 NODE-08

NODE-09 NODE-10 NODE-11 NODE-12 NODE-13 NODE-14 NODE-15 NODE-16

FILE SERVER FAST / Giga ETHERNET SWITCH

CAT-5 CABLE

UPLINK

Typical ANUPAM x86 Cluster

Evolution in Hardware

• Compute Nodes:– Intel i860– Alpha 21x64– Intel x86

• Interconnection Network:– Bus : MultiBus-II, Wide SCSI– Switched Network: ATM, Fast Ethernet, Gigabit Ethernet– SAN: Scalable Coherent Interface

Evolution in Software

• Parallel Program Development API: – ANULIB (Proprietary) to MPI (Standard)

• Runtime environment– I/O restricted to master only (860) to Full I/O (Alpha and x86)– One program at a time (860) to Multiple Programs to Batch

operations

• Applications– In-house parallel to Ready made parallel applications– Commercially available parallel software

ANUPAM clusters

Sustained speed on 84 P-III processors: 15 GFLOPSYear of introduction :- 2001

Year: 2002, Sustained speed on 64 P-IV cpus : 72 GFLOPS

Sustained speed on 128 Xeon processors :- 365 GFLOPSYear of introduction :- 2003

ANU64

ASHVA

ARUNA

Anupam-Ameya 512 (1.73 Tera Flops )

Anupam-Ajeya 1152 is delivering 9.0 Tera Flops !

Software Development

• Program Development Tools – Libraries, Debuggers, Profilers etc.

• System Software– Communication drivers, monitors, firmware– Job submission and queuing system– Cluster file system

• Management and Monitoring tools– Automatic installation– Cluster Monitoring System– Accounting System– SMART – Self Diagnosis and Correction system

Cluster Monitoring and Information System

Single interface for monitoringthe health of the system

Detects error conditions and gives alerts to the administrator

Provides different views of thesystem – Cluster view, job view,metric view etc.

Medium Range Weather Forecasting, Delhi2+8 -node ANUPAM-Alpha fully operational since December 1999

Aeronautical Development Agency (ADA),Computational Fluid Dynamics calculations related to Light Combat Aircraft (LCA) LCA design computed on 32 node ANUPAM-860 in 1994 , ANUPAM-PIII 16 node system installed in June, 2000 at ADA. Extended to 38 nodes in Jan 2002

VSSC, Trivandrum 8 node Alpha and 16 node Pentium for Aero Space related CFD applications

ANUPAM Applications outside DAE

Typical output produced at NCMRWF

Processing ( parallelization of computation)

I/O ( parallel file system)

Visualization (parallelized graphic pipeline/Tile Display Unit)

Complete solution to scientific problemsby exploiting parallelism for

Visualization Rendering Pipeline

Geometry Database

Geometry Transformation

Rasterization

Image

Per Vertex

Per Pixel

Transformation,clipping,Lighting, etc

Scan-conversion,shading and visibility

Parallel Visualization Taxonomy

Snapshots of Tiled Image Viewer

We now have ,

Large tiled display

Rendering power with distributed rendering

Scalable to many many pixels and polygons

An attractive alternative to high end graphics system

Deep and rich scientific visualization system

Post-tsunami: Nagappattinam, India (Lat: 10.7906 N Lon: 79.8428 E)This one-meter resolution image was taken by Space Imaging's IKONOS satellite on Dec. 29, 2004 —just three days after the devastating tsunami hit. 1M IKONOS Image Acquired: 29 December 2004 Credit "Space Imaging"

So, This is how things have been transformed !!

HPC - THEN

Vector and MultiprocessorCray Supercomputer

SGI Origin 2000 – 3000Multiple Graphics HW

Number Crunchers Graphics Engines

HPC - NOW

Cluster based Parallel-processor

Number Crunchers

Cluster based tiled displays

All are Commodity Processors

Anupam - Parallel Supercomputing Facility

So, We Need Lots Of Resources Like High PerformanceComputers, Visualization Tools, Data Collection Tools, Sophisticated

Laboratory Equipments Etc.

“ Science Has Become Mega Science”

“Laboratory Has To Be A Collaboratory”

Key Concept is “Sharing By Respecting Administrative Policies”

User Access Point

Resource Broker

Grid Resources

Result

GRID CONCEPT

LHC is a very large scientific instrument…Lake Geneva

Large Hadron Collider27 km circumference

CMS

ATLAS

LHCb

ALICE

A collision at LHC

29

Bunches, each containing 100 billion protons, cross 40 million times a second in the centre of each experiment

1 billion proton-proton interactions per second in ATLAS & CMS !

Large Numbers of collisions per event

~ 1000 tracks stream into the detector every 25 nsa large number of channels (~ 100 M ch) ⇒ ~ 1 MB/25ns i.e. 40 TB/s !

Tier 0 at CERN: Acquisition, First pass processingStorage & Distribution

Ian.Bird@cern.ch 30

LEMONarchitecture

QUATTOR

• Quattor is a tool suite providing automated installation,

configuration and management of clusters and farms

• Highly suitable to install, configure and manage Grid

computing clusters correctly and automatically

• At CERN, currently used to auto manage nodes >2000 with

heterogeneous hardware and software applications

• Centrally configurable & reproducible installations, run time

management for functional & security updates to maximize

availability

QUATTOR

Node Configuration Manager NCM

CompA CompB CompCServiceA ServiceB

RPMs / PKGsSW Package Manager

SPMA

Managed Nodes

SW server(s)

HTTP

SWRepository RPMs

Install server

HTTP / PXE System installer

Install Manager

base OS

XML configuration profiles

HTTP

CDB

SQL backend

SQL

CLI

XML backend

SOAPGUI

SCRIPTS

ServiceC

DAEGrid Sites and Computational Resources

Legend:BARC, Mumbai

RRCAT, Indore

VECC, KolkataIGCAR, Kalpakkam

6 leased lines 4 Mbps, mesh network

4 Clusters

1. Aruna -- dual processor 64 node xeon

2. Surya -- dual processor 32 node xeon (EM64T)

3. Aksha – dual processor 10 node Itanium-2

4. Arya -- 3 node dual processor xeon (EM64T)

Dual processor 8 node xeon (EM64T)

Dual processor 8 node xeon (EM64T)

Dual processor 8 node xeon (EM64T)

Total : 4 Sites

7 Clusters

266 Processors

Provides to scientists of all participating institutes, a unified and seamless access to any resource independent of its location, leading in increase of computational power, fair utilization and availability of resources.

Current ActivitiesGraphics and Visualization Section

• CollabCAD – java/java3D based distributed 2D&3DCAD/CAM and CAE software system being developed incollaboration with Computer Aided Design Group ofNational Informatics Centre, New Delhi and VikramSarabhaii Space Centre, Trivandrum.

• Development of multi-block finite volume CFD solver(Anupravaha) for non-orthogonal hexahedral structuredgrids in collaboration with IIT-Kanpur.

• Development of Visualization Framework (AnuVi) tovisualize and extract data from solver output.

Current ActivitiesGraphics and Visualization Section

Simulation of reactor using AnuPravaha and streamline display on 6X6 Tiled Display using AnuVi to study the velocity of particle under pressure

Current ActivitiesGraphics and Visualization Section

Picture shows distributed nature of CollabCAD on Tiled Display

Authentication, the process of proving that someone is who he claims to be.

Two factor authentication provides an increased level of Security.

Two factor security combines :-• What you know?• What you have?

Password is used for “Something you know” factor. Cell-phone is used for “Something you have” factor.

BARC External Webmail Service(Using Two factor authentication)

Login steps:-• User is prompted to enter his

Username and Password.• Server sends a random 6-digit

alphanumeric access-code to user’s registered cell no.

• User enters access-code and completes the authentication process.

• Access-code is for one-time use. A new access-code is generated for every successful login.

Contd…

BARC External Webmail Service(Using Two factor authentication)

INTERNET

USER at Home

USER on Deputation

BARC Firewall

SMSGateway

WebmailServer

E-MailServer

USER in Office

Password

Password + Access CodePassword +Access Code

First Factor (Password)Second Factor (Access Code)BARC - Webmail

Architecture

Contd…

BARC External Webmail Service(Using Two factor authentication)

Secure Applications• Personnel Tracking and Time Attendance System• Public Key Infrastructure in BARC• Web Application for Digital Signing using above

PKI.• Telephone (PSTN) /GSM Line Encryptor.• Secure Network Adapter.• Hand-held RFID reader.• Applications on mobile phones :

– Secure SMS– Secure MMS– File Encryptor

RFID CARD READERS

IN

OUT

IMAGE DISPLAY

PIMS DATABASE

TAS DATABASEWITH STANDBY

IMAGE DISTRIBUTION

ID CARD SYSTEM(EMPLOYEES ANDNON BARC PERSONS)

WEB SERVERSWIPE COLLECTION

TAS USERS

READER CONFIGURATION

CEP SYSTEM PIMS USERS

1

2

3

4

5

6

7

8

9

10

TAS At BARC

http://visemp.barc.gov.in

Public Key Infrastructure in BARC• BARC has become the Registration Authority under the

Certification Authority of National Informatics Centre, New Delhi.• 55 employees have been issued digital certificates.

• PKI DetailsCertifying Authority : NIC CAAlgorithm : RSA with SHA1Key length : 1024 bitsCertificate compliant : X.509

CCA

Certificate Chain path

NIC - CA

BARC

Secure Network Adapter Developed using open source IPSEC tools Transport as well as Tunnel Mode of Operation IKE based Key Communication protocol.Authentication can be based on preshared secrets or X.509 Digital Certificate. Implemented on AVR32 Arm Processor.AVR32 Linux as a operating System. Throughput up to 2.4 Mbps.

Applications on mobile phones (These systems are developed using Java 2 Micro Edition.)

• Secure SMS– This application is being used by the authorities in BARC

for last 3 years. – It is meant for closed user group.– Encrypted SMS are received on a different port.

• Secure File Transfer via MMS– facilitates users to send and receive large files through

MMS in an encrypted manner. – Through MMS large files can be sent and received via

GPRS among the mobile phones, so it eliminates the limitation of 160 characters in SMS.

Very Important but not so Pictorially presentable projects !!

- Online Confidential Records (OCR)

- Parinay (Project Management Systems)

- Huge communication infrastructure

- Information Security measures

- Migrating Grid Infrastructure to NKN

- BTS has become indispensable !

……

Thank you