etf feature list

8/3/2019 ETF Feature List

1/35

Management Dynamics, Inc.One Meadowlands PlazaEast Rutherford, NJ 07073

Tel 201.935.8588 8

ETF Feature List

MANAGEMENT DYNAMICS CONFIDENTIAL AND PROPRIETARY INFORMATION


2/35

TABLE OF CONTENTS

Introduction .......................................................................................................................... 4ETF Application ...................................................................................................................5

Application Definition ..................................................................................................... 5

Roles ................................................................................................................................ 5Company Specific Roles ..............................................................................................5

Components ..................................................................................................................... 5

Company .......................................................................................................................... 5Service Providers ............................................................................................................. 6

Organizational Hierarchy .................................................................................................6

Internationalization .......................................................................................................... 7Security ................................................................................................................................ 8

Functional Role Security ..................................................................................................8

Company Specific Functional Role Security ............................................................... 8

Widget Level Security ..................................................................................................... 8

License Management ....................................................................................................... 9Entity ..................................................................................................................................10

Object Relational Mapping (ORM) ...............................................................................10Entity Persistence Delegation ........................................................................................10

Row Key ........................................................................................................................10

Entity Triggers ............................................................................................................... 11Entity Auditing ...............................................................................................................11

Application Databases ....................................................................................................... 12

Connection Pools & Connections .................................................................................. 12

Database Schema ...........................................................................................................12Database Transactions ................................................................................................... 12

JDBC Utility ..................................................................................................................13Application Configuration ................................................................................................. 14

Configuration Files ........................................................................................................14

Named Configurations ...................................................................................................14

Property Sheet Configurations ....................................................................................... 14Data Groups ................................................................................................................... 15

Application Extension & Customization ...........................................................................16

Data Extensions ............................................................................................................. 16Extension Compiler/Loader ...........................................................................................16

ETF Runtime ......................................................................................................................18

Services ..........................................................................................................................18

Singleton .................................................................................................................... 18Replicated .................................................................................................................. 18

Events .............................................................................................................................19

High Availability/Failover .............................................................................................19Work Management .............................................................................................................20

Consumer-Producer .......................................................................................................20

Typed Consumer-Producer ............................................................................................ 20Consumer-Producer Bootstrap .......................................................................................20



3/35

Work Persistence ........................................................................................................... 20

Integration Framework ...................................................................................................21

Core Application Services & API ......................................................................................22Caches ............................................................................................................................22

Application Locking Service .........................................................................................22

Application Sequence Service ....................................................................................... 22Application Logging ...................................................................................................... 23

Scheduler ........................................................................................................................23

Mail & HTTP Messages ................................................................................................23Document Repository & Attachments ........................................................................... 23

Business Logic ...................................................................................................................25

Core Logic ..................................................................................................................... 25

Business Process Logic ..................................................................................................25Messages ....................................................................................................................25

Status Rollup .............................................................................................................. 26

Business Process Orchestration Definition (This Capability is Still a Work in

Progress) ........................................................................................................................26Node ........................................................................................................................... 27

Node Table Context ................................................................................................... 27Connector ................................................................................................................... 27

Entity Split & Consolidation ......................................................................................27

Business Process Flow ............................................................................................... 28

Work Flow Features ...................................................................................................28Components .......................................................................................................................29

Window Component ...................................................................................................... 29

Compliance Document Component ...............................................................................29Excel Workbook/PDF Component ................................................................................ 30

Search Component .........................................................................................................30

Work Queue Component ...............................................................................................31Work Queue Template ...............................................................................................31

Work Queue Work-Item ............................................................................................31

Menu Component ...........................................................................................................32Validation Component ...................................................................................................32

Transformation Component ...........................................................................................32

Data Pushes & Data Pulls .......................................................................................... 33

Dashboard Component ...................................................................................................33Custom Entity Component .............................................................................................34

Custom Entity Designer ............................................................................................. 34

Enhanced Custom Entity Designer ............................................................................ 35



4/35

IntroductionThe ETF platform is an enterprise application development framework. It is based on a

rapid application development premise that incorporates self-contained, meta-datacomponents that can be developed or extended, tested, and promoted at run-time or post-

deployment.

The ETF feature list document is meant as a quick reference that provides projectmanagers and developers a consolidated list of features available within the ETF

platform. The intent of this document is to communicate the features at a very high level.

The concerned individual should engage colleagues or reference other material for adeeper understanding of how to take advantage of the feature within his/her application.



5/35

ETF ApplicationAn ETF application consists of a packaging of core logic, business logic, configurations,

and components of various types. These artifacts are delivered under an applicationumbrella in a way that allows applications (or modules) to be deployed separately or

easily combined as a single deployment.

Application Definition

An application definition serves as a container for roles, licenses, security, components

within the component repository, and other application resources. Multiple applicationscan be combined in a single deployment and a company account can be given access to

multiple applications within the deployment. In that sense an application is almost better

thought of as a module rather than an application.

Roles

Roles define the different roles a user can play within an application. A role largelydetermines what resource the user has access to and the operations that he/she can

perform on those resources while within that role. A user may have access to multiple

roles, possibly spanning multiple applications, but can only play a single role at any point

in time.

Company Specific Roles

A pre-defined set of roles are delivered as part of the application but the company has the

ability to created its own set of roles that have a specific meaning within the company.

Components

Components are named, self contained, meta-data definitions of application artifacts thatare organized within the component repository. Examples of components includewindows, compliance documents, reports, transformations, searches, work-queues, etc.

The component resolution rule always looks for a component under the company folderand defaults to looking under the System folder only if it cannot locate the target

component under the company.

Because components are fully self contained, they can easily be exported and imported

holistically.

CompanyA company within an ETF application represents an organization that has been given

access to the application. The company is usually created by an administrator that grantsaccess to specific modules (applications) and the licenses, if any, within the module that

the company has purchased



6/35

By default a client-administrator user account is created for the company. The client-

administrator has the ability to setup the organization hierarchy and additional user

accounts.

There are different types of companies: client companies (normal case), service-provider

companies, and managed companies. A service-provider is described below. A managedcompany is a company that is being managed by another company. Typically this arises

when an application is being re-sold and the manager company is managing one or more

companies that were re-sold. The manager company has the ability to manage users onthe managed companys behalf.

Service Providers

A Service Provider (SP) is a company that provides one or more services to another

company. A good example can be seen in the restricted part screening (RPS) application.

In this application, a client company represents the company whose parties are beingscreened against a set of restricted entities. When one of the clients parties potentially

matches one or more entities, a person must review the potential match to determinewhether the match is true or false. The company may designate users of its own to

perform this manual review or may want to allow a 3rd party service provider(s) toperform this service on their behalf. The service provider model described here would

facilitate such a setup. Another good example of service providers is the set of services

provided by various 3rd party logistics providers (e.g. consolidators, forwarders, etc.). Itis important to point out that a service provider itself may have service providers of its

own.

A company is not explicitly designated as a service provider but users of a company may

serve a service provider role. A user is operating as a service provider when her current

role is set to an application role that has been designated as a specific type of serviceprovider. While in that role, the user operates on his clients data if the following twoconditions hold true.

1. The client company has designated the users company as a service provider forthat particular service type.

2. One or more transactions have been assigned to the users company.

Organizational Hierarchy

Each company defines its own organizational hierarchy that usually governs things such

as data visibility, configuration inheritance or refinement, and access privileges to

application resources. The company is the root of the organizational hierarchy for eachcompany.

Data visibility and mutability is defined by the structure of the organizational hierarchy.

While the application is free to choose the visibility and mutability rules that make sense,

most ETF applications will allow users at a parent org to see all data below but onlyallow mutability of data that belongs to the users current org.



7/35

Internationalization

To do



8/35

SecurityAn ETF application provides mechanisms to protect both application resources and

application data. Security is defined at the application level (System level) but can berefined at the company level.

Data security is defined by an applications data visibility and mutability rules within the

organizational hierarchy.

Functional Role Security

Functional role security provides a coarse access security that makes it easy for an

administrator to administer security easily and intuitively.

Resources and operations are organized into a functional hierarchy of arbitrarycomplexity. The roles of an application are either restricted from accessing a functional

area or granted read or write access. If access privilege at any level within the functional

hierarchy is not defined, it is inherited from its closest ancestor where it has been defined.

Because of this bundling of multiple resource and operations into a particular functional

group, functional role security has the disadvantage in that all resources/operations withinthat group have to be restricted or granted in their entirety. In other words, the functional

group is the most granular level that an administrator can operate at.

Company Specific Functional Role Security

The restriction or granting of access to functional areas by role is delivered by an

application but can be refined at a company level. Any new roles created by a companyhave to obviously be mapped to the functional hierarchy. The application can restrict

portions of the functional hierarchy so that access to the functional areas within that

portion cannot be granted.

Widget Level Security

Widget level security is a more granular security framework that allows fine grainedcontrol where its most likely required at the widget level within the applications user-

interface pages.

A UI component is a hierarchy that is rooted by the window. The widget level security

allows an administrator to control access (no-access, read-only, write-access) at any level

within the hierarchy which is then inherited by widgets below that hierarchy. Forexample, restricting access (no-access) to a tab within a tab-folder would eliminate the

tab altogether from the rendered UI.

Widget level security provides full control but makes it difficult for an administrator tomanage security because of the granularity that it allows.



9/35

Widget level security can be delivered at the System level with the application and

refined at the company level.

License Management

An application can decide to organize its various functions into a collection of licenses.

A newly created company is granted access to one or more licenses of an application.The licenses that a company is given access to can be thought of as the functionality

within the application that a company has purchased. The functions contained within the

licenses that a company is not granted can never be accessed.

License management is utilized by back-end logic to ensure that a request, possibly

originating via integration or an engine API call, is licensed prior to fulfilling therequest.



10/35

EntityAn entity in ETF is the enabler for using many of the other features within the ETF

platform. An entity may be defined directly by the database (by reading the DDL thatdescribes tables and relationships) or custom defined via XML. A custom entity can be

painted using either the custom entity designer or the enhanced custom entity designer.

An entitys schema defines the fields of the records that comprise an entity and how thoserecords are related. The entity schema is available at runtime to generically traverse and

manipulate an arbitrary entity.

Object Relational Mapping (ORM)

The ORM framework within ETF is often referred to as Row Data Container or RDC.

The framework is a generic object-relational mapping that loads data into the middle tierby instantiating and populating an entity container. An entity is a collection of data that

is described by its schema or entity definition. The entity can be fully accessed,

traversed, and manipulated using a set of defined operations on thecom.mdi.core.db.RowDataContainerclass. It is different from other ORM frameworks

such as Hibernate in that it does not map relational data to concrete Java classes.

The ETF ORM framework is heavily leveraged and a detailed understanding of the API

is very helpful.

Entity Persistence Delegation

As mentioned above, the population and persistence of an entity may be delegated to a

specific class. This can be specified for all entities, including those that are definedentirely by the database. Though database defined entities are capable of self population

and persistence, the application may need to customize this process.

An entity persistence delegate can be specified directly within a custom entity definition.

Alternatively, an entity delegate can be registered in the configuration file. The latter

allows a delegate to be registered for non-custom entities as well as custom entities.

It is important to note that an entity may not be loaded or persisted to the database or file-

system at all. For example, an entity may be defined to collect some required

information from a user. This information may never be persisted but rather used toinvoke some middle-tier logic with the corresponding results being populated either

within the same entity or a separated entity so that the user can view the results.

An entitys persistence delegate must implement the

com.mdi.core.db.schema.TablePersistence interface class.

Row Key

A row key identifies a specific entity and includes the schema name, root table name, and

one pair of column name and column value for each key column of the root table. A row



11/35

key needs to be specified when loading an existing entity or performing a save-as. A row

key uniquely identifies an entity in the system and represents exactly one row of data

from the root table within the specified schema. See the com.mdi.core.db.RowKey class.

Entity Triggers

An entity trigger works much like a normal database trigger. It allows the registration ofa trigger class whose appropriate methods are invoked as the entity and the records within

the entity are loaded and saved. An entity can have multiple registered triggers which are

chained together and invoked serially. Reference the com.mdi.core.db.Trigger interfaceclass.

Entity Auditing

Entity auditing is a generic framework that supports entity level auditing based on a

configuration. An audit is created by default whenever an entity is persisted. It is

possible, however, for an audit to be created on demand.

If an entity is audited in its entirety, the auditing framework is able to re-create what anentity looked like at a particular point in time by applying the audit records in reverseorder.



12/35

Application DatabasesAn ETF application is usually configured to access one or more relational databases.

Whether you use the ORM framework or access the database directly using JDBC, it isimportant to understand how an ETF application exposes connection pools, schema

information, and the proper usage of database transactions.

Connection Pools & Connections

Connection pools are configured in a configuration file (usually they wrapper an

application server defined data source) and allow users to obtain a connection from aparticular pool by referencing it by its configured name. The connection can be obtained

either explicitly or thread-wise. A thread-wise connection is associated to the current

thread of execution and allows the thread to operate on a single connection without

requiring the reference to be passed from method to method.

The connection must be explicitly returned back to the connection pool when the user has

finished using it. A reference to a connection pool can be obtained using thecom.mdi.core.db.connection.ConnectionPoolRegistry class. A connection pools in ETF

implements the com.mdi.core.db.connection.ConnectionPool interface class.

Database Schema

A schema is available at runtime for introspection. It can be used to obtain an entitys

column information along with its parent and child relationships.

You can obtain a reference to a schema by passing in the schema name to the

com.mdi.core.db.schema.SchemaRegistry class. From the schema, you can obtain anentity or table definition by passing in the name of the entity or table.

A schema is associated with exactly one connection pool. However, a connection poolcan have multiple schema definitions. An entity residing in the same database may differ

in its definition based on different schemas.

Database Transactions

A database transaction must be explicitly started and either committed or rolled-back

upon completion of the atomic operation. Returning a connection back to theconnection pool with an outstanding transaction will result in an error.

Starting a transaction while another transaction is already outstanding is referred to asnested transactions. Nested transactions are handled at the application level and simplyincrement the transaction nesting count. As transactions are committed, the nesting level

is decremented. Only when the nesting level reaches 0, is the transaction actually

committed to the database.



13/35

A transaction that is rolled-back at any transaction nesting level cannot be subsequently

committed. An attempt to do so will result in an exception. To prevent this, an exception

should always be thrown or re-raised after a roll-back is performed.

Use the com.mdi.core.db.transaction.DBTransaction class to start, commit, and rollback

a transaction.

JDBC Utility

The JDBC Utility class provides a lot of helpful utility functions to make interactionswith the database via JDBC simple while reducing/eliminating common errors.

Reference the com.mdi.core.db.JDBCUtility class.



14/35

Application ConfigurationAn ETF application can be configured in various ways. The configuration mechanism

employed depends on things such as the complexity of the configuration, how ofter theconfiguration is likely to be modified, whether or not the configuration is System wide or

can be modified in a company specific manner, etc.

Configuration Files

An ETF application is configured using a set of configuration files that by convention

reside in a sub-folder called config that exists directly beneath the application root folder.

The configuration supports scalars, arrays, collection or namespace, and an array of

collections/namespaces. Use the com.mdi.core.config.ProcessConfiguration class to

obtain configuration values.

Named Configurations

A named configuration is a collection of configuration settings that collectively configure

a certain portion of the application. The configuration settings of a named configuration

can be accessed easily in the middle-tier and is cached for fast access. Named

configurations are typically defined by the application which also delivers the defaultsettings for these configurations at the System level. A company or an organization

within a company may choose to inherit a specific configuration or manage its own

configuration.

Any change to a configuration impacts the company or organization where the change is

made and all descendant company or organizations that inherit from this configuration.A company or an organizational unit within the company may choose to inherit certain

named configurations while managing others.

The UI for managing a named configuration has to be custom developed. Typically the

configuration UI is backed by a configuration entity that is serialized for persistence. See

the class com.mdi.core.app.config.AppConfigManager.

Property Sheet Configurations

A property sheet configuration is a simple type of named configuration. The advantageof using a property sheet is the following

1) No need to define a custom entity

2) No need to paint a configuration UI3) The back-end has a simple API for accessing a property sheet and the propertyvalues within the property sheet.

The property sheet setup UI can be used to design the configuration options that the useris presented with when they go to the property sheet. Use property sheets when the

configuration is simple or flat. See the com.mdi.core.app.config.SimplePropertySheet

class.



15/35

Data Groups

Data groups are a type of configuration which, like the named configurations, can be

inherited or managed at a company or organizational level. A data group is a namedcollection of values that is organized into group types. By convention if there is no type,

the data group is organized into a single type called DEFAULT. Each group type either

explicitly specifies a set of values (actual value and display value pairs) or specifies aSQL select statement that is executed to dynamically determine the values.

Data groups can be used as a data-source for certain types of widgets (drop-down, fill-in-

field, check-box group, radio group, multi-select drop-down, etc.) within the windowpainter, search painter, and work-queue painter to auto-populate the widget with the

values for a specific type from the configured data group.

It is highly recommended that application make heavy use of data-groups for the

following reasons.

1) If the same options are used in multiple places, simply adding, modifying, or

deleting options from the data-group is automatically reflected in each UI, search,and work-queue that the data-group is referenced in.

2) A developer, implementation consultant, or administrator can easily modify the

data group options, provided the application developer has designated that aparticular data-group is mutable.

3) A company or organizational unit within a company can modify the data-group

for their specific needs, just like a named configuration.

The application developer can mark a data-group as either mutable or non-mutable. Non-

mutable data-groups can only be inherited and not managed. They can only be modifiedat the System level.



16/35

Application Extension & CustomizationAn ETF application can be customized by creating new components or customizing

application delivered components, extending the data model (various business entities) tohold customer specific data elements, and by writing java code at run-time.

Data Extensions

Data extensions are an incredibly powerful feature of ETF. It allows run-time extension

of the core schema. The extensions can be of arbitrary complexity including one-to-

many descendant relationships to any depth.

A data extension extends an entity and is either a static or contextual extension. A static

extension is a straight-forward extension of the underlying entity where the extension

simply adds additional fields and tables to the concrete entity definition. A contextualextension extends an entity conditionally based on the value of a designated field. It is

easiest to think of a contextual extension as a collection of extensions of which only one

is valid at any given time (again based on the value of the field).

A data extension not only defines the additional fields and tables that extend an entity but

also define multiple renderings of how these additional fields and tables are to berendered on the UI. Within the UI design of the concreted entity, the designer designates

an area (using a data-extension panel widget) where the extension layout for a given table

is to be rendered. Alternatively, a designer can paint a UI directly against the extended

entity. The later is more applicable for static extensions.

Data extensions are fully supported by the ETF framework and are directly supported by

the ETF ORM framework, window painter, and search painter. All other painters and

engines are capable of operating against a concrete entity or extended entity.

The ORM framework is able to access fields and tables of an extension in the samemanner that the concrete fields and tables are accessed. The ORM framework also takes

care of loading and persisting extension to the database. Data extensions are persisted in

a relational manner instead of in a CLOB and therefore data residing within the extendedentity is searchable via SQL.

Extension Compiler/Loader

The extension compiler and extension loader is a key part of the ETF run-time

environment. The extension compiler provides a run-time mechanism to compile

component code-hooks and extension java classes against the application jar files and anyother extension classes and jar files that exist within the extension repository. The

extension loader is responsible for loading component logic classes as well as classes

residing within the extension repository.

The extension compiler can be accessed using the

com.mdi.core.runtime.extensions.compiler.JavaCompiler class while the extension

loader can be accessed using the



17/35

com.mdi.core.runtime.extensions.loader.ExtensionLoader class. It is important to load

a class usingExtensionLoader.m_Instance.loadClass(myClass) instead of using the

normal Class.forName(myClass). The latter is not capable of loading classes residingin the extension repository.



18/35

ETF RuntimeThe ETF run-time is comprised of three core services: Process Registration Service,

Name Service, and Event Service. It is unlikely that an application developer wouldinteract directly with the process registration service. It is, however, almost guaranteed

that an application developer will need to interact with the name service and eventservice, though not directly.

The ETF run-time is a distributed component architecture that is built on top of Java RMI

and depends on it for marshalling/de-marshalling and for transport (or delivery) of

messages to and from the target service.

Services

An ETF service is an exposed service interface that can be accessed remotely. Theconsumer of the service need not be aware of whether the service resides locally or

remotely. The consumer also does not need to know whether the service is a singleton or

replicated.

Services are an important part of the ETF component architecture and provide load-

balancing and fail-over capabilities in an application architecture that needs to distributeload amongst multiple processes, possibly residing on multiple physical machines. It

provides load-balancing of back-end work much as an application cluster provides load-

balancing for user-initiated actions.

A service can be obtained using the com.mdi.core.runtime.remoting.NameServiceLocalclass.

SingletonA singleton service is one where only one instance of the named service is registered inthe distributed application. All operations on the service are handled by the singleton

instance. A singleton is a useful concept for services requiring control management or

synchronization.

Replicated

A replicated service can have multiple instances that are distributed across multipleprocesses, possibly across multiple physical machines. A consumer consuming a

replicated service is automatically routed to different instances (or replicates) of the

replicated service based on a round-robin scheme. A replicated service that is started orstopped at run-time is automatically detected by the ETF runtime.

Replicated services, by their replicated nature and the ETF runtimes ability toimmediately detect when a service is started or stopped, inherently provide load

balancing and fail-over capabilities.



19/35

Events

The ETF event management framework provides a publish-subscribe event capability

that facilitates System level synchronization. Interested classes register for events thatthey need to pay-attention to. Other classes in the application post or publish events to

notify the subscribers that an event has occurred. The posted event is sent to a central

event manager service, possibly remote. The central event manager determines all thesubscribers that have registered for this event and invokes the event-handler method on

the registered class, also possibly residing remotely. The posting of an event can be

accompanied by arbitrary event-data. The only requirement is that the event data that is

posted must implement thejava.io.Serializable interface.

Reference the com.mdi.core.runtime.remoting.EventServiceLocal class to register for an

event or to post an event. A handler class for one or more events must implement the

com.mdi.core.runtime.remoting.EventHandler interface class.

High Availability/Failover

An ETF application can be run in regular mode or in a highly available or failovermode. Replicated services by default are highly available and support failover as the

ETF runtime is able to detect when a service fails or when a new one becomes available.The issue is with the singleton services. When a singleton service fails, there is no other

replicate that the runtime can failover to. Therefore the singleton services represent a

single point of failure.

The solution to overcome the single point of failure is to run all of the singleton services

in a process called the Admin Server. Only a single Admin Server process is live orservicing requests at any given point in time. If the live Admin Server fails, a backup

will take its place. A Highly Available deployment would start multiple Admin Server

processes but only one will be designated as the live process with the others serving ashot back-ups. When the live Admin Server goes down, one of the hot back-ups willbecome the new live Admin Server process and all other ETF application processes will

connect and re-register services and events with the new Admin Server.



20/35

Work ManagementWork management refers to the process of managing back-end work in an organized

manner that maximizes throughput without over-loading the server. A synchronousrequest where response time, and not overall throughput, is the primary concern usually

does not fall into this category. Typically work-management is reserved forasynchronous requests, scheduled processes, or application integration.

Consumer-Producer

ETF provides a distributed consumer-producer framework where a central producer cansupply work to one or more consumers residing in remote processes. The total number of

configured consumers determines the maximum number of requests of a particular type

that can be handled in parallel. Additional request are queued up until a consumer frees

up.

A producer must implement the com.mdi.core.workmgmt.Producerinterface while a

consumer must implement the com.mdi.core.workmgmt.Consumer interface class.

Typed Consumer-Producer

A typed consumer-producer is one in which the producer produces work of various types.Consumers working with a typed producer are divided into work-sets with consumers of

a particular work-set capable of handling a sub-set of the work types that a producer can

produce. A typed producer is a good way of capping the maximum number of parallelthreads while still being able to control which types of work get more processing power

based on anticipated load or relative throughput requirements.

A typed producer must implement the com.mdi.core.workmgmt.TypedProducerinterface

while a typed consumer must implement the TypedProducer.java.TypedConsumerinterface class.

Consumer-Producer Bootstrap

Both normal and typed consumer-producers can be configured via a configuration file asdictated by the com.mdi.core.workmgmt.ConsumerProducerStartup class. Statistics on

consumer producer is visible through an administrative user interface and allows an

administrator to ascertain how effective the configured consumers are in handlingapplication load.

Work PersistenceThe work that is handed out by a producer may need to be persisted so that the

application does not lose the request if the application goes down prior to the work being

processed.



21/35

Integration Framework

The integration framework provides a persisted implementation of a typed consumer-

producer along with the capability of receiving work by scanning a file system orreceiving via HTTP.



22/35

Core Application Services & APIThis sections describes the core services like caching, locking, logging, etc. that most

applications will need to leverage.

CachesCaching is an essential part of an application to minimize database access and speed upthe application as it accesses data and configuration settings. ETF provides distributed

caching support that can be viewed administratively to see the effectiveness of the

caching scheme.

Currently there are three caching schemes that are supported and described below

1) LRU Cache A fixed sized cache that caches data in memory based on a least-

recently-used algorithm. Data that is moved out of the cache is discarded. Seethe com.mdi.core.utils.cache.LRUCache class.

2) LRU Persisted Cache Same as a LRU cache but data that is moved out of the

cache is persisted to disk and restored when needed. An element is only removedfrom cache or disk when it is explicitly removed from the cache. See the

com.mdi.core.utils.cache.LRUPersistedCache class.

3) Full Cache A full cache caches all data in memory but employs a lazy loadingmechanism, meaning that an element is only loaded when it is first requested. See

the com.mdi.core.utils.cache.FullCache class.

4) Pass through Cache A pass through cache implements the cache interface but

does not actually cache any data. This is a useful implementation for a class thatimplements the com.mdi.core.utils.cache.Cache interface but does not currently

require any data caching.

Distributed caches (the same cache residing in multiple processes) are kept in sync viadistributed events.

Application Locking Service

The application locking service provides an application level pessimistic locking

mechanism to ensure data integrity. An application should ensure that a lock is obtainedon an entity prior to modifying the entity. The ETF UI framework and the business

process engine automatically obtain a lock and free it when no longer required.

See the com.mdi.applock. ApplicationLockServiceInterface interface class. It is

important to obtain a reference to the application lock service using the

NameServiceLocalclass just in-case the locking service resides remotely.

Application Sequence Service

The sequence service is a simple service that provides a unique sequence number basedon a named sequence. It is guaranteed that a subsequent request for value from a

specifically named sequence will give you an incrementally higher values but a

sequential number is not guaranteed.



23/35

The sequence service, like the lock service, is a singleton service and may reside either

locally or remotely. However, you should use the com.mdi.seqmgmt.Sequence class toobtain a sequence value instead of directly accessing the sequence service.

Application LoggingETF presently wrappers the Log4j package and provides an internal API for both debug

and performance logging. It is important to log information using the logging framework

instead of printing data to standard out or standard error so that the logging can becontrolled by enabling a specific log level. It is also important to log enough information

but not too much information.

There are lots of places within the ETF framework that serve as good examples of how to

use the logging API for both debug and performance logging. You should also reference

the Log4j documentation.

SchedulerThe ETF scheduler, like most schedulers, provides a mechanism to schedule jobs basedon an interval or calendar. A job must implement the

com.mdi.scheduler.SchedulerJobInterface or the

com.mdi.scheduler.EnhancedSchedulerJobInterface interface class.

ETF provides an UI to manage job schedules at a company level.

Mail & HTTP Messages

ETF provides a mechanism to send mail as part of a business process but does so in an

asynchronous manner so that it does not impact the performance of the business process.

The mail message can specify the body or use a pre-defined template by passing inparameter values that are referenced within the template. The latter allows for easy

modification of the message without modifying code. Email that fail to be sent (usuallybecause of an issue with the SMTP server or relay server) will be persisted in an error

directory. An administrator can monitor that directory and move any messages found

there to the outbound directory to have the application re-try.

HTTP(S) messages are also supported and work in the same asynchronous manner.

There is some re-try logic after which a message is moved to the error directory. The

HTTP(S) supports basic authentication which is communicated via the URL.

Document Repository & AttachmentsAttachments provide the ability for an ETF application to link information to dataentities. For example, it can be used to attach compliance documents that are generated

within the application to the transaction entity for which the documents were generated.

Attachments can be in the form of internally generated documents, uploaded documentvia a UI or via integration, or in the form of a URL.



24/35

Non-URL attachments live within the ETF document repository and are linked from the

entity to an id returned by the document repository when checking the document into the

repository. With this id, ETF creates a link between the entity and the document andassociates various meta-information with it. The meta-information consists of basic

information such as the id, name, content-type, etc. It can also associate arbitrary

application specific meta-data with the link as well.

The default implementation of the document repository is a file system based repository.

It is, however, possible to plug in a third-party document repository by implementing the

com.mdi.core.docrepos. DocumentRepositoryInterface interface class and registering

the implementation in the configuration file.



25/35

Business LogicBusiness logic forms the heart of what the application was built to solve. A portion of an

applications business logic forms the core functionality that is the essential part of theapplication and is not intended to be modified. Some portion of an applications business

logic represents the default, or generic, implementation but it is feasible that the defaultimplementation would be modified, supplemented, or replaced.

The business logic is delivered as modular and dynamic back-end business rules. By

providing a dynamic mechanism to glue these components together along with user

interaction steps, we can allow customer to build their business process in a dynamic andintuitive manner.

Core Logic

The core logic is delivered in a set of application jar files. It consists of modular steps

that can be accessed in a variety of ways. For example the core logic can be executed

directly via an AJAX handler of a window, during integration, as part of a larger businessprocess, etc. The core logic is most likely configurable and possibly supplemented but is

not intended to be modified. Examples of core logic are things like HS validation,

Restricted Party Screening, License Determination, Document Determination, OGAdetermination, etc.

Again, the core logic is delivered in a modular way with a default wiring or glue. The

idea is that a customer implementation may change how this core logic is wired in alarger business flow that incorporates custom logic along with user interaction to define

an overall business process.

Business Process LogicEnterprise applications often involve complex processing of transactional data. Forexample, export orders and shipments require detailed validation and complex

compliance processing. Often, there are varying requirements for this processing. A

shipment being exported out of the US requires very different processing from a

shipment being exported out of China. In addition, software solutions being implementedfor various customers often have additional individual processing requirements.

The business process defines a set of automated steps that are glued or wired togetherto form a business process. Each of these steps in our nomenclature represents a business

rule. An entity runs through a business process by executing the individual steps (or

rules) in a serial fashion. The rules of the business process may modify the entity datawhich is possibly saved at the end of the process execution.

Messages

The heart of the business process is the handling of the error, warning, and informational

messages that are generated by the business process and the impact those messages have

on the status of the entity.



26/35

Messages that are raised by the various rules are pre-defined and have several important

attributes associated with them.1) Message Id The well-defined id of the message. The message ids are published

and can be used during application development and during implementation to

programmatically act on the outcome of a business process execution.2) Message Text Parameterize message text that can be internationalized. The

message parameters can be in the form of an entity table and column or a named

parameter that must be passed in when raising the message.3) Severity The severity of the message (Error, Warning, or Informational) that has

an impact on the status of the entity.

4) Category Defines the roll-up category that this message belongs to. An entity

has several statuses. More precisely, an entity has an overall status and anindividual status for each category. This is important because an application may

want to differentiate whether a transaction is, for example, on an import hold or

an export hold.

5) Overrideable Defines whether a message can be overridden by a user or not.

Each execution of a business process assigned a process-id with the execution andpersists all messages raised during that business execution. An overridable message that

is re-raised remains overridden unless specific records or field values on the entity

change. These specific records and fields are specified by the business process.

Status Rollup

At the end of a business process execution, the status roll-up calculates the status at eachlevel of the entity by rolling-up the status from the bottom up, separated by category.

A status-rollup configuration defines the propagation, by category, at each level of the

entity by specifying what level the message propagates to. By default messages from aparticular category always propagate all the way to the root record of the entity. The

overall status of the entity is defined to be the most sever status amongst the defined

categories. The overall status is saved with the entity but the individual category statusesare stored with the process execution results.

Business Process Orchestration Definition (This Capability isStill a Work in Progress)

A business process orchestration definition is an amalgamation of a business process and

user interaction. The aim is to define an overall business process that an entity flowsthrough based on a specific customers needs. It will incorporate a visual flow

construction consisting of nodes and connectors that allow an entity to move from onenode to the next based on a condition defined on the connector.

At this time it is not all together flushed out how the business process and business rules

will be utilized within the context of the business process orchestration.



27/35

Node

Each node of a business process orchestration is defined as either an automated step or a

manual step. An automated step is most likely wrappered as a business rule and mayrepresent things such as a validation, transformation, document generation, custom

business logic, etc. A manual step would require a user to manually act on the entity,

most likely via an application defined UI.

A manual step would define the UI that the user would use to view the entity at this

particular step along with an indication of whether it is assigned to a specific user or agroup of users. An assignment to a specific user would utilize the ETF work-item

assignment support while assignment to a group of users would leverage the ETF work-

queue capability.

Each node has several code hooks that allow for more of the glue to build the

appropriate business process. One important code hook is the processEntity code hook

which can be used to define truly custom processing of an entity.

Node Table Context

The node table context defines the table within the entity that a particular node operates

on. For example the flow may be defined to operate on a Shipment transaction node but

it is possible that there are steps within this node that operate on an individual line-itemor individual partner record. The specific context row is passed for processing at this

node for both manual and automated nodes.

Connector

A connector connects a source node to exactly one target node. An entity once processed

at the current node is ready to move to the next step in the flow. All outbound connectors

of the current node are evaluated to determine if the entity moves to the target node ornot. Currently this determination is in the form of a code hook that returns a true or false.

Entity Split & Consolidation

It is possible for an entity to flow from one node to multiple target nodes. This is called a

split and may occur for one of two reasons.1) The condition of two or more connectors is satisfied as it leaves the current node.

2) The context table of the target node is a descendant table of the current nodes

context table. For example flowing from a transaction node to a line-item nodewould cause a split since the entity would enter the target node once for each line-

item.

When an entity splits, it operates in theory in parallel though in reality it is still likely to

be processed serially but the important distinction is that sequence is not guaranteed

amongst the split flows.

A consolidation occurs after a split when all the split flows consolidate at a particular

node. A split flow arriving at a consolidation node does not proceed until the other split

flows arrive as well.



28/35

Business Process Flow

A business process flow will be a special type of business process orchestration that will

consist of only automated nodes each of which executes a business rule. The entire flowwill represent an automate business process. These business processes may be re-used by

other process defintions.

Work Flow Features

The primary features are provided by the work flow node and include the following1) Entity Audit An entity can be audited at a particular node so that is possible to

ascertain the changes that occurred during the processing at that node. The audit

configuration can be specified directly within the flow definition.

2) Alert It is possible to alert a group of users when an entity either arrives orleaves a particular node. The body of the alert can be specified within the flow

definition.

3) Leveraging of work-queue and personal work-item for manual nodes4) Scheduled or trigger based processing



29/35

ComponentsAll component painters are backed by an entity. As stated earlier, an entity may be

defined directly by the database DDL, representative of a business entity as defined in themiddle tier, or created specifically to facilitate component creation.

Component painters provided a drag-and-drop facility to easily construct a large portion

of a component definition but allow the developer flexibility to go beyond by providingcode hooks that are invoked as the component is executed.

Window Component

A window component allows a developer to paint arbitrarily complex UI that is backed

by an entity. The entity defines the data structure that the UI presents to the user for

viewing, editing, or creating.

The window painter allows for 80% of the UI to be created via drag and drop. The

developer can write the javascript event handlers, AJAX handler, javascript logic, andback-end java logic directly within the painter.

The window framework is built on top of a sophisticated binding framework that binds anentitys data to the HTML markup. The framework automatically synchronizes the UI

with both the client side javascript data model and the server side Java data model, and

vice-versa. This allows the developer to easily write validations and business logic either

on the client (javascript) or server (Java) after factoring in security, performance, andmaintenance considerations.

Window components can be used to create any type of UI from a CRUD window to an

administrative UI to a complex development tool UI. When ever there is a need to createa UI, chances are it can be done using the window painter and reduce the development

time to a fraction.

Compliance Document Component

Compliance documents often need to be laid out according to a government providedtemplate where specific data needs to be placed at specific location within the page.

These sorts of documents are easily created using the compliance document painter as it

combines page-layout features with data-iteration capabilities. Code hooks provide away to manipulate, massage, and filter the data as the document is being generated.

The document painter is a sophisticated layout tool that makes the process of creating acompliance document a snap while containerizing the definition, java logic, images, PDF

templates, etc. into a self-contained component.

A document component can be rendered as either HTML, PDF, or PDF form. The latterallows the user to modify values within designated fields that have been marked as form

fields. The modified data can be re-submitted back to the server and processed. The

default processing is to re-generate the document using the modified data.



30/35

Idea: We should provide a code-hook or perhaps a full class editor within the document

painter where the user can write his own document data submission processing logic.Why shouldnt this be part of the document component definition???

Excel Workbook/PDF ComponentAn excel report allows for the creation of a grid report. We dont want to call it a tabular

report generator because the painter is sophisticated enough to allow an arbitrary nesting

of data iterations. Data can be organized on a single sheet or spread across multiplesheets.

Various code hooks at the data iteration level, row level, and cell level allow for fine-grain control during the generation process. Dynamic creation of excel formulas using

cell address macros and formatting features such as font, text color, background color,

borders, cell merging, etc. provide everything you need to create sophisticated excelreports.

An Excel Workbook/PDF component can be rendered as either an Excel workbook or a

PDF document. A PDF rendering of a component ignores any cell formulas and excelspecific formats. The rendering of each sheet is started on a new PDF page. A PDF page

setup allows for the specification of page size, orientation, and margins.

Search Component

A search component typically defines a SQL based search. It allows the developer to

define the following three aspects of the search.1) The list of tables that will be used in the search and how those tables are related to

one another.

a. A table can be a physical table or a dynamic table defined by specifying anarbitrary select statement.

b. The same table can be used multiple times by aliasing the table with

different names.c. A table can be designated as optional in which case it will only be used if

the user filter makes use of one or more columns from that table.

2) A list of column that are displayed in the search results when it is executed.3) A list of columns from the set of tables that could be used by the user to filter the

search results.

4) A list of commands or navigations, each of which operates on a single result row

or on the entire data-set. In the former case, the command may be executed by

associating it with one of the display columns. The display column would be inthe form of hyper-link that when clicked would execute the command.

a. A dataset can define the set of columns that make up an entity row-key. Acommands or navigation can reference this row key as a command

parameter or as part of the URL.

Searches are typically used to find a set of entities or records and the navigations defined

by the dataset allow the user to see the details of the entity corresponding to that row.



31/35

This is typically accomplished by specifying the URL to a window component designed

to display the appropriate entity along with the entity row-key. When look at the details

of a particular row, the ETF framework automatically allows the user to navigate throughall the records in the search results or navigating back to the search results.

A search component can be used for both primary searches and popup searches.Javascript logic and an AJAX handler can be written directly within the search painter.

Idea 1: It would be nice to allow some java based manipulation of each dataset rowbefore it is returned to the front-end. This would allow sophisticated formatting without

requiring an AJAX call to be made for each row.

Idea 2: We should allow a way to easily create a filter component using the existingdataset painters and back-end support. This would allow an application developer to

create a filter for components like the Excel Workbook/PDF component or for a custom

process.

Work Queue Component

A work-queue provides the ability for a group of users to systematically work through abatch of work. It is easiest to think of a work-queue in the way that a telemarketing firm

assigns work to its telephone operators. At the beginning of the shift, the managers

create several pools of telephone numbers that are segregated by things such asgeography. Several operators from the shift are assigned to each batch. The System

makes calls for each batch and when a person answers the phone, the call is switched to

one of the users assigned to that batch. The operators assigned to a particular batch dontknow which specific numbers each member will call, they just know that as a group they

will work through all the numbers. An ETF work-queue is a similar concept.

Work Queue Template

A work-queue is created from a work-queue template. A work-queue template is similar

to a search component except that it only allows the developer to define the set of tablesand how they are related along with the filter. A work-queue template also defines the

entity row-key and the detail page in which a work-queue item (or entity) will be

displayed. A work-queue is given a name and assigned users, user-groups, and/or rolesto define who has access to that queue.

Work Queue Work-Item

The first time a named work-queue is accessed; the server executes the select statement

and loads the entity row-keys into memory. The items of the work-queue are processedin order one-by-one. Each time a user accesses the work-queue, the work-queue managerattempts to obtain a lock on the next work-item and display that work-item within the

designated detail page. If the lock could not be obtained the work-item is skipped and the

work-queue manager attempts to get a lock on the next work-item. When the work-queue manager has processed the last work-item, it re-executes the query to re-load the

work-queue entity keys.



32/35

Work queues are heavily leveraged in ETF applications as it provides a powerful work-

flow oriented way of presenting work that needs attention to the appropriate users.

Several portlets allow for the status of the work-queues that a user has access to bepresented on the users individual dashboard with a drill-down that takes directly into the

work-queue to process the work.

Menu Component

A menu is associated with an application or module. Each menu is an arbitrary hierarchy

of sub-menus with each sub-menu containing zero or more menu items. A menu item isassociated with a URL that navigates the application to the associated resource.

The menu component painter makes it easy to navigate to various applicationcomponents by providing an ability to select the appropriate type of component from a

popup-search. A functional area can be associated with a sub-menu or menu-item which

controls whether the menu is visible or not for a user in a particular role.

Validation ComponentA validation allows for data within an entity to be validated using either a pre-defined setof simple validations or custom validation logic in the form of Java code. Simple

validations include the following

1) Required value2) Data type

3) Regular Expression matching

4) Value Range

5) Value List (SQL or Explicit)

Individual validations within a validation definition are organized into groups where a

group may be associated with an entity table. A group associated with an entity tableiterates over the records for that table with respect to the current record for the closest

ancestral group iterating over a table (or with respect to the root record of the entity).

The actual validations are defined on a specific column. If the column validation iscontained within an iterating group, the validation is executed once for each iteration of

its containing group. Otherwise, the validation executes once for each record for the

table that the column belongs to, again with respect to the current record for the closestancestral group iterating over a table.

Transformation Component

A transformation defines the transformation of data from one entity type to another entity

type. A transformation is executed by passing in a source entity and a target entity,where the target may be either a new or an existing entity.

A source entity may be loaded using either its standard persistence logic or via a

transformation adapter. Likewise the resulting target entity may be persisted using its

standard persistence logic or via a transformation adapter. A transformation adapter isdefined to handle a specific schema in a one of the supported adapter formats. Each

adapter must implement the com.mdi.core.transformation.transformationadapter.



33/35

TransformationAdapterinterface class. The primary responsibility of an adapter is the

following three things

1) Generate an entity definition based on the message schema2) Populate an entity (defined earlier by this adapter) by reading and parsing a

source message.

3) Generate a message from a populated entity (defined earlier by this adapter).

Currently the ETF transformation engine supports three types of adapters: XML, CSV,

and EDI. Other adapter types can be added in the future if required. The XML adapterrequires a XML schema, the CSV adapter requires a template CSV file, and the EDI

adapter requires a SEF file that defines the EDI message.

The transformation painter and engine supports the following key features1) Traversal of the source entity

2) Identification of the target record (or creation of a new record if one cannot be

identified) based on several pre-defined target identification rules or a code hook.

With this target identification scheme, it is possible toa. Map one source record to a single target record

b. Map multiple source records to one target record (Consolidation)c. Map one source record to multiple target records (Split)

3) Mapping of field from a source record to a target record using direct mapping,

macros, or java manipulation.

4) Data pushes & data pulls

Data Pushes & Data Pulls

Data pushes and data pulls allow the developer to push data from a source entity to a

resource table. Data pulls allow the developer to pull onto a target-entity from a resource

table.

Data pushes are an important to update or create resource data as it arrives with various

message types. Data pulls, mean while, allow a source entity to simply pass a reference(without passing in all the information with every message) to existing resource data and

have the transformation pull the data onto the target entity based on the resource

reference.

Dashboard Component

A dashboard is a collection of portlets defined by an application for each role with theuser possibly having the ability to add, delete, or modify the portlets on their own

dashboard copy. When a user first logs into the application and play a particular role thatshe has access to, the dashboard for that user is created by copying the default dashboard

definition for that role.

The default dashboard for each role is delivered by the application but these template

dashboards can be modified by each company.



34/35

The dashboard supports multiple tab pages each of which is organizes a collection of

portlets into a top area and bottom area, with the top area being a single column view and

the bottom section a multi-column view. The user can, provided his role has sufficientprivileges, move the portlets around within the dashboard to re-organize. The portlets of

a dashboard tab page are only rendered when the user clicks on that tab page.

The following dashboard portlet types are supported. This is almost guaranteed to

change as new protlet types are added to the core platform.

1) Dataset Graph Portlet A specific type of graph (pie, bar, column, or line)generated by executing an existing dataset (search) definition. Drill down

capability is supported by allowing the user to click on an area of the graph which

causes navigation to the search results corresponding to the area that was clicked

on.2) Pivot Table Portlet A pivot table view generated by executing an existing

dataset (search) definition. Drill down capability is supported on the column cell

values, column totals, row totals, and grand total. The drill down navigates to the

search results corresponding to the value that was clicked on.3) Detail Dataset Portlet Shows the record details as defined by the dataset

definition for the first N records matching the filter criteria. Clicking on a recordnavigates to the detail view for that record.

4) Dataset Portlet A collection of metrics based on an existing dataset definition.

5) Saved Searches Portlet Shows the count of records that would result if the saved

search were to be executed. Drill down on a particular saved search navigates tothe search results for that dataset after executing using the filter persisted in the

saved search.

6) Work Queue Status Portlet Shows the list of work-queues defined for eachwork-queue template that the current user has access to. Drill down on a

particular work-queue navigates the user to the next item in that work-queue.

Custom Entity Component

Custom entities are non-database defined entities and require a persistence delegate to

handle the loading and saving process. Custom entities can be used for a variety ofreasons but typically they are created to be used in the definition of a new component.

For example, a custom entity can be created to collect information from a user via a

window component and use that information to execute a business process. A customentity is exactly the same as a regular database entity. The only difference is the presence

of a persistence delegate.

Custom entities are defined by XML and can be assembled manually. However, ETFprovides two painters to create custom entities. Each is described below.

Custom Entity Designer

The custom entity designer is capable of creating or editing an arbitrarily complex entity

definition. The entity definition can specify a persistence class that would be responsiblefor the population and persistence of an instance of this entity. The persistence class

must implement the com.mdi.core.db.TablePersistence interface.



35/35

Enhanced Custom Entity Designer

The enhanced custom entity designer has all the capabilities of the custom entity designer

in terms of defining an arbitrarily complex entity definition. However, the enhancedcustom entity designer also provides the ability for the developer to specify how the

entity is loaded and/or persisted. This is accomplished by specifying SQL or writing

snippets of java code within the various code hooks.

etf feature list

Documents