original research open access the simcyp population based

Jamei et al. In Silico Pharmacology 2013, 1:9http://www.in-silico-pharmacology.com/content/1/1/9

ORIGINAL RESEARCH Open Access

The Simcyp Population Based Simulator:Architecture, Implementation, andQuality AssuranceMasoud Jamei1*, Steve Marciniak1, Duncan Edwards1, Kris Wragg1, Kairui Feng1, Adrian Barnett1

and Amin Rostami-Hodjegan1,2

Abstract

Developing a user-friendly platform that can handle a vast number of complex physiologically basedpharmacokinetic and pharmacodynamic (PBPK/PD) models both for conventional small molecules and largerbiologic drugs is a substantial challenge. Over the last decade the Simcyp Population Based Simulator has gainedpopularity in major pharmaceutical companies (70% of top 40 - in term of R&D spending). Under the SimcypConsortium guidance, it has evolved from a simple drug-drug interaction tool to a sophisticated andcomprehensive Model Based Drug Development (MBDD) platform that covers a broad range of applicationsspanning from early drug discovery to late drug development. This article provides an update on the latestarchitectural and implementation developments within the Simulator. Interconnection between peripheral modules,the dynamic model building process and compound and population data handling are all described. The SimcypData Management (SDM) system, which contains the system and drug databases, can help with implementingquality standards by seamless integration and tracking of any changes. This also helps with internal approvalprocedures, validation and auto-testing of the new implemented models and algorithms, an area of high interest toregulatory bodies.

Keywords: ADME, Pharmacokinetics, Pharmacodynamics, Physiologically-based pharmacokinetic, Simcyp,Model based drug development

BackgroundThe relative complexity of differential equations involvedin physiologically based pharmacokinetic and pharmaco-dynamic (PBPK/PD) models, together with the need forcomprehensive drug and system data, has previouslylimited their use to a small group of modelling scientists;however such models have been around since the 1930s.(Teorell 1937). Recently, applications and acceptance ofPBPK models joined with In Vitro – In Vivo Extrapola-tion (IVIVE) of Absorption, Distribution, Metabolismand Excretion (ADME) in drug development and regula-tory assessment have significantly increased (Zhao et al.2011, Rostami-Hodjegan 2012). This trend can be attrib-uted to many factors, including, the availability of user-

* Correspondence: [email protected] Limited (a Certara Company), Blades Enterprise Centre, John Street,Sheffield S2 4SU, UKFull list of author information is available at the end of the article

© 2013 Jamei et al.; licensee Springer. This is anAttribution License (http://creativecommons.orin any medium, provided the original work is p

friendly software and an increase in computing power,which have facilitated the use of PBPK modelling (Rowlandet al. 2011). Bouzom and co-workers reviewed the featuresand limitations of both off-the-shelf and the traditionaluser customisable software packages which are frequentlyused for building and applying PBPK models (Bouzomet al. 2012). The Simcyp Population Based Simulator is acommercially available package used for Model BasedDrug Development (MBDD). An overview of the frame-work and organisation of the Simulator and how it com-bines different categories of information was previouslydescribed (Jamei et al. 2009a). Each year more than 40man-years of effort go into updating and refining theSimulator; hence it is necessary to provide an update onthe latest architectural and implementation developments.Interconnection between peripheral modules, the dynamicmodel building process and compound and populationdata handling are described. Also, the Simcyp Data

Open Access article distributed under the terms of the Creative Commonsg/licenses/by/2.0), which permits unrestricted use, distribution, and reproductionroperly cited.

mailto:[email protected]

http://creativecommons.org/licenses/by/2.0

Jamei et al. In Silico Pharmacology 2013, 1:9 of 14http://www.in-silico-pharmacology.com/content/1/1/9

Management (SDM) system, which contains the systemand drug databases and facilitates implementation of qual-ity assurance procedures, is explained. This article alsoprovides details of the regression autotesting proceduresimplemented to ensure the consistency and integrity ofnew models and their interaction with already imple-mented modules.

Chronology of modulesSince its inception the Simcyp Simulator has been devel-oped in conjunction with a Consortium of major phar-maceutical companies, who share pre-competitiveinformation and provide guidance on the addition of fur-ther capabilities and features. The usage and functionalityof the Simulator is further enhanced by collaborationswith drug regulatory bodies and academic centres of ex-cellence worldwide. Every year a new version of the Simu-lator with new features is released. A rough chronologicalorder of the development of models and databases withinthe Simulator is shown in Figure 1. The Simulator devel-opment started with simple static drug-drug interactioncalculations (“Static CYP DDI” section of Figure 1). Thiswas expanded to dynamic models and the minimal PBPKmodel that was subsequently expanded to full PBPKmodels. The recent developments include handling oftherapeutic proteins, the ability to add custom PD scriptsand to model time-variant physiology in paediatric popu-lation as the subjects grow and in pregnant women overthe duration of pregnancy.Various architectural and procedural modifications

and enhancements introduced to maintain performanceand minimise simulation time are explained in the

MultipleInhibitors

MetaboliteKinetics

MechanisticOral

Absorption

Dermal& Lung

Absorption

RatDog

Mouse

Non

TransPD & Tox

(BBB, Kidneyand Cardiac

tox)

200to

201

Figure 1 The chronology of expansion of the Simulator features fromdevelopment started with static metabolic drug-drug interaction calculatiobody PBPK and so on.

following sections. The general workflow, quality assur-ance procedures and module testing methodologies arealso described.

MethodsEach year a new version of the Simulator is developedand released. This requires a rapid application develop-ment environment that allows many iterations of codewithin a development cycle to present and implementnovel modules in the most expedient manner. This nat-urally dictates a specific workflow and procedures fordealing with model development/implementation anddata generation/maintenance. The development work-flow, data structure and integrity check, dynamic modelbuilding and testing procedures are explained in thissection.

Development workflowAgile and scrumFrom the early days of development in 2002, an ap-proach to software development has been used whichmaps quite closely to methodologies known as Agile(Martin 2011, Larman and Basili 2003). For Simcyp, thisapproach usually involves a group of scientists, who areresponsible for creating a design or implementationdocument for the new feature, along with at least onedeveloper. Each project team has a lead person whoheads the team and coordinates their efforts to defineand deliver the project deliverables within the giventimeframe. Each project also has a supervisor who pro-vides advice and guidance whenever needed and also isresponsible for evaluating scientific and technical aspects

Static CYP DDI

Dynamic CYP DDI

WholeBodyPBPK

-CYP &porters

Fitting & Sensitivity

Therapeuticproteins including

mAb’s

1 3

Time variant physiology & pregnancy

Start

2001–2013 under the Simcyp Consortium guidance. Thens then dynamic drug-drug interaction models followed by whole


of the deliverables and approve the implementation doc-uments before they are passed onto developers forimplementation.Although the implementation document contains the

scientific requirements of the new feature it is not pos-sible to capture the nuances of interfacing with multipleother concurrent projects nor can it capture interfacingwith an ever evolving computational core. As such theproject development is carried out in small sections withiterations to modify and embed the new feature. In orderto accomplish this smoothly a Scrum approach (Simsand Johnson 2012) is taken where regular updates aimto ensure that developers working on different projectsare aware of what other projects and developers are bothdoing and planning to do in the short term. This highlycommunicative approach can remove a large amount ofambiguity and ‘surprise’ during the development cycles.

Microsoft visual studio and team foundation server (TFS)The Simcyp Simulator is currently developed usingMicrosoft Visual Studio 2010. Development is split intotwo principal areas, the addition of new functionalityand the updating of existing features. As a result, mul-tiple developers may be working on similar areas of codethat could in practice lead to significant developmentcollisions. To counteract this and to implement rigidquality control procedures Microsoft Team FoundationServer (TFS) is used as the central code repository butalso as the principal marshalling tool for code develop-ment (Guckenheimer and Loje 2012).Addition of code to the Simulator is strictly controlled.

Firstly, no code can be added to the Simulator withoutbeing linked to a Task (see next section). The Task is aunit of work defined within TFS and relates to either aProject that is under way or a bug fix or an enhance-ment. In this way there is traceability between documen-tation and lines of code. Secondly the code beingchecked in must ‘merge’ with the current code that is inthe repository. It is highly probable that another devel-oper may have been working on the same files at thesame time. This could lead to collisions where one de-veloper modifies or overwrites another’s code changes.TFS prevents this through its intelligent Merge toolsand by calling on the developer to make executive deci-sions where merging becomes ambiguous. As each codesection is added a build is carried out to verify the integ-rity of the added code and ensure the overall code isnever ‘broken’.

Projects and tasksWith the increasing use of Simcyp in regulatory sub-missions, full code control and audit trails need to berigorously imposed. To achieve this aim a policy of nocode development without accompanying approved

documentation has been introduced (see next section).In order to marshal the documentation, TFS is used toprovide a direct link from the project based documenta-tion or the ‘bugs database’ through to the code that isdeveloped.In the first instance, the project team develops an im-

plementation document which is approved by two su-pervisors (the project supervisor and another supervisorwho only checks/approves the final deliverables) beforeit is passed onto developers.This document is used to generate an overarching

Task within TFS. This Task itself can have Subtasksdependent on the complexity of the project, for exampleGraphical User Interfaces (GUI) development, algo-rithms subtasks, outputs etc. All code that is then gener-ated by the development team has to be linked to theTask or Subtask thus providing a path between lines ofcode and the implementation document.Following on from first stage implementation there

may be a need to fix bugs and add enhancements. TFShas the facility to report a bug or request additional fea-tures that is accessible by all developers as well as thescientists who are involved in the design and testing ofthe platform. This allows for the raising and tracking ofreported/fixed bugs.

Data structure and integrity checkInherent in the design of the Simcyp Simulator is a sep-aration of data within discrete files from a relatively be-nign GUI that operates on those files and an Engine thatuses file-based inputs as its ‘menu of operations’. Apartfrom being in line with systems pharmacology concepts(in term of separation of systems, drug and trial designdata), this has a number of significant benefits: notablythe ability to share discrete collections of data more eas-ily within data libraries, to develop fast throughput sys-tems minimising user interaction and perhaps the mostimportant, an ability to develop an automated testingstrategy which both reduces testing time and ensures agreater integrity of the testing process.

Data files structureThe design of the Simcyp Simulator has been basedaround the portability of the underlying data. In otherwords all compound and population information can befed into Simcyp as external data files. There are twoprincipal classes of data that are necessary to run anysimulation, namely compound data and population data.These data classes respectively correspond to propertiesof (or values dependent on) a particular compound, andto data describing the demographic, physiology, geno-typic and phenotypic characteristics of a specific ethnicor disease population with relevant inter-individual vari-ability distributions.


These two types of data are conveniently stored withinXML-based files that can be viewed and accessed via theSimcyp GUI as well as other tools such as MicrosoftInternet Explorer. The schema of these files has beendesigned internally to allow forwards compatibility offiles over time. In other words the schema is fixed witheach release version and new parameters that are addedare done so without disrupting what already exists. Thisallows files created with a version of the Simulator todayto be used with later versions when they come outwhere any possible missing values are automatically re-placed with default values. All files contain a degree ofmeta-data showing varying information including theSimulator version used to create the file. In order to en-sure users are aware of changes a warning message isdisplayed indicating some data have been added/updatedand they should consult with the help file.

WorkspacesThe compound and population data files have limita-tions in that neither captures the simulation trial condi-tions nor the user optimisation options utilised on theuser screens. To cater for this a third type of file struc-ture known as a workspace has been developed. Againthe workspace file is XML-based; however this time itacts as a container for one or more compound files, oneor more population files as well as all trial informationand user defined settings. Because of its all-encompassingnature it is used as a snapshot of the running condition ofany simulation. In other words, to reproduce any simula-tion exactly, all that is needed is a copy of the workspacetaken at the time the simulation was run.As a Workspace contains full details of the simulation,

it is possible to use the workspace file alone to drive asimulation. In this manner the screens can become de-tached from the main engine enabling much greaterflexibility in designing and updating the interface. An-other benefit of defining workspaces is the ability to runa simulation in an automated/fast throughput mode.

Reporting simulation resultsAfter running simulations Microsoft Excel (MicrosoftCorp., Redmond (WA), USA), is used as the outputmedium to present the results. Excel was selected be-cause the majority of computers in the workplace wouldhave Microsoft Office software installed and additionallyExcel contains many powerful analysis tools that couldbe used to analyse the output data once in Excel.The reporting process is implemented through the

Excel Automation interface which is based on the OfficeObject Model. The Simcyp platform uses this technologyto create or connect to an Excel application ComponentObject Model (COM) object, to manipulate and addworksheets as required. Each worksheet is a bespoke

output based on the simulation input selections: eachcell is effectively created individually with the selectionof font (including size and weight), colour (both fore-ground and background), alignment of text within thecell, number format (based on the users’ machine selec-tion) as well as many other specifications.After the data have been rendered, Excel charts are

added if applicable. These may be concentration-timeprofiles or, for example, pie charts of enzyme contribu-tion which are created based on the data on the work-sheet and formatted individually based on userselections such as number format and also the colour‘skin’ chosen before outputting the data.The cost of using the Automation interface is speed:

sending large amounts of data across process boundariesis a great bottleneck. This can be improved using alter-native methods such as creating the Excel file directlywhich means the whole process will be a simple file in-put/output process and hence much quicker. Unfortu-nately, the development time required for this is hugeand the entire process would need a great deal of testingas this would require writing the code from scratch.Currently, an option is provided to write a subset of datato a CSV file. This is extremely quick and the data cansubsequently be imported by most packages (includingExcel). The down-side of CSV transfers is that no for-matting or graphs can be included.Other output options are the use of a relational data-

base. This is currently used by the Simcyp Batch Proces-sor but could be expanded to include standard Simcypoutputs. This would be a powerful tool to put the Simu-lator as part of a company workflow as it could thenwrite directly into a corporate database. The downsideof this is similar to the CSV option: all formatting andvisualisation would have to be done by the user.

Insertion of workspaces into excelThere is frequently a desire to reproduce a completesimulation from the results of an earlier simulationwhether to verify original findings or to continue run-ning similar simulations. In order to facilitate this, theSimcyp Simulator can embed a copy of the workspaceinto the Excel file while creating the Excel outputs. Thisis a two phase operation with the first phase being thecreation of a workspace ‘on the fly’ after a simulationhas taken place but before any output to the Excel file.The second phase is the embedding and subsequent en-cryption of the workspace within the Excel file. The em-bedded workspace can only be accessed by theSimulator at a later stage; therefore it is not possible tomanipulate or change the workspace data, so it can beused as a quality control measure. Although the embed-ding of a workspace is desirable both in terms of internaltraceability (input and output data fully interlocked) and


when submitting results to authorities, there is a smalloverhead involved both in terms of reporting time andfile size.

Simcyp Data Management (SDM) systemSimcyp Ltd often release a number of new and revisedCompound and Population files, typically on an annualbasis. These files are the culmination of a large amountof meta-analysis and changes may include new parame-ters for previous files when new features are incorpo-rated. This can result in variation between file contentsfrom version to version.There are two quality issues that surround the files

themselves. The first is the curation of data. Both theCompound files and the Population files can each havein excess of 1000 data items and generation of each ofthese files is usually the outcome of a meta-analysis ofmultiple sources. In order to impose a level of controlover this huge pool of data and to allow intelligentquerying, a tool known as the Simcyp Data Management(SDM) system has been developed. This tool takes in thenumbers that will go into the Simulator from scientistsvia whatever mechanism they exist in at the time andpasses the data through a series of approval proceduresbefore allocation to particular files. This happens for allvalues within a file, ensuring that a full history of thesource of each number is maintained. As the system isdatabase driven, it also enables searches to be carriedout across files and versions to allow cataloguing ofchanges to take place rapidly. As an end point the SDMgenerates the compound and population files themselvesthat are then bundled with the Simcyp Simulator. Thisremoves possibility of human error in the process andensures data integrity.These files are stored in an xml format. When new

values are added to the files i.e. when new features areadded, these new values are appended with no change tothe existing structure. A big benefit of this feature is toenable version control while allowing a new release ver-sion to read an older version’s xml file.

Model structure and testingSimcyp PBPK models are built using ordinal differentialequations (ODE). The models’ differential equations arehandled in a module called ‘Simpak’ which is an inde-pendent and scalable environment to accommodate thecontinuing evolution of the models and algorithmscontained within the Simulator (Jamei et al. 2009a). Oneof the main objectives in the Simulator’s design is to takeaway the model building burden from users. Therefore,users only select their desired models using the Simula-tor GUI and these selections are dynamically translatedto the relevant models in ‘Simpak’. To allow more flexi-bility and scalability ‘Simpak’ is highly modularised to

facilitate the combination of many different components.Currently, the Simulator handles over 1000 PK modelcombinations, involving:

� Single (small and large molecules) or multiple (upto 4) chemical moieties,

� Different absorption models, namely one-compartment,enhanced Compartmental Absorption and Transit(CAT), and Advanced Dissolution, Absorption andMetabolism absorption (ADAM) models,

� Different distribution models such as minimal andfull PBPK models with different perfusion- andpermeability-limited models, including multi-compartment liver, kidney, blood–brain-barriermodels and an additional multi-compartment userdefined organ/tissue,

� Modelling of up to 3 metabolites.

As an example, if the ADAM and full PBPK modelsare used while the gut, liver, brain and kidney transportersare incorporated for the substrate and main inhibitoraround 500 differential equations are automatically assem-bled to run the simulation.The ever increasing number of models and the expo-

nential rise of possible combinations have made modelchecking/testing a challenging task; therefore a range ofsolutions for model testing has been deployed which areexplained below.

Regression testingReproducibility of simulation result is a key requirementwhen users install different versions of the Simulatorand plays an equally important role during the develop-ment lifecycle of the Simulator. As the complexity of theSimulator has increased automated tools have had to bedeveloped to allow a large proportion of the testing to bedone automatically. The key players in this regime are theSimcyp Autotest package coupled with an in-house ExcelComparison tool; which compares the same worksheets intwo given Excel workbooks; that enables a full regressiontesting (comparing the results from a new build againsthistorical results) procedure to take place directly andautomatically after any development build of the Simula-tor. For this purpose a repository of workspaces that isever increasing, which cover a wide range of GUI options,is established and used during regression testing.

AutotestingThe Simcyp Autotest package allows the automated run-ning of multiple workspaces with pre-selection of rele-vant outputs. In order to make use of this as part of aregression testing, two areas need to be addressed.Firstly, any new build of the Simulator needs to generatesimilar results to those generated in a previous build for

MultipleWorkspaces

AutotestPackage

Excel Outputs

Summary

Results

Metrics

Excel ComparePackage

Historical resultsin database

Figure 2 The overall autotesting process which starts from running the repository of workspaces to the generation of summary reports.


an existing feature in similar scenarios. Secondly, testingneeds to be done to ensure any new feature needs to gen-erate defined outputs without interfering with existing fea-tures in any unpredictable/unexpected manner.To check this, existing results generated by running

defined Workspaces on early versions/builds need to bechecked against results from the current released buildof the Simulator to ensure that no unexpected discrep-ancies (i.e. where the underlying algorithms and/or datahave not been intentionally altered) occured. Addition-ally new features need to be run via newly definedWorkspaces to ensure that initially a benchmark andthen subsequently regression testing against that bench-mark can take place. In practice, project teams initiallymanually verify the results generated by new modulesoften against previously developed prototypes in otherenvironments such as the MATLAB (The MathworksInc., Natick (MA), USA) or R (1) packages. Further, pro-ject teams are asked to prepare a range of workspaces tocover required features. These workspaces will be re-peatedly run for each successive build and compared tothe previous run. The Autotest package was initially de-veloped as Python Scripts but has since been recoded inC# to add extra layers of functionality. The exposed fea-tures within the Simulator Engine however allow anyscripting engine to be used.Over time, new literature data leads to the revision

and updating of physiological and compound parametersused within the Simulator. When this happens, theworkspace files are updated with the latest values.

Consequently the corresponding benchmark test resultsare also modified for future regression testing.The Autotest package provides two options for deter-

mining the outputs. The first and default method is touse the Excel outputs that the user defined at the timethat the Workspace was saved. In this case the work-space serves as a complete input and output unit. Thesecond method is to define the required outputs viacommand line inputs within the script. Although effect-ive the latter approach can be time consuming given thelarge number of different sheets that can be generated.A final by-product of the Autotest system is metrics. A

wide range of metrics are gathered for each simulationand can be compared against previous Autotest runs inorder to ensure that either programmatic issues (mem-ory usage, simulation run time, etc.) and/or algorithmicissues (stiff differential equations) have not crept intothe design and had any adverse effect on time and/ormemory usage. The overall autotesting process is shownin Figure 2.

Excel comparison toolAn issue that occurred when running Autotest was thehuge number of Excel files that are potentially generatedas a result of running multiple workspaces across manyprojects. Since checking these files manually was very te-dious and time consuming, a new in-house package wasdeveloped to compare one set of Excel files (the currentrun) against an earlier set. Built into this utility was away of setting a threshold so that discrepancies could be


reported on a master report sheet with colour codingadded to the Excel files themselves to ‘mark’ the discrep-ancies. In this way a user can allow discrepancies of say1% to exist between values to cater for potentialrounding errors in Excel yet still report significant dis-crepancies. Similar checking is carried out on the met-rics files to verify time or memory differences are notoccurring.

Code coverage testing as part of regression testingGiven the high number of model combinations andavailable options, there were cases where the workspacesdid not cover specific areas. Hence, the question washow to make sure all areas of the code are sufficientlycovered for testing. To address this issue and assesswhich parts of the code were activated whilst running aspecific workspace, a code coverage tool called the Bulls-eye Coverage package (Bullseye Testing Technology,Washington, USA) is used. It determines which pathsthrough the code within the new module are called andtested and which remain to be. Therefore, this additionalchecking assists with identifying any shortcomings in thetest workspaces and allows an iterative approach to gen-erating a good quality set of test workspaces that coveras many options as practical.

Computer virus checkingThe Simulator is usually used in an environment(pharmaceutical companies and regulatory agencies)where it is essential to be free from any computer vi-ruses or malware and additionally for the software notto send any sensitive information outside of the com-pany. Therefore, the Simulator (after passing all internaltesting and checking and when it is ready to be releasedto clients) is sent out to the National Computing Centre(NCC). The NCC is an independent membership organ-isation for IT professionals and is the single largest andmost diverse corporate membership body in the UK ITsector. The centre runs a variety of tests on all Simula-tors (for human and animal species) to verify they arefree from any computer viruses or malicious codes anddo not broadcast anything outside of the computer. TheNCC then issues a certificate and endorses that specificrelease of the software after which the installer is re-leased to users.

Peripheral modulesDuring the course of Simulator development many addi-tional (peripheral) modules were added to extend its fea-tures and functionality. To retain the modularity of theplatform and facilitate its maintenance these moduleswere developed independently whenever possible andcalled in to perform specific tasks when needed. Oneof the very first peripheral modules was the Batch

Processor that allows running many simulations withoutuser intervention (Jamei et al. 2009a). In the current art-icle three major peripheral modules added since SimcypVersion 9, namely the Automated Sensitivity Analysis(ASA) tool, Parameter Estimation (PE) modules and thePharmacodynamics (PD) module, are described. Further-more, an update is provided on the differential equationsolver.

Automated sensitivity analysis toolOn occasion there is uncertainty in the true value of aninput parameter. This may be due to some particularparameter being unavailable for the drug of interest orbecause for that particular compound the in vitro data isunreliable. In these cases it is useful to check the impactthat the input parameter has on the simulation outcome.This can be achieved using the automated sensitivityanalysis (ASA) tool. This is a local sensitivity tool whichscans the selected parameters within a given range andreports the selected endpoints for a population represen-tative subject. ASA can be used to assess the impact ofchanging specific parameters (maximum of two at atime) on a range of PK/PD parameters or concentration-time profiles. For investigating more than two parame-ters the Batch processor (Jamei et al. 2009a) can be usedinstead. Identifying whether an input parameter has a sig-nificant impact on the outcome of a simulation is highlyvaluable as it assists with making decision on what in vitroassays should be done at what stages and how much re-source should be invested in obtaining a particular param-eter for a particular compound. ASA can be performed onvirtually any parameter displayed on the Simulator inter-face. After a parameter is selected the ASA interface toolis called within this interface the user can define the par-ameter ranges and the number of steps the range shouldbe divided into, as shown below (Figure 3).The user then selects the endpoint parameters to be

shown as Excel output, many of the PK/PD parametersand profiles are available for selection when ASA is used.Within the Excel outputs the selected endpoint valuesfor any of the selected input values along with the sensi-tivity index, which is the ratio of change in endpoint tothe change in input, and elasticity index, which mea-sures the relative change in endpoint for a relativechange in input, are provided. Generally, sensitivity ana-lysis is recommended prior to fitting a model to the ob-served data as these help to decide which parametersshould be or can be fitted.

Parameter estimation (PE)Another recently introduced feature of the Simulator isthe ability to simultaneously estimate up to 10 parame-ters to match clinical observations using the ParameterEstimation (PE) module (Zhao et al. 2012). One of the

Figure 3 A screen shot of the automated sensitivity analysis tool in Simcyp Version 12 Release 2; an example for assessing the impactof fraction unbound in plasma and the absorption rate constant on specific outputs where the minimum and maximum values, thesteps and the step-size distributions are defined.


biggest advantages of this module is that a user does notneed to write any models and their relevant differentialequations as all of these are automatically performed.This module allows non-modellers quickly and effi-ciently to fit very sophisticated models to clinical obser-vations. Historically, model fitting has been an area ofexpertise that only a limited number of modellers couldhandle and this limited the wide use of modelling andsimulation in the pharmaceutical industry. Introductionof modules such as PE has enabled a larger group of sci-entists to apply model-based drug development.Since many PK covariates are already built into the

Simulator (Jamei et al. 2009b), the covariate selection,which can be a challenging step in typical NonlinearMixed Effect (NLME) modelling, becomes a less challen-ging task. This is a major advantage of parameter esti-mation using models that have been developed in asystems pharmacology context.To allow entering the observed clinical data, PE tem-

plates are designed and implemented as add-ins withinExcel. The PE templates are structured in a manner thatfacilitates entering clinical data which are already savedin NONMEM (GloboMax, ICON Development Solu-tions, USA) software format. Within PE templates, twosets of data checking are carried out: first, while the data

are being entered and second, right before saving the fileto check the consistency of overall entered data. The lo-cation and reason of any errors are reported to users sothey can quickly ratify them. At the end the data aresaved in XML format.The available on screen parameters are dynamically

changed depending on the selected models and userscan easily select parameters to be estimated from thescreen which makes the fitting process intuitive. Afterselecting the parameters to be estimated invoking the PEmodule brings up an interface from which the optimisa-tion methods, error models, ending criteria and the pa-rameters' initial values and ranges can be entered. ThePE can fit models using both rich and independent indi-vidual data and sparse population data as it is done intypical PoPPK data analysis.A range of least square objective functions with different

weighting methods can be selected for fitting. These ob-jective functions can be fitted using classical Nelder-Mead(Nelder and Mead 1965) or Hooke-Jeeves (Hooke andJeeves 1961) methods or more modern methods such asGenetic Algorithms (Goldberg 1989). For NLME fittingthe Expectation-Maximisation (EM) (Dempster et al.1977) method is used to solve either of the MaximumLikelihood or Maximum A Posterior problem. A screen

Figure 4 A screen shot of the Parameter Estimation (PE) module that allows either of simulation or estimation modes. The observedclinical data are loaded in XML format and in the shown case the data include both plasma concentration and a PD response profile forsimultaneous fitting of PK and PD dependent variables.


shot of the main interface of the PE module is shown inFigure 4.

The PD (pharmacodynamic) frameworkSince Version 11 Simcyp has provided a collection ofPD models for simulation and parameter estimation tocomplement PK/ADME modelling facilities. The initialemphasis in development of this PD module has beento capture some well-known semi-empirical modelswithin building blocks (PD Response Units) which canreadily be linked together by generalized transductionto represent more complex responses. This early frame-work envisaged connecting simulated compounds (PKsubstrate/inhibitor/metabolite) to one or more responsesvia a directed acyclic graph (DAG) of linked units.Complexities of PD to PK feedback and other systemsbiology motifs were to follow on a slower timescale.Similarly, mechanistic details of receptor-based in vitro

to in vivo extrapolation (PD-IVIVE) were to be graftedon more gradually, as the science developed, within thisframework.Two types of PD Response Unit have so far been

implemented: the PD Basic Unit (Version 11) and thePD Link Unit (Version 12). Just one PD Basic Unit is allthat is needed to represent elementary PD (such as aHill model or Emax response) and a PBPK compartmentproviding a concentration or amount driving the re-sponse can be selected from a simple dropdown list;thus a specialist in metabolism or biopharmaceutical sci-ence does not need further training to add a quick pre-view of possible PD effects, consequent to somesimulated PK interaction or a formulation change.Nevertheless, with a PD Link Unit attached, Simcyp ex-tends the offered response models to include many in-direct models via link models well known topharmacometricians and PK/PD modellers, combining


concepts from Generalized Linear Models (McCullaghand Nelder 1989), Survival Analysis (Kalbfleisch andPrentice 2002), empirical disease progression (Chan andHolford 2001) and “Indirect Physiological PK/PD”(Dayneka et al. 1993). Figure 5 shows the screen repre-senting the PD Link Unit.PD Response Units are themselves subdivided into a

sequence of data processing steps with associated modelchoices from unit input to unit output and this is repre-sented on-screen as a flow from top to bottom. More-over, a Simcyp response comprising just two connectedPD Basic Units can be cast at different mechanisticlevels according to a user-chosen transduction mode:other than just empirical mode; operational mode maybe used to capture receptor binding-activation (up-stream) and stimulus–response (downstream) with theefficient transducer-ratio parameterisation of operationalagonism models (Black et al. 1985); or intrinsic efficacymode representing intricacies more familiar to receptorpharmacologists (Stephenson 1956, Karlin 1967), withexplicit receptor abundance (at least) as a system prop-erty to be pulled from a Simcyp population library. As ofSimcyp Version 12, PK/ADME extensions, particularly

Figure 5 Simcyp screen representing a PD link response unit: input frand then feed into either a growth/progression/turnover model, surv

for biologic drugs with high target affinity, have includedtarget-mediated drug disposition (TMDD) models(Mager and Jusko 2001) where target receptor bindingaffects drug disposition profiles: from Version 12 Release2 these models are also available for PD response.Completion of the full DAG framework would require

two further items, namely a PD Interaction Unit tojoin responses of different compounds according tophamacodynamic interaction models and a responsesplitting motif which allows unit outputs to input tomore than one downstream unit.The framework so far described, and included in Ver-

sion 11, is sufficient to allow a user to input elementarydescriptions of inter-individual variability in PD modelparameter distribution as well as optimise the popula-tion mean parameters (possibly within the context of ahigh-dimensional PBPK model). However it is importantultimately to provide cross-links from potential predic-tors in Simcyp’s virtual populations (“covariates”) whichcondition PD model parameters to account for PK/PDcorrelation mechanisms to underpin pharmacometricsconcepts active within the NONMEM community: theintrinsic efficacy and custom scripting facilities (see next

om the previous unit can go through a Link transform modelival model or custom scripted model.


section) already go some way towards this idealframework.In addition to simple time-independent and PK/PD

profiles modes, the PD Simulator in Version 12 also ac-commodates possibilities of time-summaries convertingprofile responses into time-independent responses and the“LONG” timescale mode feature in a Link Response Unitwhich allows some link models (such as survival models)to be run on an extended timescale after a time-summarised or otherwise time-independent input.

Custom PD scriptingA facility allowing user-scripted custom models to beinserted as replacement processing steps into the PDframework was developed in Version 12 Release 2. Lua(2) was selected for (first) implementation of this require-ment as a freely available, lightweight and efficient embed-ded scripting language with widespread support. Inparticular, to function well, a script interpretation needs tobe very fast. It is likely that later versions of Simcyp willinterface more extensively with other (typically compiled)modelling languages familiar to pharmacometricians andPK/PD scientists (such as Pharsight Modelling Language(PML)). In any case, the provision of scripting extends therange of models that Simcyp can simulate.The execution context of user-scripted code in a

population Simulator with hierarchical variability andmany predefined models needs careful consideration. InSimcyp Version 12 Release 2 this is primarily controlledby the positioning of a script as a replacement for a spe-cific response step in a specific response unit for a particu-lar compound. Simcyp also provides several placeholdersin its code for users to implement call-back functions fordifferent simulation contexts. Simcyp Version 12 Release2 includes a dedicated Lua script editor (based on Scintillaediting software components (3)) which supplies tem-plates for such code. It also implements a library ofSimcyp functions callable from Lua scripts: for storing var-iables in the Simulator and accessing or manipulating ele-ments of the PK and PD simulation. This notably includesread access to potential covariates in a virtual population.The custom scripting module also facilitates handling

of user defined differential equations. For this purpose, ablock of state variables and corresponding time gradientsare reserved for user scripting of ordinary differentialequations (ODEs). Just as the built in engine has accessto the state variables and predict time gradients for thegeneric ODE solvers, so will the custom defined differ-ential equations do the same for the user block, and thisblock will be re-indexed into the full state array bySimcyp.User step functions will be called many times as the

ODE solver updates the model predictions for eachtime-step so script efficiency is crucial here. Benchmark

performance tests of Lua prototype ODE code havehowever demonstrated impressive performance: thesetests contributed to the selection of Lua over other pos-sible scripting options.

Expansion of differential equation solversPBPK models are dynamic models constructed using dif-ferential equations. Therefore, having a robust, efficientand reliable differential equation solver engine is an es-sential part of any software that deals with such models.As the number and complexity of implemented modelswithin the Simulator are increasing, the possibility of en-countering stiff ordinary differential equations (ODEs)has increased. Therefore it was decided to introduce asecond differential equation solver that can efficientlyhandle stiff differential equations. In practice the stiff-ness of ODEs is a transient phenomenon: i.e. an ODEmay be stiff in some interval and non-stiff in others.Hence, it is desirable to choose a scheme which can dy-namically follow the qualitative behaviour of the ODE.After assessing a range of solvers it was decided to useLSODE (Livermore Solver for Ordinary DifferentialEquations) (Radhakrishnan and Hindmarsh 1993).LSODE is a very robust method that combines the cap-abilities of the GEAR approach (Gear 1971) and solvesexplicitly given stiff and non-stiff system of ODE. It of-fers a number of features (see (Hindmarsh 1983)) whichare more convenient, flexible, portable, and easier toinstall in software libraries.A variant of LSODE called LSODA (the suffix A

stands for automatic) (Petzold 1983) was adopted. Thealgorithm used within LSODA has the capability toswitch automatically between stiff and non-stiff methods.This is very convenient for the Simulator as it optimisesthe simulation speed. Moreover LSODA is known to bepotentially more efficient than the basic LSODE whenthe nature of the problem changes between stiff andnon-stiff in the course of the solution. The Livermoresolver was introduced in Simcyp Version 9 in additionto the fifth-order Runge–Kutta method (Jamei et al.2009a). The fifth-order Runge–Kutta method is very ro-bust for the majority of simulations. Hence we kept bothsolvers and set the Runge–Kutta as the default solver.Users can switch between the solvers and there are caseswhere according to the selected combination of modelsthe Simulator suggests or enforces selecting the Livermoresolver.Both of the solvers are coded in C++ and optimised to

address specific needs in dealing with complex dose ad-ministration regimens. The ODE solvers are located inan independent module which is completely separatedfrom the other modules so it can be used as an inde-pendent engine and called up wherever needed evenoutside the Simulator.


Memory usage and speed maintenanceMemory managementDue to the number of additional outputs required ineach version as well as the need for larger populationsizes, improvements had to be done on the way the gen-erated data are maintained. The first stage for improve-ments was the removal of duplicated data; this wasmostly in the area of how individuals’ data were stored.Removing duplicated data significantly increased thepossible number of simulated subjects and also im-proved the time taken to create the individuals data.The next step was to improve the way concentration

time profile data were handled. This was accomplishedvia two stages of improvements. Firstly the stored datawere split so that systemic plasma concentrations werekept in greater detail than the rest of the concentrationtime profiles. Secondly a compression algorithm was in-troduced, with a pre-processing step that increases thecompressibility. The pre-processing step takes advantageof the fact that both integer and floating point data use4 bytes of memory and keeps the integer value of sub-traction of two consecutive values rather than their ab-solute values. Without pre-processing representativedata were compressed to 40% of original size; howeverafter pre-processing they reduced to 25%.

Using multiprocessor and multicore advantagesThe advancement of laptops and workstation machineshas meant that support for multiple processing cores isparamount for modern software. Since the main compu-tational burden in the Simulator is solving simultaneoussets of equations, there were limitations on how to ap-proach the use of these additional cores. Nevertheless,given the population based nature of the simulationsone candidate solution was to simulate multiple individ-uals concurrently. Interestingly, there were limitationson the gained efficiency when the computation load wasdivided among different cores. Testing different com-puter configurations revealed that the main consider-ation for the scalability appears to be the size of theprocessor cache, the larger the cache the greater thescalability. Therefore, whenever multiprocessor and/ormulticore machines are available the Simulator has beendesigned to take advantage of them.

Maintaining the simulation speedAs the complexity of models and the number of differ-ential equations increase so does the possibility ofslowing down the simulation speed. To obtain quantita-tive measures of the speed performance of the Simulatoran in-house package called Profiler is developed andused to measure the runtime duration for a wide rangeof internal tasks. The Profiler statistics are regularlyassessed during the development process and in

particular when new modules are introduced and com-pared against the historical records. Typical types of col-lected data include which functions in the code arebeing called, how many times, how much time is spentinside the function and in many cases how much timewas spent on specific sections of code inside those func-tions. Whenever a speed reduction is observed the rele-vant functions are investigated and are possiblyoptimised to regain the lost speed. Sometimes the savingof only a few milliseconds in a function can add up tolarge savings over the course of an entire simulation.One of the major tasks undertaken as part of these

optimization processes was to rewrite portions of bothof the Simcyp Simulators differential equation solversusing processor instruction sets which take advantage ofSingle Input Multiple Data (SIMD) facilities known asStreaming SIMD Extension (SSE) Intrinsics (4). Ma-chines supporting SSE2, which allows multiple floatingpoint instructions to be carried out simultaneously, havebeen available since 2001 (Intel, Santa Clara, CA, USA)and 2003 (AMD, Sunnyvale, CA, USA) and are now in-cluded in all commercially available x86 processors. Thedifferential equation solvers in the Simcyp Simulator usedouble precision data, so application of SSE2 instruc-tions allowed two calculations to be performed for eachprocessor clock cycle thus improving the speed and effi-ciency. In order to make full use of the power of theseinstructions the data has to be specifically aligned to 16byte boundaries; also the size of the data must always bean even number, so if an odd number of items is re-quired a dummy set must be added to the end. Thechanges significantly improved the overall simulationtime of complex models by approximately 10% for theRunge Kutta solver and 20% for the Livermore solver.

ConclusionsThe iterative and incremental development of theSimcyp platform has facilitated its rapid expansion andthe consortium guidance has kept it in line with users’requirements. Applying the concept of separation of da-tabases and models has assisted with implementing thesystems pharmacology paradigm. Further, since scientistsare directly involved in the design and developmentprocess and working closely with developers the wholesoftware development life cycle has improved and short-ened. Implementing the auto and regression testing pro-cesses has reduced the burden of testing and the SDMsystems has smoothened the integration and tracking ofdata in the databases. As the role and applications ofmodel-based drug development approach increase sodoes the need for peripheral modules (e.g. PD, PE, ASA,Batch processor, etc.) that can seamlessly connect andinteract with variety of different models and databases.Although more than a decade of internal experience and


many more years of external knowledge have gone intothe development of the Simulator there is a long way togo and lots more development and expansions are yet tocome. Handling of large molecules with full PBPKmodels, expansion of the current models to all com-pound types and expansion of the population databasesto other special groups (e.g. geriatric populations, differ-ent ethnicities) are only some of the potential future de-velopments. More exciting and challenging directionsinclude identifying and incorporating response covariates,incorporating physiological changes in disease progressionand addressing safety issues such as cardiotoxicity,nephrotoxicity, hepatotoxicity and neurotoxicity withmechanistic models. It is also envisaged to connect theSimulator to other packages such as Tripos (a Certaracompany, St. Louis, USA) and Pharsight (a Certara com-pany, St. Louis, USA) products to apply the MBDD para-digm in different phases of the drug discovery anddevelopment cycle.

Websites(1) The R-Project for Statistical Computing [http://www.r-project.org](2) Scintilla and SciTE [http://www.lua.org](3) The Programming Language Lua [http://www.scintilla.org](4) Microsoft Developer Network - Visual Studio .Net2003 MMX, SSE, and SSE2 Intrinsics 05/03/2013 [http://msdn.microsoft.com/en-us/library/y0dh78ez(v=vs.71).aspx]

AbbreviationsADME: Absorption distribution, metabolism, and excretion;PK: Pharmacokinetics; PD: Pharmacodynamics; PBPK: Physiologically-basedpharmacokinetic; MBDD: Model based drug development; DDI: Drug-druginteraction; SDM: Simcyp data management; TFS: Team foundation server;PML: Pharsight modelling language; NONMEM: Nonlinear mixed effectsmodelling; NLME: Nonlinear mixed effects; GLM: Generalised linear model;COM: Component object model; ODE: Ordinary differential equation;LSODE: Livermore solver for ordinary differential equations; LSODA: LSODEwith automatic switching; SSE: Streaming SIMD extensions; SIMD: Singleinstruction multiple data; NCC: National computing centre; IT: Informationtechnology; XML: Extensible markup language; ASA: Automated sensitivityanalysis; PE: Parameter estimation; ADAM: Advanced dissolution absorptionand metabolism.

Competing interestsMasoud Jamei, Steve Marciniak, Duncan Edwards, Kris Wrag, Kairui Feng andAdrian Barnett are employees in Simcyp Limited (a Certara Company). AminRostami-Hodjegan is an employee of the University of Manchester and part-time secondee to Simcyp Limited (a Certara Company). Simcyp’s research isfunded by a consortium of pharma companies.

Authors’ contributionsAll authors have contributed in the design and development of algorithmsand modules within the Simulator. SM, DE, KW, KF and AB have beeninvolved in implementing different aspects of the code. All authorscontributed in writing the paper. All authors read and approved the finalmanuscript.

AcknowledgmentsThe authors thank the rest of the Simcyp team for their contributions to thedevelopment of the Simulator. We also appreciate the continued support of

the Simcyp consortium members. The Simcyp Simulator is freely available,following completion of an appropriate training workshop, to approvedmembers of academic institutions and other non-for-profit organizations forresearch and teaching purposes. The help of James Kay in preparing themanuscript and Kristin Lacy in preparing Figure 2 is appreciated.

Author details1Simcyp Limited (a Certara Company), Blades Enterprise Centre, John Street,Sheffield S2 4SU, UK. 2Centre of Applied Pharmacokinetic Research, theSchool of Pharmacy and Pharmaceutical Sciences, the University ofManchester, Manchester, UK.

Received: 28 March 2013 Accepted: 16 May 2013Published: 3 June 2013

ReferencesBlack JW, Leff P, Shankley NP, Wood J (1985) An operational model of

pharmacological agonism: the effect of E/[A] curve shape on agonistdissociation constant estimation. Br J Pharmacol 84:561–571

Bouzom F, Ball K, Perdaems N, Walther B (2012) Physiologically basedpharmacokinetic (PBPK) modelling tools: how to fit with our needs?Biopharm Drug Dispos 33:55–71. doi:10.1002/bdd.1767

Chan PL, Holford NH (2001) Drug treatment effects on disease progression. AnnuRev Pharmacol Toxicol 41:625–659

Dayneka NL, Garg V, Jusko WJ (1993) Comparison of four basic models ofindirect pharmacodynamic responses. J Pharmacokinet Biopharm21:457–478

Dempster AP, Laird N, Rubin DB (1977) Maximum likelihood from incompletedata via the EM algorithm. J R Stat Soc Series B 39:1–38

Gear CW (1971) Numerical initial value problems in ordinary differentialequations. Prentice-Hall, Englewood Cliffs, NJ

Goldberg DE (1989) Genetic algorithms in search, optimization, and machinelearning. Addison-Wesley Pub. Co., Reading, Mass

Guckenheimer S, Loje N (2012) Visual studio team foundation server 2012:adopting agile software practices, from backlog to continuous feedback(Microsoft windows development). Pearson, Boston

Hindmarsh AC (1983) ODEPACK, a systematized collection of ODE solvers. IMACStransactions on scientific computation, 10th IMACS world congress onsystems simulation and scientific computation, August 8–13, 1982. :55–64

Hooke R, Jeeves TA (1961) Direct search solution of numerical and statisticalproblems. J Assoc Comp Mach 8:212–229

Jamei M, Marciniak S, Feng K, Barnett A, Tucker G, Rostami-Hodjegan A (2009a)The Simcyp® population-based ADME simulator. Expert Opin Drug MetabToxicol 5:211–223

Jamei M, Dickinson GL, Rostami-Hodjegan A (2009b) A framework for assessinginter-individual variability in pharmacokinetics using virtual humanpopulations and integrating general knowledge of physical chemistry,biology, anatomy, physiology and genetics: a tale of ‘bottom-up’ vs ‘top-down’ recognition of covariates. Drug Metab Pharmacokinet 24:53–75

Kalbfleisch JD, Prentice RL (2002) The statistical analysis of failure time data. JohnWiley & Sons, Hoboken, New Jersey

Karlin A (1967) On the application of “a plausible model” of allosteric proteins tothe receptor for acetylcholine. J Theor Biol 16:306–320. doi:10.1016/0022-5193(67)90011-2

Larman C, Basili VR (2003) Iterative and incremental development: a brief history.Computer 36:47–56. doi:10.1109/mc.2003.1204375

Mager DE, Jusko WJ (2001) General pharmacokinetic model for drugs exhibitingtarget-mediated drug disposition. J Pharmacokinet Pharmacodyn 28:507–532

Martin RC (2011) Agile software development, principles, patterns, and practices.Pearson, London

McCullagh P, Nelder JA (1989) Generalized linear models. Chapman & Hall, BocaRaton

Nelder JA, Mead R (1965) A simplex method for function minimization. Comput J7:308–313. doi:10.1093/comjnl/7.4.308

Petzold L (1983) Automatic selection of methods for solving stiff and nonstiffsystems of ordinary differential equations. SIAM J SCI Stat Comput 4:136–148

Radhakrishnan K, Hindmarsh AC (1993) Description and Use of LSODE, theLivermore solver for ordinary differential equations. Lawrence Livermorenational laboratory, university of California. U.S. Department of Energy ReportUCRL-ID-113855

http://www.r-project.org

http://www.r-project.org

http://www.lua.org

http://www.scintilla.org

http://www.scintilla.org

http://msdn.microsoft.com/en-us/library/y0dh78ez(v=vs.71).aspx

http://msdn.microsoft.com/en-us/library/y0dh78ez(v=vs.71).aspx

http://dx.doi.org/10.1002/bdd.1767

http://dx.doi.org/10.1016/0022-5193(67)90011-2

http://dx.doi.org/10.1016/0022-5193(67)90011-2

http://dx.doi.org/10.1109/mc.2003.1204375

http://dx.doi.org/10.1093/comjnl/7.4.308


Rostami-Hodjegan A (2012) Physiologically based pharmacokinetics joined within vitro-in vivo extrapolation of ADME: a marriage under the arch of systemspharmacology. Clin Pharmacol Ther 92:50–61

Rowland M, Peck C, Tucker G (2011) Physiologically-based pharmacokinetics indrug development and regulatory science. Ann Rev Pharmacol Toxicol51:45–73. doi:10.1146/annurev-pharmtox-010510-100540

Sims C, Johnson HL (2012) Scrum: A breathtakingly brief and agile introduction.Dymaxicon

Stephenson RP (1956) A modification of receptor theory. Br J PharmacolChemother 11:379–393

Teorell T (1937) Studies on the diffusion effect upon ionic distribution: II.Experiments on ionic accumulation. J Gen Physiol 21:107–122

Zhao P, Zhang L, Grillo JA, Liu Q, Bullock JM, Moon YJ, Song P, Brar SS,Madabushi R, Wu TC, Booth BP, Rahman NA, Reynolds KS, Berglund EG,Lesko LJ, Huang SM (2011) Applications of physiologically basedpharmacokinetic (PBPK) modeling and simulation during regulatory review.Clin Pharmacol Ther 89:259–267

Zhao P, Vieira ML, Grillo JA, Song P, Wu TC, Zheng JH, Arya V, Berglund EG,Atkinson AJJ, Sugiyama Y, Pang KS, Reynolds KS, Abernethy DR, Zhang L,Lesko LJ, Huang SM (2012) Evaluation of exposure change of nonrenallyeliminated drugs in patients with chronic kidney disease usingphysiologically based pharmacokinetic modeling and simulation. J ClinPharmacol 52:91S–108S. doi:10.1177/0091270011415528

doi:10.1186/2193-9616-1-9Cite this article as: Jamei et al.: The Simcyp Population Based Simulator:Architecture, Implementation, and Quality Assurance. In SilicoPharmacology 2013 1:9.

Submit your manuscript to a journal and benefi t from:

7 Convenient online submission

7 Rigorous peer review

7 Immediate publication on acceptance

7 Open access: articles freely available online

7 High visibility within the fi eld

7 Retaining the copyright to your article

Submit your next manuscript at 7 springeropen.com

http://dx.doi.org/10.1146/annurev-pharmtox-010510-100540

http://dx.doi.org/10.1177/0091270011415528

original research open access the simcyp population based

Documents