adobe marketing cloud data workbench controlled experiments · data workbench controlled...

Adobe® Marketing Cloud

Data Workbench Controlled Experiments

Contents

Data Workbench Controlled Experiments.............................................................3

How Does Site Identify Visitors?.............................................................................................3

How Do Controlled Experiments Work?..................................................................................3

What Should I Test?................................................................................................................4

What Are the Requirements?..................................................................................................5

How Do I Implement a Controlled Experiment?......................................................................5

Enabling Controlled Experimentation...................................................................6

Modifying the ExpFile Parameter............................................................................................6

Modifying the ExpCookieURL Paramter (Optional).................................................................6

Modifying the ExpPartialMatch Parameter (Optional).............................................................7

Creating a Controlled Experiment..........................................................................8

Defining the Objective.............................................................................................................8

Forming a Hypothesis.............................................................................................................9

Determining the Number of Visitors Needed.........................................................................11

Creating the Test Content......................................................................................................11

Configuring and Deploying the Experiment...........................................................................11

Validating the Experiment.....................................................................................16

Viewing the Results...............................................................................................18

Configuring the Dataset........................................................................................................18

Modifying Log Processing.cfg..................................................................................................................18

Modifying Transformation.cfg....................................................................................................................19

Viewing the Experiment Results............................................................................................21

Evaluating the Experiment....................................................................................................21

Experiment Design Spreadsheet..........................................................................24

Data Workbench Controlled ExperimentsLast updated 2/8/2017

Data Workbench Controlled ExperimentsControlled experiments are tests that enable you to compare results obtained from an experimental sample groupwith those from a standard control group.

Site enables you to implement, measure, and analyze controlled experiments and their results as they relate todifferent aspects of your website. Doing so enables you to test hypotheses regarding the improvement of websiteperformance before spending significant time or money fully implementing the proposed changes.

Note:

Site experiments can be analyzed only in datasets where the only method of visitor identification in use is theSensor set persistent cookie method. Sensors running on J2EE servers (JBoss, Tomcat, WebLogic, andWebSphere) do not support controlled experimentation. For more information, see the following section.

Using Site, you can implement A/B, A/B/A, and multivariate controlled experiments to gather enough test data toprovide statistically accurate data for a detailed evaluation of your hypothesis, without impacting current websiteperformance.

How Does Site Identify Visitors?

A typical configuration of Site uses cookies to uniquely identify visitors to your website and track their behavior overtime.

The first time that a particular browser (considered a visitor) makes a request of your website, Sensor works withyour web server to set a persistent cookie (by default cs(cookie)(v1st)), which is interpreted internally withinthe system as x-trackingid. This cookie is set only once, on the very first request made to your website by thatvisitor. It then is collected from that visitor each time that browser makes a request (either page or embedded objectrequest) of your website in the future.

Accepting a persistent cookie is at the browser’s discretion. If a user does choose to block persistent cookies, theirpage view requests are still logged, but the measurement data from those requests are not correlated to a particularvisitor or their sessions on the website unless you implement an alternate method of visitor identification, such asusing the Hash transformation on the IP and UserAgent fields.

Note: Site experiments can be analyzed only in datasets where the only method of visitor identification in useis the Sensor set persistent cookie method. Sensors running on J2EE servers (JBoss, Tomcat, WebLogic,and WebSphere) do not support controlled experimentation.

During a controlled experiment, users who do not accept cookies could be placed in different experiment groupsfrom one click to the next. This becomes an issue only if you perform your test analysis with the Broken SessionFilter turned off in Insight, which Adobe does not recommended.

For more information about the Broken Session Filter, see the Insight User Guide.

If a visitor clears the cookie during an experiment, the visitor is assigned a new cookie and potentially could beassigned to a different group. Because Adobe identifies the visitor as new, the experiment is not invalidated.

How Do Controlled Experiments Work?

In an experiment, you can define any number of test groups in addition to the control group.

3Data Workbench Controlled Experiments

When an experiment is running, all visitors to your website become part of the experiment, either as part of a testgroup or of the control group, as soon as they access any page involved in the experiment. Visitors are allocated toyour experiment groups randomly, in proportions defined during the experiment configuration.

Controlled experiments are implemented using the Sensor software that is installed on each of the content serversin your web cluster. As the content servers receive requests, Sensor randomly selects visitors for your test groupsand redirects their page requests to the experimental content.When Sensor selects a visitor to view the test content,the address bar continues to list the originally requested URI but the visitor is routed to the test URI. Because thisprocess takes place internally in the server application, users are not aware of when they are being tested, whichis an important consideration for unbiased experimentation.

Sensor passes the test URIs, not the original URI displayed to the user, to the log files for use in analysis.

The results of the experiments can be analyzed easily using Insight to determine whether the experimental hypothesisthat you were testing is correct.

Note:

Adobe strongly recommends that controlled experiments be coordinated and performed with input from thoseindividuals in your organization who are responsible for configuring and maintaining your analysis datasets.

What Should I Test?

Test results must be clear and meaningful so that you can feel confident making large dollar decisions based onthose results.

Although you can test various page layouts with Sensor and Site, Adobe suggests that you focus on testing high-value,strategic business initiatives, or new or redesigned website functionality that address the goals that you have setfor your website as well as for your business.You can test for such issues as best price guarantees, personalizationfunctionality, market offers (for example, packages or bundles), creative design, and application processes.

The following concepts are most important when developing your controlled experiment:

• Understand the right changes to make. This requires some research into how your website functions and thebusiness processes underlying the front-end website.You want to make changes that provide the most impactand can be tested easily.

• Small changes can have significant impact. Not all of the changes that you test need to be drastic to have asignificant impact on your business. Always be open to making small, but very important changes.

Supported Methodologies

Many types of experiments with many different goals can be performed using Site. The following list provides a fewexamples:

• Altering pages, content, and website processes to improve conversion rates.

• Changing marketing campaigns, promotions, cross-sells, and up-sells to increase revenue.

• Varying page load times to understand customer quality of service and the actual value of infrastructure performance.

To reach these goals, Site supports the following types of methodologies for controlled experimentation and testing:

• Page Replacement: Replace static URL X with static URL Y. This methodology is of limited use in a dynamicenvironment.


• Dynamic URI Replacement: This is a variant of Page Replacement that replaces static page X with dynamic pageY to render dynamic content.

• Object Replacement: Replace fixed object X with fixed object Y.

• Content Replacement: Replace content set X (multiple objects, pages, table, and so on) with content set Y.

• Experiment Variable Replacement: Replace JavaScript object /writeCookie_X.js with JavaScript object/writeCookie_Y.js to write a cookie that can be used by a back-end system to serve particular content.

Note:

Controlled experiments are based on URI replacement, not query string replacement.The URI within a particularURL is highlighted in the following example:

http://www.omniture.com/index.asp?id=1

For example, in your controlled experiment you could specify that the control group URI index.asp bereplaced with the test group URI index2.asp to determine which page design would result in more value.

What Are the Requirements?

To perform a controlled experiment on your website using Site, you must meet the following requirements:

• Sensor must be installed and working properly on each web or application server that supports the website contentthat you are testing.

Note: Sensors running on J2EE servers (JBoss, Tomcat, WebLogic, and WebSphere) do not supportcontrolled experimentation.

• You must have a process in place for pushing content to all of your web or application server(s), such as a contentmanagement system.

How Do I Implement a Controlled Experiment?

Controlled experimentation with Site and Sensor is designed to be simple, feasible, and actionable.

The rest of this guide describes how to fully implement a controlled experiment:

• Enabling Controlled Experimentation

• Creating a Controlled Experiment

• Validating the Experiment

• Viewing the Results


Enabling Controlled ExperimentationTo enable controlled experimentation, someone with administrator access to your web or application servers mustmodify the ExpFile parameter in the Sensor configuration file (usually named using txlogd.conf) on each webor application server in your web cluster on which a Sensor is installed.

In addition, two other parameters in this file can be modified to implement a testing tool (ExpCookieURL parameter)or to remap large sections of your website (ExpPartialMatch parameter). This chapter provides more informationabout these parameters.

To edit the txlogd.conf file

If you have administrator access, complete the following steps. If you do not have administrator access, contactyour system architect to request the changes, providing them with the following steps.

1. Navigate to the Sensor installation folder on a web or application server in your web cluster on which a Sensoris installed.

2. Open the Sensor configuration file (usually named using txlogd.conf) using a text editor and edit the file asindicated in Modifying the ExpFile Parameter and optionally in Modifying the ExpCookieURL Paramter (Optional)and Modifying the ExpPartialMatch Parameter (Optional).

3. Save and close the file.

4. Repeat this procedure for each web or application server in your web cluster on which a Sensor is installed.

Modifying the ExpFile Parameter

The ExpFile parameter points to the location of the experiment configuration file, which defines your experiment.

Setting this parameter enables you to run experiments. For steps to create the experiment configuration file, seeConfiguring and Deploying the Experiment.

Following is an example of the ExpFile parameter:

ExpFile /home/experiment.txt

This tab delimited text file (.txt) can be located anywhere in the Sensor folder and can have any convenient name.

Make sure you record the location of the experiments directory and the name of the configuration file that you specifybecause you need to save your experiment configuration file (to be described later in this guide) using this nameand in this directory.

Note: If you do not set this parameter identically on each machine in your web cluster on which a Sensor isinstalled, controlled experimentation does not work.

This entry can be preconfigured and remain in the Sensor configuration file on an ongoing basis with no adverseeffect. If the experiment configuration file name specified is not found by Sensor or it is blank (that is, it exists buthas no content), Sensor does not conduct the experiment, logs an error event on the HTTP server, and continuesto operate normally in all other respects.

Modifying the ExpCookieURL Paramter (Optional)

The ExpCookieURL parameter can be used to test that your controlled experiment is working properly.

6Enabling Controlled Experimentation

This parameter defines the URL of a virtual page that when requested places you into a specified experiment andgroup and then redirects you to the root of your website. From that point through the end of the experiment, you arepart of the specified experiment and group.

The default page for this parameter is setcookie.htm, but you can use any valid virtual URL.

Note: This must be a virtual URL and must not be a real page or piece of existing content. There should beno file on the web server at the specified path with the specified name.

Following is an example of the ExpCookieURL parameter:

ExpCookieURL /setcookie.htm

Modifying the ExpPartialMatch Parameter (Optional)

If you want to enable your controlled experiments to remap your entire website or an entire subdirectory of yourwebsite to another location, you can set the ExpPartialMatch parameter in the txlogd.conf file to “on.” The defaultis “off.”

Following is an example of the ExpPartialMatch parameter:

ExpPartialMatch off

Be very careful when setting this parameter to “on” because it can result in an inadvertent remapping of your entirewebsite.

7Enabling Controlled Experimentation

Creating a Controlled ExperimentThe overall objective of running a controlled experiment on a website is to determine whether and to what degreea defined change or set of changes produces an effect on the users of the website.

Defining a controlled experiment involves the following steps:

1. Defining the Objective

2. Forming a Hypothesis

3. Determining the Number of Visitors Needed

4. Creating the Test Content

5. Configuring and Deploying the Experiment

Defining the Objective

When defining your objective, consider the purpose of the website or website process you are analyzing: What isits primary function? Who is its target audience?

Common website objectives include converting more visitors into customers or increasing the average amount ofrevenue gained for all visitors within an experiment over the duration of that experiment.

Common website process objectives include improving the steps or pages in a process that cause visitors to abandonthe process, removing unnecessary and confusing options in the process that tend to stop visitors from reachingthe end of the process, or consolidating or expanding the process to eliminate or add steps or pages.

Make sure to think carefully about what it is specifically that you want to understand about your website. Carefullyplanning your experiment makes the results much more meaningful to your business.

Objective: To increase the number of visitors to our website who request a demo of our product using the “Requesta Demo” graphical link, as shown in the following image:

8Creating a Controlled Experiment

Forming a Hypothesis

A hypothesis is an assumption, which can be taken as true for the purpose of argument, that provides a tentativeexplanation that can be tested by further investigation.

Try to think about your hypothesis in terms of alternative pages, images, or processes that could be substitutedeasily for existing ones.

Your hypothesis must be able to produce a result with statistical significance. This can be achieved by increasingthe percentage of visitors included in the test, or by running the test for a longer period of time.

At this point, you also should define your visitor-based success metrics, either as part of the hypothesis or as anadditional matrix.

Hypothesis: Moving the “Request a Demo” graphical link closer to the top of the page results in a Visitor Conversionincrease of at least 1.5%.

In our example hypothesis, we have defined the success criterion for this experiment as an increase in VisitorConversion of at least 1.5%.


Determining the Number of Visitors Needed

To create a statistically significant experiment, you must determine how long you need to run the experiment toinclude enough visitors to effectively evaluate the results of the changes to your website.

If you need help determining the minimum length of your experiment, you can use the experiment design spreadsheetprovided by Adobe as a tool to help you design your experiment. This file, named VS Controlled ExperimentDesign.xls by default, functions not only as a worksheet but also as a record of your decisions about the experiment.For more information about this file, see Experiment Design Spreadsheet.

Note: The experiment design spreadsheet can provide useful statistical inferences only when the metric inquestion is defined as a percentage of visitors that meet some criteria. That is, it is useful only when trying totest a visitor-based metric hypothesis.

Creating the Test Content

Before you configure the experiment, you should create the alternate content that you want to use in the experiment.

The control group is sent to the original URI, while the test group is sent to the new, alternate URI.

To avoid confusion, do not reuse test group file names. For example, if you run an experiment using a test groupfile named test2.asp, do not use test2.asp as the name for the test file in your next experiment.

For the hypothesis that moving the “Request a Demo” graphical link on your home page impacts Visitor Conversion,we create the alternate home page containing the “Request a Demo” graphical link in the new position.The followingsection describes how you then specify that the control group URI index.asp be replaced with the test group URIindex2.asp for a certain percentage of visitors.

Configuring and Deploying the Experiment

After you have defined your objective, hypothesis, and experiment details as well as created your test content, youmust configure Sensor to deploy the controlled experiment.

You do this by completing the following two steps:

Configuring the Experiment Configuration File

Deploying the Configuration File and Test Content

Configuring the Experiment Configuration File

To configure the experiment, you must complete the experiment configuration spreadsheet provided by Adobe(named TestExperiment.xls by default). This file configures Sensor to perform the experiment and is the Excelversion of the text file that you specified in Modifying the ExpFile Parameter.

This file can contain information about multiple experiments, which can run at the same or at different times and usedifferent groups and percentages, but these experiments are not correlated in any way.

Users are placed into a group for each experiment listed in the file that is configured to be running at this time.


Note: Each experiment is independent of all other experiments. Changes you make to one experiment donot affect any other experiment, and although visitors may be in multiple experiments, the results do not relateto one another. If you think a correlation exists between the changes in multiple experiments, you must createa new experiment that tests these changes together.

To configure your experiment

You should complete this file before the experiment begins and not modify the information while the experiment isrunning.

Note: Any experiment is promptly invalid if the definition of the experiment changes after the experiment hasbegun.

1. If you have administrator access to your web or application servers, navigate to the Sensor installation folder onany Sensor machine in your web cluster to access the TestExperiment.xls file. If you do not have administratoraccess, contact your Adobe account manager to request the TestExperiment.xls file.

2. Open the TestExperiment.xls file (you can rename this file if desired) and complete the following fields:

DescriptionField

A descriptive name for the experiment. Each experiment name must be unique and cannotcontain spaces.

Experiment

Experiment names are used when displaying the results of experiments in Insight. Thenames appear as the first half of the element names in the controlled experiment dimension.The second half of the element name is the group name from the Group field in this file.Each group is named in the following format using the experiment name followed by thegroup name:

ExperimentName.Group Name

For example: New_Homepage.Control

The date and time that you want the experiment to begin. If you do not enter values, theexperiment begins immediately after the file is deployed.

Start

Format: MM/DD/YYYY H:MM

Note:

• If you leave the start and stop times empty, the experiment runs indefinitely.

• You can predefine start and stop times well in advance; therefore, you can configureall of your experiments for the next year at once if desired.

• Start and stop times are based on the system time of the web server. If that clockchanges for any reason, your experiment may start or stop unexpectedly.

• If you would like to add an experiment as a configuration file entry but do not wantthe experiment to run in the near future, you can comment out the experimentinformation using the number sign “#” or define start and stop times in the past.


DescriptionField

The date and time that you want the experiment to end.When the stop date and time occurs,Sensor will stop sending the cookie values identified as a test group to the test URIs andwill send all cookies to the control group URIs.

Stop

Format: MM/DD/YYYY H:MM

Note: See the notes for the Start field.

A descriptive name for each group of visitors in the experiment. Group names cannot containspaces.

Group

Group names are used when displaying the results of experiments in Insight. For moreinformation, see the Experiment field description.

A control group can be implicitly or explicitly defined based on the value entered in thePercentage field.

Note: To meet the number of visitors needed during the defined time period for theexperiment to be statistically valid, you may need to either decrease the confidencelevel or increase the time period. For example, if your time frame is five days, yourconfidence level is 98%, and your number of visitors needed exceeds the numberexpected for that time period, you need to either increase the time period or decreasethe confidence level until the number of visitors expected exceeds the number neededto run a statistically valid experiment.

The percentage of website visitors to include in each defined group. These values can beexpressed as either percentages or decimal values. In addition, both values must be eithergreater than or less than one.

Percentage

For example:

33.3% and 66.7%

.99 and .01

If the sum for all groups is less than 100, the undefined excess defaults to a control group.

The URI of the content to be remapped, followed by $. This value is case-sensitive.Original URL

Format: index.asp$

Original URIs can be specified using a dollar sign ($) at the end of the URI to denote thatan exact match of the file name is required. For example, the expression/product/product_view.asp$ matches that exact page only, while /product matchesany page in the /product directory and could be used to remap that entire sub-tree. OriginalURL entries that do not specify the $ character at the end of the file name are ignored bythe experiment unless the ExpPartialMatch parameter has been set to “on.” For moreinformation about this parameter, see Modifying the ExpPartialMatch Parameter (Optional).


DescriptionField

Note:

The controlled experiment functionality ignores any query strings appended to the URIstem. For example, the page

/product/product_view.asp?productid=53982 is not a valid URI, but thepage /product/product_view.asp is a valid URI.

The URI of the alternate content.Remapped URL

Format: index2.asp

Note: See the notes for the Original URL field.

The following is an example of a completed TextExperiment.xls spreadsheet:

Note: Do not modify the column positions in the spreadsheet.

This example indicates that the “New_Homepage” experiment starts on June 1, 2006, ends on June 30, 2006,and contains a control group with 50% of the visitors and a test group with 50% of the visitors, who see differentcontent for one URI.

Note: Although the sample file above has an explicit control group defined, it is not necessary to explicitlydefine a control group — the experiment automatically creates the control group. If the sum of thepercentages for all groups in an experiment is less than 100%, an implicit control group is assigned to usersthat do not fall into one of the explicit groups.

3. To insert comments to provide additional information about specific experiments, begin the cell with a numbersign (#) and follow with your comments. Comments can be inserted anywhere in the file.

4. After you have completed the variables in the experiment configuration spreadsheet, save the changes, thensave the file in tab-delimited text format (*.txt) using the name that you specified in the ExpFile parameter inthe Sensor configuration file. See Modifying the ExpFile Parameter.

The following is an example of an experiment configuration text file:


Note: Because of the tabs required in this file, do not edit the experiment configuration text file by hand.If you need to make changes to the file, make the changes in the experiment configuration Excel file andre-save the file as a tab-delimited text file.

If you have defined Start and Stop times, there is no reason to ever delete an experiment from the experimentconfiguration file. Keeping all of your experiments listed in the experiment configuration file is actually a good wayto keep a record of how you defined each of your experiments.

Deploying the Configuration File and Test Content

You must deploy the experiment configuration file to each machine in your web cluster that is running a Sensor andserving the pages involved in the experiment.You can do so using either a manual procedure or your existing contentmanagement system.

To deploy your test content

• On each application or web server running a Sensor that is serving pages involved in the experiment, use yourexisting publishing process to deploy the test content to the appropriate location.

For example, if you want to publish the test group page index2.asp to the test folder for your website(mysite.com), you would publish the file to www.mysite.com/test.

Note: Do not link to any of your test files directly from a page on your website. Doing so invalidates yourtest results and your index scores.

To deploy your experiment

• On each application or web server running a Sensor that is serving pages involved in the experiment, place theexperiment configuration text file in the directory that you specified in the ExpFile parameter in the Sensorconfiguration file. See Modifying the ExpFile Parameter.

Sensor randomly selects website visitors for each group based on the percentages that you defined in the file andserves the test or control group content to them as appropriate.


Validating the ExperimentAfter you have deployed your experiment, you should validate that the experiment is working properly.

As discussed in Modifying the ExpCookieURL Paramter (Optional), the page specified in the ExpCookieURLparameter in the Sensor configuration file can be used to place yourself in a specific experiment group.

The default virtual page is /setcookie.htm, but you must use the value that you set in the ExpCookieURLparameter.

Requesting the Test Page

To test a specific experiment group for your website, your browser must be configured to accept cookies and youmust not already have a cookie for this website.

Each time you want to test a new group, make sure to clear your cookies for the website.

To place yourself into a specific group within a specific experiment, request the test page with a query string in thefollowing form:

http:// <sitename/?Experiment Name=Group Name>

For example:

http://www.omniture.com/setcookie.htm?New_Homepage=index2

When the virtual URL request is sent to the server, Sensor identifies you as a member of the specified group withinthe specified experiment and then redirects you to the root of the website.You now can navigate to the appropriatelocation on the website to validate whether the correct content displays for that experiment and group.

If you were to type the following into your browser, the browser would display the home page of the website andplace you into the index2 group within the New_Homepage experiment:

http://www.omniture.com/setcookie.htm?New_Homepage=index2

When visitors in the index2 group request the home page, the “Request a Demo” graphical link displays higher onthe page, as in the following graphic:

16Validating the Experiment

17Validating the Experiment

Viewing the ResultsYou can view the results of an experiment in a visualization within Insight.

Configuring the Dataset

Before you can view controlled experiment data in Insight, you must modify your dataset configuration files to includea field and an extended dimension to contain information about your experiments, which enables you to view theresults of an experiment in a table within Insight.

To do so, you must complete the following steps:

• Modifying Log Processing.cfg

• Modifying Transformation.cfg

Note: Editing these configuration files causes your dataset to begin reprocessing immediately. It is importantto schedule this for a time that has the least effect on the users of your system. For more information aboutreprocessing a dataset, see the Server Products Installation and Administration Guide as well as the DatasetConfiguration Guide.

Modifying Log Processing.cfg

You must add the x-experiment field to the Log Processing.cfg file, which is used to create an extendeddimension.

See Modifying Transformation.cfg.

To modify Log Processing.cfg

1. In Insight, open the Profile Manager by right-clicking within a workspace and clicking Admin > Profile Manager,or by opening the Profile Management workspace on the Admin tab.

2. In the Profile Manager, click Dataset to show its contents.

3. Right-click the check mark next to Log Processing.cfg and click Make Local. A check mark for this fileappears in the User column.

4. Right-click the newly created check mark and click Open > in Insight. The Log Processing.cfg windowappears.

5. Click Fields to show its contents.

6. Right-click the last field in the current list and click Add new > Field.

7. Type x-experiment in the newly created field, as shown in the following example:

18Viewing the Results

8. Right-click (modified) at the top of the window and click Save.

9. In the Profile Manager, right-click the check mark for Log Processing.cfg in the User column, then clickSave to > <profile name> to save the locally made changes to the working profile.

Note: The dataset begins reprocessing immediately.

For more information about Log Processing and Fields see the Dataset Configuration Guide.

Modifying Transformation.cfg

Now that the x-experiment field is available, you must create an extended dimension to include the x-experimentfield in your dataset, which allows you to view your results in Insight.

To do so, you must add a new dimension to the Transformation.cfg file.

If you plan to run multiple experiments, you also must add a new Split transformation to the Transformation.cfgfile. This Split transformation separates the different experiment and group names so that the information is easierto interpret. To avoid reprocessing your data again if you were to need to add additional experiments at a later date,Adobe recommends that you add the Split transformation even if you are not currently planning to run multipleexperiments.

The following procedure includes the creation of both the new Split transformation and the extended dimension. Ifyou do not want to add the Split transformation, simply skip steps 5–7.

To modify Transformation.cfg

1. In Insight, open the Profile Manager by right-clicking within a workspace and clicking Admin > Profile Manager,or by opening the Profile Management workspace on the Admin tab.

2. In the Profile Manager, click Dataset to show its contents.

3. Right-click the check mark next to Transformation.cfg and click Make Local. A check mark for this fileappears in the User column.

4. Right-click the newly created check mark and click Open > in Insight. The Transformation.cfg windowappears.

5. Click Transformation to show its contents.

6. Right-click Transformations and click Add new > Split.

7. Complete the new split on comma transformation as shown in the following example:


Note: You can enter any value in the Name field.

8. Right-click Extended Dimensions and click Add new > ManyToMany.

9. Complete the new dimension as shown in the following example:

Note:

• You can enter any value in the Name field.

• If you did not include the Split transformation, you must type “x-experiment” in the Input field.

10. Right-click (modified) at the top of the window and click Save.

11. In the Profile Manager, right-click the check mark for Transformation.cfg in the User column, then clickSave to > <profile name> to save the locally made changes to the working profile.

Note: The dataset begins retransforming immediately.

For more information about Transformation.cfg and extended dimensions, see the Dataset ConfigurationGuide.


Viewing the Experiment Results

After you have added the new field to Log Processing.cfg and created the new Split transformation and extendeddimension, you can view the new extended dimension that you created as soon as the Fast Input stage of datareprocessing has finished.

This dimension, by default, displays the number of sessions for each of your experiment groups.

To view the experiment dimension

• Within any workspace in Insight, open a table with the experiment dimension that you created.

The experiment dimension elements, which represent each experiment you are currently running and each groupwithin each experiment, display with the current number of sessions for each group. Each group is named in thefollowing format using the experiment name followed by the group name:

Experiment Name.Group Name

For example: New Homepage.Control

The following table shows the Controlled Experiment Groups dimension that was created in Transformation.cfgand each of the experiments and their groups.

The New Homepage experiment is shown at the bottom of the table with its two groups: Control and index2.

You now can use the experiment dimension and any relevant metrics to explore and interpret the experiment results,as well as create useful reports detailing those results.

Evaluating the Experiment

After running the experiment until the required minimum number of visitors have participated in the experiment, youcan be assured of sufficient statistical confidence to evaluate the results of your experiment.

Using Insight, compare whichever metrics or key performance indicators were defined as part of the hypothesis todetermine whether the experiment was a success (that is, the hypothesis was validated with the specified confidence.)

In our example experiment, our hypothesis is proven correct if the Visitor Conversion increases by at least 1.5%,which is the success criterion we defined earlier.

The following workspace example shows that the Conversion for the index2 test group was actually 1.8% higherthan for the control group, proving our hypothesis.


• Summarizing the Experiment Results

• Taking Action Based on the Results

• Monitoring Your Actions

Summarizing the Experiment Results

Using Insight, you can create detailed reports to summarize and illustrate the results of your experiment.

You then can use your reports, as shown in the following example, to make recommendations based on the results,which are backed up by the visual information you have provided in your reports:


Taking Action Based on the Results

After the results are clear, you are ready to act on those results by making production-level changes to the testedpages, applying these same changes to other areas of your website, and making sure to completely document thetest, its results, and the changes that you have made.

Monitoring Your Actions

After the controlled experiment is complete and you have implemented the appropriate changes, make sure tocontinue to monitor the changes that you made by, for example, viewing validation metrics, creating control charts,and providing dashboard metrics.

Always be prepared to re-test your hypothesis if you think the changes that you tested and made are not bearingout the original results.


Experiment Design SpreadsheetThis file functions not only as a worksheet but also as a record of your decisions about the experiment.

If you need help designing your experiment, you can use the experiment design spreadsheet (named VS ControlledExperiment Design.xls by default) provided by Adobe.

The experiment design spreadsheet can provide useful statistical inferences only when the metric in question isdefined as a percentage of visitors that meet some criteria. That is, it is useful only when testing a visitor-basedmetric hypothesis.

To design your experiment using the experiment design file

1. If you have administrator access to your web or application servers, navigate to the Sensor installation folder onany Sensor machine in your web cluster. If you do not have administrator access, contact your Adobe accountmanager to request the file.

2. Open the VS Controlled Experiment Design.xls file. (You can rename this file if desired.)

The spreadsheet on the following page is an example of how you would complete the spreadsheet when preparingto test the example hypothesis used throughout this guide.

24Experiment Design Spreadsheet

3. Enter text or values for all of the fields in blue in this file, which are described in the following table.The calculatedfields are defined in the second table.

Fields to Define

SpecifyIn this field…

A descriptive name for your experiment.Experiment Title

A textual description of the experiment.Experiment Description

The name of the metric on which the experiment is based.Metric Being Studied

Example: Visitor Conversion

The definition of the metric on which the experiment is based.Metric Definition

Format: Visitors[X]/Visitors

Example: Visitors[URI='conversionpage.asp']/Visitors

The date and time you want the experiment to begin.Intended Start Time

The date and time you want the experiment to end.Intended End Time

(Optional) The dimension name and element set or range by which you want tofurther segment the dataset.

Applicable Selections

The URIs involved in your hypothesis.You define the current URIs for the controlgroup and the alternate URIs that you have created or will create for the test group(s).

Experiment URIs

Heading for the metric values that you expect for your website.Expected Metrics forApplication Selections

The average number of visitors to your website per day.Average Visitors per Day

The average visitor conversion rate for your website.Visitor Conversion

Heading for how the metric values should be compared.Experiment Will Determineif the metric name for theTest Groups is …


SpecifyIn this field…

Set this field to True if you want the ability to conclude that the test group’s metricincreased during the experiment. Set this field to False to reduce the number ofvisitors needed to draw conclusions. Adobe recommends that you set it to True.

Greater Than The ValueFor the Control Group?

Set this field to True if you want the ability to conclude that the test group’s metricdecreased during the experiment. Adobe recommends that you set it to True.

Less Than The Value Forthe Control Group?

The percentage by which you want the metric for the test group to be higher or lowerthan that for the control group.

By at Least (DetectionLevel)

The desired confidence level for the test group values. The confidence leveldetermines the number of false positives to measure the probability that the statedexpectation is true.

With a Confidence Level ofat Least

The desired power level for the test group values. The power level determines thenumber of false negatives.

and a Power Level of

Heading for the percent of visitors values.% of Visitors

Percent of visitors you want to include in the test group.You can play with thisnumber until the value in the Total (Usually 100%) field in the Visitors section is

Test Group

equal to or greater than the value in the Minimum Visitors Required (Test+ControlGroups) field, both of which are described in the following table.

Percent of visitors you want to include in the control group.Control Group

Any notes that you want to save for future reference.Other Design Notes

The remaining fields are calculated based on the values that you entered and are described in the following table.

Calculated Fields

DescriptionField

Heading for the metric values that you expect for your website.Expected Metrics forApplication Selections

This field is normally automatically calculated by the spreadsheet. It relies on theassumption that on most days the website receives many more new visitors than

Expected Visitors perPeriod

return visitors. If this is not the case, this cell’s calculation should be overwritten withthe actual number of visitors expected during the experiment.

The Z score for a false positive result. This is an intermediate statistical calculation.Calculated Z Score forType I Error

The Z score for a false negative result. This is an intermediate statistical calculation.Calculated Z Score forType II Error

Minimum number of visitors needed in your experiment to meet your specifiedconfidence level, power level, and Z score, expressed as a percentage of the valuein the Expected Visitors per Period field.

Minimum VisitorsRequired (Test+ControlGroups)


DescriptionField

Minimum number of visitors needed in your experiment to meet your specifiedconfidence level, power level, and Z score. This value must be less than or equal tothe value in the Total (Usually 100%) field in the Visitors section.

Minimum VisitorsRequired (Test+ControlGroups)

Minimum number of days you need to run your experiment to meet your specifiedconfidence level, power level, and Z score. This calculated number is subject to the

Minimum ExperimentTime (Days)

same issues as discussed in the Expected Visitors per Period field. In the case of awebsite with many returning visitors, the Minimum Experiment Time (Days) field isthe expected number of days it takes to see a number of unique visitors equal to thevalue in the Minimum Visitors Required field.

Heading for the visitors values.Visitors

Number of visitors needed in the test group.Test Group

Number of visitors needed in the control group.Control Group

Total number of visitors needed for the experiment. This value must be equal to orgreater than the value in the Minimum Visitors Required (Test+Control Groups) field.

Total (Usually 100%)

Percentage indicating that there is a chance equal to the specified confidence levelthat the measured value of the metric calculated for the test group will be within thispercentage of its true value.

Test Group Accuracy (atTarget Confidence Level)

Percentage indicating that there is a chance equal to the specified confidence levelthat the measured value of the metric calculated for the control group will be withinthis percentage of its true value.

Control Group Accuracy(at Target ConfidenceLevel)

Number of standard deviations a given value is from the test mean.Z Score (at TargetAccuracy)

The confidence level achieved for the experiment. The confidence level measuresthe probability of the stated expectation to be true.

Actual Confidence Level(at Target Interval)

The confidence interval achieved for the experiment, which provides an estimatedrange of values that is likely to include an unknown population parameter. This rangeis calculated from a given set of sample data.

Actual Interval (at TargetConfidence Level)

You need to look at the value in the Minimum Visitors Required (Test+Control Groups) field . . .

and compare it to the value in the Total field in the Visitors column.


For your experiment to be statistically valid, the value in the Total (Usually 100%) field must be equal to or greaterthan the value in the Minimum Visitors Required (Test+Control Groups) field.

Given the inputs provided, what the example worksheet shows you is that 10,475 visitors need to participate inthis experiment to achieve the entered 95% confidence rate (which is the minimum suggested confidence forany controlled experiment, although you can increase this number).The experiment as currently designed includes30,000 visitors, which is well over the minimum number of visitors required.

If you keep the number of days the same, you could increase the confidence level as long as the total numberof visitors continues the meet or exceed the required minimum.

4. Save the file for your records and then use the information from the file to configure the experiment using theexperiment configuration spreadsheet. For more information about this spreadsheet, see Configuring and Deployingthe Experiment.


adobe marketing cloud data workbench controlled experiments · data workbench controlled...

Documents