Sample Code to Randomly Send Notifications for A/B Testing in Java

Paint QR Code ISO/IEC18004 in Java Sample Code to Randomly Send Notifications for A/B Testing
Listing 141 Sample Code to Randomly Send Notifications for A/B Testing
Quick Response Code Maker In Java
Using Barcode generation for Java Control to generate, create QR image in Java applications.
$notification1 = "sent <fb:name uid='$target' /> a " "<b><a href='"FB_APP_URL"&n=1'>Compliment</a></b>"; $notification2 = "sent <fb:name uid='$target' /> a " "<b>Compliment</b>" "<a href='"FB_APP_URL"&n=2'>Click here to send your own</a>"; if ( rand(1 , 2) == 1 ) { $notification = $notification1; else { $notification = $notification2; } $return = $facebook->api_client->notifications_send( array($target), $notification, 'user_to_user');
Create Barcode In Java
Using Barcode printer for Java Control to generate, create bar code image in Java applications.
14 Measuring Application Success
Recognize Bar Code In Java
Using Barcode decoder for Java Control to read, scan read, scan image in Java applications.
We ran this test for a day, and Table 141 shows the results
QR Code Creator In Visual C#
Using Barcode encoder for VS .NET Control to generate, create QR image in Visual Studio .NET applications.
Table 141 Option
QR Code Maker In Visual Studio .NET
Using Barcode encoder for ASP.NET Control to generate, create QR-Code image in ASP.NET applications.
Original New
Denso QR Bar Code Maker In Visual Studio .NET
Using Barcode maker for .NET framework Control to generate, create QR image in .NET framework applications.
Day 1 Results for Notification A/B Test Visitors
Denso QR Bar Code Creation In Visual Basic .NET
Using Barcode drawer for .NET Control to generate, create QR image in .NET applications.
51 50
Create Code39 In Java
Using Barcode maker for Java Control to generate, create Code 39 Full ASCII image in Java applications.
Clicks
Code 128C Creator In Java
Using Barcode drawer for Java Control to generate, create Code 128B image in Java applications.
Conversion Rate
Bar Code Printer In Java
Using Barcode creation for Java Control to generate, create barcode image in Java applications.
2% 12%
DataMatrix Drawer In Java
Using Barcode generation for Java Control to generate, create Data Matrix ECC200 image in Java applications.
It looks like the new notification is a winner, but we have to check that it is statistically significantThe more notifications we send out, the more likely that the results aren t just random noiseTo determine its significance, we can use the following calculation (a Pearson s Chi-Square test with a confidence of 95 percent):
Painting Bar Code In Java
Using Barcode encoder for Java Control to generate, create bar code image in Java applications.
N = Total number of clicks = 7 D = (Option 1 click Option 2 clicks) / 2 = 25
Leitcode Generator In Java
Using Barcode generator for Java Control to generate, create Leitcode image in Java applications.
D2 = 625
Draw Barcode In VS .NET
Using Barcode creator for Visual Studio .NET Control to generate, create barcode image in .NET framework applications.
The results are significant if D2 is greater than N 625 is not bigger than 7, so we don t have enough test results yetWe need to let this test run longerAfter another day, we get the results shown in Table 142
Code 39 Extended Generation In VB.NET
Using Barcode generation for .NET framework Control to generate, create Code-39 image in .NET applications.
Table 142 #Option
Code 128 Creator In Visual C#
Using Barcode creation for VS .NET Control to generate, create Code 128 Code Set C image in .NET framework applications.
Original New
Drawing UCC.EAN - 128 In Visual Studio .NET
Using Barcode maker for ASP.NET Control to generate, create EAN / UCC - 13 image in ASP.NET applications.
Day 2 Results for Notification A/B Test Visitors
ANSI/AIM Code 128 Reader In .NET
Using Barcode reader for Visual Studio .NET Control to read, scan read, scan image in VS .NET applications.
100 100
Printing UPC-A Supplement 2 In VS .NET
Using Barcode creation for ASP.NET Control to generate, create GTIN - 12 image in ASP.NET applications.
Clicks
Recognize DataMatrix In .NET Framework
Using Barcode scanner for .NET framework Control to read, scan read, scan image in .NET applications.
2 12
Conversion Rate
2% 12%
This time, the calculation is
N = 14 D=5
D2 = 25
25 is greater than 13, so our new notification is a good change and needs to be made permanent Note
Pearson s Chi-Square test is a statistical procedure for comparing the distribution of results versus expectations More recently, the G-test has becoming increasingly used as computers are able to handle the more time-consuming calculations
Metrics Dashboards
One downside to A/B testing is that if the new option isn t good, all the users that experience it are getting a bad experienceYou can mitigate this by reducing the percentage of users seeing the new option below 50 percent, but you then need to wait longer for the results to be statistically significant It also doesn t tell you why users picked one over the other; you have to conduct interviews to figure that out
Metrics Dashboards
A dashboard is a single place to reference all the metrics you have collected and possibly perform additional analysis or display trends If you are collecting your own custom metrics, we encourage you to build a simple dashboardThis section goes over the dashboards supplied by Facebook and the Kontagent Analytics package
Facebook Insights and the Metrics API
We already covered using Facebook s Insights tool to monitor user response to messaging and Facebook messaging allocations in 10, Publisher, Notifications, and Requests However, this tool provides more data than this for application developers; moreover, much of the data that backs it is available through the Facebook application programming interface (API)This gives developers a great opportunity to develop their own custom monitoring solutions, dashboards, and A/B testing frameworks Looking at Metrics in Facebook Insights Figure 143 shows the Insights tool containing actual data for Active Users (Engagement) from a live application, currently with several hundred thousand active monthly users Notice that there are several tabs worth of data available to the developerThe following sections discuss not only the metrics on each tab, but also how to get them programmaticallyAdditionally, we explain how the metrics named in the Insights tool map to the names used to access them using the API The first tab, Usage, contains several different metrics that focus on user engagement with the application and that application s interaction with the Facebook Platform Table 143 describes them in detailThe first column shows the name of the metric as shown in the Insights tool, the second is the name of the metric as it s used when accessing it the Facebook API, which we cover later Granularity explains the number of different values that are available for a given metric type
Table 143 Facebook Insights: Usage Metric Metric (API)
active_users
Metric (Insights)
Active Users (Engagement)
Description
Total number of active users for the application
Granularity
Last 1-,7-, and 30 days
14 Measuring Application Success
Table 143
Facebook Insights: Usage Metric Metric (API)
unique_api_calls
Metric (Insights)
Active Users (API)