We connect your dotsAds in the cloud Andrea Fiore Managing Director Copyrights 2012 DotAndMedia – www.dotandmedia.com
Dot&Ads is our multichannel ad-serving system used by severalleading publishers in Italy.
Dot&Ads delivers more than 7 Billions Imps per monthon a 24/7 up service
How we use AWS:We use EC2, Auto-scaling and Load balancing fordelivering ads along side our main infrastructure. We automatically absorb peaks through scripts monitoring our local farm, and change entries in the Route 53 (DNS), when needed, allowing more traffic towards AWS load balancers.
The Earthquake Case Study• Experiencing a peak of traffic of 60 Mbit/s (+66% of traffic) after an earthquake in Northern Italy (June, 2012)• Automatically half of that peak was diverted to Ec2 infrastructure avoiding service interruptions or delays in response
BigData and MapReduce• We produce about 8 billions log-entries that have to be processed to count distinct browsers and other variables;• We decided to use the Hadoop framework and the MapReduce to complete the task.
MapReduce Steps• In the Map phase data are parsed to find the key/value pairs matching your search;• Then a partition function will assign that pairs to the reducers trying to distribute them uniformly;• After a comparison phase in which the pairs are sorted, a Reduce function will iterate through the data producing zero or more results;• Finally an output writer will write the results (i.e. on a local storage or in the S3).
MapReduce Steps Amazon Elastic MapReduce Reporting UI MR Results S3 BucketCorporate Data center Log retrieving Application Logs S3 Bucket
How we use EMR• Our frontends save application logs on S3;• A script consolidate them in bigger files and move to the data repository bucket on S3;• Through a dedicated UI, our user can interrogate logs drilling down by several dimensions/filters;• Then a script prepare and execute a job on EMR;• When the job is completed another script will collect all the part-files produced by the MapReduce and add the column names.