このページは http://www.slideshare.net/jcrobak/data-engineermeetup-201309 の内容を掲載しています。
Building a reliable pipeline of data ingress, batch computation, and data egress with Hadoop can ...
Building a reliable pipeline of data ingress, batch computation, and data egress with Hadoop can be a major challenge. Most folks start out with cron to manage workflows, but soon discover that doesn’t scale past a handful of jobs. There are a number of open-source workflow engines with support for Hadoop, including Azkaban (from LinkedIn), Luigi (from Spotify), and Apache Oozie. Having deployed all three of these systems in production, Joe will talk about what features and qualities are important for a workflow system.