{"id":8071,"date":"2019-09-23T02:00:57","date_gmt":"2019-09-23T09:00:57","guid":{"rendered":"http:\/\/softwareengineeringdaily.com\/?p=8071"},"modified":"2021-10-06T07:17:21","modified_gmt":"2021-10-06T14:17:21","slug":"kafka-data-pipelines-with-robin-moffatt","status":"publish","type":"post","link":"https:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/","title":{"rendered":"Kafka Data Pipelines with Robin Moffatt"},"content":{"rendered":"<p><img data-attachment-id=\"2475\" data-permalink=\"https:\/\/softwareengineeringdaily.com\/2016\/04\/19\/googles-container-management-brendan-burns\/brendan-burns\/\" data-orig-file=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2016\/04\/brendan-burns.jpg?fit=175%2C175&amp;ssl=1\" data-orig-size=\"175,175\" data-comments-opened=\"0\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"brendan-burns\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2016\/04\/brendan-burns.jpg?fit=175%2C175&amp;ssl=1\" data-large-file=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2016\/04\/brendan-burns.jpg?fit=175%2C175&amp;ssl=1\" decoding=\"async\" loading=\"lazy\" class=\"alignright size-full wp-image-2475\" style=\"border-radius: 50%; border: 1px solid #000000; max-width: 175px; max-height: 175px;\" src=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2019\/09\/robin-moffatt.jpg?resize=175%2C175&#038;ssl=1\" width=\"175\" height=\"175\" data-recalc-dims=\"1\" \/><\/p>\n<p><span style=\"font-weight: 400;\">A new software product usually starts with a single database. That database manages the tables for user accounts and basic transactions.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">When a product becomes popular, the database grows in size. There are more transactions and more users. A company grows around that product, and the company starts to accumulate more data in different sources. Analytics systems, time series databases, and logging tools start to generate data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Moving this data around between systems starts to become complicated. Apache Kafka is often used as a system for moving data between these different systems, performing transformations, and generating aggregations and summaries of these large quantities of data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Robin Moffatt works at Confluent, and has written numerous articles about how to move data between systems and design effective workflows for data pipelines. Robin joins the show to talk about modern data platforms and databases, and the patterns for using Kafka to connect those systems to each other.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If you are interested in learning more about how companies are using Kafka, the <\/span><a href=\"https:\/\/kafka-summit.org\/events\/kafka-summit-san-francisco-2019\/\"><span style=\"font-weight: 400;\">Kafka Summit in San Francisco is September 30th &#8211; October 1st.<\/span><\/a><span style=\"font-weight: 400;\"> Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily.<\/span><\/p>\n<p>Sponsorship inquiries: <a href=\"mailto:sponsor@softwareengineeringdaily.com\">sponsor@softwareengineeringdaily.com<\/a><\/p>\n<h2><span style=\"font-weight: 400;\">Check out our active companies and projects:<\/span><\/h2>\n<ul>\n<li style=\"font-weight: 400;\"><a href=\"https:\/\/findcollabs.com\/\"><span style=\"font-weight: 400;\">FindCollabs<\/span><\/a><span style=\"font-weight: 400;\"> is a place to find collaborators and build projects. Find a project to work on<\/span><\/li>\n<li style=\"font-weight: 400;\"><a href=\"http:\/\/podsheets.com\"><span style=\"font-weight: 400;\">Podsheets<\/span><\/a><span style=\"font-weight: 400;\"> is an open source podcast hosting platform built with the learnings from Software Engineering Daily. Our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com. <\/span><\/li>\n<li style=\"font-weight: 400;\"><a style=\"font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;\" href=\"https:\/\/itunes.apple.com\/us\/app\/software-engineering-daily\/id1253734426?mt=8\">The SEDaily app for iOS<\/a><span style=\"font-weight: 400;\"> and <\/span><a style=\"font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;\" href=\"https:\/\/play.google.com\/store\/apps\/details?id=com.koalatea.sedaily\">Android<\/a><span style=\"font-weight: 400;\"> includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. <\/span><a style=\"font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen-Sans, Ubuntu, Cantarell, 'Helvetica Neue', sans-serif;\" href=\"https:\/\/softwareengineeringdaily.com\/subscribe\">Subscribe<\/a><span style=\"font-weight: 400;\"> for ad-free episodes.<\/span><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>A new software product usually starts with a single database. That database manages the tables for user accounts and basic transactions. When a product becomes popular, the database grows in size. There are more transactions and more users. A company grows around that product, and the company starts to accumulate more data in different sources.<\/p>\n","protected":false},"author":3,"featured_media":8082,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[1363,1081,2143,14],"tags":[1272,547,3218,3217,3216],"jetpack_publicize_connections":[],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Kafka Data Pipelines with Robin Moffatt - Software Engineering Daily<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Data Pipelines with Robin Moffatt - Software Engineering Daily\" \/>\n<meta property=\"og:description\" content=\"A new software product usually starts with a single database. That database manages the tables for user accounts and basic transactions. When a product becomes popular, the database grows in size. There are more transactions and more users. A company grows around that product, and the company starts to accumulate more data in different sources.\" \/>\n<meta property=\"og:url\" content=\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/\" \/>\n<meta property=\"og:site_name\" content=\"Software Engineering Daily\" \/>\n<meta property=\"article:published_time\" content=\"2019-09-23T09:00:57+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-10-06T14:17:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2019\/09\/Confluent.png?fit=700%2C400\" \/>\n\t<meta property=\"og:image:width\" content=\"700\" \/>\n\t<meta property=\"og:image:height\" content=\"400\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"SE Daily\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@software_daily\" \/>\n<meta name=\"twitter:site\" content=\"@software_daily\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"SE Daily\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/#article\",\"isPartOf\":{\"@id\":\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/\"},\"author\":{\"name\":\"SE Daily\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8\"},\"headline\":\"Kafka Data Pipelines with Robin Moffatt\",\"datePublished\":\"2019-09-23T09:00:57+00:00\",\"dateModified\":\"2021-10-06T14:17:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/\"},\"wordCount\":311,\"publisher\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#organization\"},\"keywords\":[\"Apache Kafka\",\"Confluent\",\"data pipelines\",\"Kafka Summit\",\"Robin Moffatt\"],\"articleSection\":[\"All Content\",\"Data\",\"Exclusive Content\",\"Podcast\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/\",\"url\":\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/\",\"name\":\"Kafka Data Pipelines with Robin Moffatt - Software Engineering Daily\",\"isPartOf\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#website\"},\"datePublished\":\"2019-09-23T09:00:57+00:00\",\"dateModified\":\"2021-10-06T14:17:21+00:00\",\"breadcrumb\":{\"@id\":\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/softwareengineeringdaily.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kafka Data Pipelines with Robin Moffatt\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#website\",\"url\":\"https:\/\/softwareengineeringdaily.com\/\",\"name\":\"Software Engineering Daily\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/softwareengineeringdaily.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#organization\",\"name\":\"Software Engineering Daily\",\"url\":\"https:\/\/softwareengineeringdaily.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1\",\"width\":296,\"height\":139,\"caption\":\"Software Engineering Daily\"},\"image\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/twitter.com\/software_daily\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8\",\"name\":\"SE Daily\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg\",\"caption\":\"SE Daily\"},\"description\":\"The SE Daily podcast.\",\"sameAs\":[\"https:\/\/softwareengineeringdaily.com\"],\"url\":\"https:\/\/softwareengineeringdaily.com\/author\/erikawho\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kafka Data Pipelines with Robin Moffatt - Software Engineering Daily","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/","og_locale":"en_US","og_type":"article","og_title":"Kafka Data Pipelines with Robin Moffatt - Software Engineering Daily","og_description":"A new software product usually starts with a single database. That database manages the tables for user accounts and basic transactions. When a product becomes popular, the database grows in size. There are more transactions and more users. A company grows around that product, and the company starts to accumulate more data in different sources.","og_url":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/","og_site_name":"Software Engineering Daily","article_published_time":"2019-09-23T09:00:57+00:00","article_modified_time":"2021-10-06T14:17:21+00:00","og_image":[{"width":700,"height":400,"url":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2019\/09\/Confluent.png?fit=700%2C400","type":"image\/png"}],"author":"SE Daily","twitter_card":"summary_large_image","twitter_creator":"@software_daily","twitter_site":"@software_daily","twitter_misc":{"Written by":"SE Daily","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/#article","isPartOf":{"@id":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/"},"author":{"name":"SE Daily","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8"},"headline":"Kafka Data Pipelines with Robin Moffatt","datePublished":"2019-09-23T09:00:57+00:00","dateModified":"2021-10-06T14:17:21+00:00","mainEntityOfPage":{"@id":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/"},"wordCount":311,"publisher":{"@id":"https:\/\/softwareengineeringdaily.com\/#organization"},"keywords":["Apache Kafka","Confluent","data pipelines","Kafka Summit","Robin Moffatt"],"articleSection":["All Content","Data","Exclusive Content","Podcast"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/","url":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/","name":"Kafka Data Pipelines with Robin Moffatt - Software Engineering Daily","isPartOf":{"@id":"https:\/\/softwareengineeringdaily.com\/#website"},"datePublished":"2019-09-23T09:00:57+00:00","dateModified":"2021-10-06T14:17:21+00:00","breadcrumb":{"@id":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/softwareengineeringdaily.com\/2019\/09\/23\/kafka-data-pipelines-with-robin-moffatt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/softwareengineeringdaily.com\/"},{"@type":"ListItem","position":2,"name":"Kafka Data Pipelines with Robin Moffatt"}]},{"@type":"WebSite","@id":"https:\/\/softwareengineeringdaily.com\/#website","url":"https:\/\/softwareengineeringdaily.com\/","name":"Software Engineering Daily","description":"","publisher":{"@id":"https:\/\/softwareengineeringdaily.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/softwareengineeringdaily.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/softwareengineeringdaily.com\/#organization","name":"Software Engineering Daily","url":"https:\/\/softwareengineeringdaily.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1","contentUrl":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1","width":296,"height":139,"caption":"Software Engineering Daily"},"image":{"@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/twitter.com\/software_daily"]},{"@type":"Person","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8","name":"SE Daily","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg","caption":"SE Daily"},"description":"The SE Daily podcast.","sameAs":["https:\/\/softwareengineeringdaily.com"],"url":"https:\/\/softwareengineeringdaily.com\/author\/erikawho\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2019\/09\/Confluent.png?fit=700%2C400&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p7GuoD-26b","_links":{"self":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/posts\/8071"}],"collection":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/comments?post=8071"}],"version-history":[{"count":0,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/posts\/8071\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/media\/8082"}],"wp:attachment":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/media?parent=8071"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/categories?post=8071"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/tags?post=8071"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}