{"id":4555,"date":"2018-02-20T02:00:43","date_gmt":"2018-02-20T10:00:43","guid":{"rendered":"http:\/\/softwareengineeringdaily.com\/?p=4555"},"modified":"2021-11-01T01:37:17","modified_gmt":"2021-11-01T08:37:17","slug":"kafka-design-patterns-with-gwen-shapira","status":"publish","type":"post","link":"https:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/","title":{"rendered":"Kafka Design Patterns with Gwen Shapira"},"content":{"rendered":"<p><img data-attachment-id=\"2475\" data-permalink=\"https:\/\/softwareengineeringdaily.com\/2016\/04\/19\/googles-container-management-brendan-burns\/brendan-burns\/\" data-orig-file=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2016\/04\/brendan-burns.jpg?fit=175%2C175&amp;ssl=1\" data-orig-size=\"175,175\" data-comments-opened=\"0\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"brendan-burns\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2016\/04\/brendan-burns.jpg?fit=175%2C175&amp;ssl=1\" data-large-file=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2016\/04\/brendan-burns.jpg?fit=175%2C175&amp;ssl=1\" decoding=\"async\" loading=\"lazy\" class=\"alignright size-full wp-image-2475\" style=\"border-radius: 50%; border: 1px solid #000000; max-width: 175px; max-height: 175px;\" src=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2018\/02\/gwen-shapira-lg-200x200.png?resize=175%2C175&#038;ssl=1\" width=\"175\" height=\"175\" data-recalc-dims=\"1\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It\u2019s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Today\u2019s guest is Gwen Shapira, a product manager at Confluent. Confluent is a company that was started by the creators of Apache Kafka&#8211;Jay Kreps, Neha Narkhede, and Jun Rao, who have all been on the show in previous episodes. In those shows, we discussed the inner workings of Kafka. This episode is more about practical use cases and design patterns.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Gwen explores a few use cases. First, reconciling data between different servers. A massive, international, multi-user game like World of Warcraft needs to keep its users in sync despite the fact that those users are pinging different server locations. Kafka can help reconcile data between the multiple servers. This discussion reminded me of the <\/span><a href=\"https:\/\/www.softwaredaily.com\/#\/post\/5a2e5b24bc9ac50004477f12\/Scalable-Multiplayer-Games-with-Yan-Cui\"><span style=\"font-weight: 400;\">awesome show we did with Yan Cui<\/span><\/a><span style=\"font-weight: 400;\"> about scalable multiplayer games.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Other examples we discussed include log management, data enrichment, and large scale analytics. It was a great conversation and I think you will enjoy it as well.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to <\/span><a href=\"mailto:internships@softwareengineeringdaily.com\"><span style=\"font-weight: 400;\">internships@softwareengineeringdaily.com<\/span><\/a><span style=\"font-weight: 400;\">. We\u2019d love to hear from you.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If you haven\u2019t seen what we are building, check out <\/span><a href=\"http:\/\/softwaredaily.com\"><span style=\"font-weight: 400;\">softwaredaily.com<\/span><\/a><span style=\"font-weight: 400;\">, or download the Software Engineering Daily app for <\/span><a href=\"https:\/\/itunes.apple.com\/us\/app\/software-engineering-daily\/id1253734426?mt=8\"><span style=\"font-weight: 400;\">iOS<\/span><\/a><span style=\"font-weight: 400;\"> or <\/span><a href=\"https:\/\/play.google.com\/store\/apps\/details?id=com.koalatea.thehollidayinn.softwareengineeringdaily&amp;hl=en\"><span style=\"font-weight: 400;\">Android<\/span><\/a><span style=\"font-weight: 400;\">. These apps have all 650 of our episodes in a searchable format&#8211;we have recommendations, categories, related links and discussions around the episodes. It\u2019s all free and also open source&#8211;if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at <\/span><a href=\"http:\/\/github.com\/softwareengineeringdaily\"><span style=\"font-weight: 400;\">Github.com\/softwareengineeringdaily<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It\u2019s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications. Today\u2019s guest is Gwen Shapira, a product<\/p>\n","protected":false},"author":3,"featured_media":4559,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"Kafka Design Patterns with Gwen Shapiro","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[1363,1081,14],"tags":[428,547,1891,1890,63,1721],"jetpack_publicize_connections":[],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Kafka Design Patterns with Gwen Shapira - Software Engineering Daily<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Design Patterns with Gwen Shapira - Software Engineering Daily\" \/>\n<meta property=\"og:description\" content=\"Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It\u2019s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications. Today\u2019s guest is Gwen Shapira, a product\" \/>\n<meta property=\"og:url\" content=\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/\" \/>\n<meta property=\"og:site_name\" content=\"Software Engineering Daily\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-20T10:00:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-11-01T08:37:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2018\/02\/confluent-platform.png?fit=1700%2C1000\" \/>\n\t<meta property=\"og:image:width\" content=\"1700\" \/>\n\t<meta property=\"og:image:height\" content=\"1000\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"SE Daily\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@software_daily\" \/>\n<meta name=\"twitter:site\" content=\"@software_daily\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"SE Daily\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/#article\",\"isPartOf\":{\"@id\":\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/\"},\"author\":{\"name\":\"SE Daily\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8\"},\"headline\":\"Kafka Design Patterns with Gwen Shapira\",\"datePublished\":\"2018-02-20T10:00:43+00:00\",\"dateModified\":\"2021-11-01T08:37:17+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/\"},\"wordCount\":356,\"publisher\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#organization\"},\"keywords\":[\"Analytics\",\"Confluent\",\"data streaming\",\"Gwen Shapira\",\"Kafka\",\"log management\"],\"articleSection\":[\"All Content\",\"Data\",\"Podcast\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/\",\"url\":\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/\",\"name\":\"Kafka Design Patterns with Gwen Shapira - Software Engineering Daily\",\"isPartOf\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#website\"},\"datePublished\":\"2018-02-20T10:00:43+00:00\",\"dateModified\":\"2021-11-01T08:37:17+00:00\",\"breadcrumb\":{\"@id\":\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/softwareengineeringdaily.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kafka Design Patterns with Gwen Shapira\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#website\",\"url\":\"https:\/\/softwareengineeringdaily.com\/\",\"name\":\"Software Engineering Daily\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/softwareengineeringdaily.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#organization\",\"name\":\"Software Engineering Daily\",\"url\":\"https:\/\/softwareengineeringdaily.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1\",\"width\":296,\"height\":139,\"caption\":\"Software Engineering Daily\"},\"image\":{\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/twitter.com\/software_daily\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8\",\"name\":\"SE Daily\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg\",\"caption\":\"SE Daily\"},\"description\":\"The SE Daily podcast.\",\"sameAs\":[\"https:\/\/softwareengineeringdaily.com\"],\"url\":\"https:\/\/softwareengineeringdaily.com\/author\/erikawho\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kafka Design Patterns with Gwen Shapira - Software Engineering Daily","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/","og_locale":"en_US","og_type":"article","og_title":"Kafka Design Patterns with Gwen Shapira - Software Engineering Daily","og_description":"Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It\u2019s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications. Today\u2019s guest is Gwen Shapira, a product","og_url":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/","og_site_name":"Software Engineering Daily","article_published_time":"2018-02-20T10:00:43+00:00","article_modified_time":"2021-11-01T08:37:17+00:00","og_image":[{"width":1700,"height":1000,"url":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2018\/02\/confluent-platform.png?fit=1700%2C1000","type":"image\/png"}],"author":"SE Daily","twitter_card":"summary_large_image","twitter_creator":"@software_daily","twitter_site":"@software_daily","twitter_misc":{"Written by":"SE Daily","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/#article","isPartOf":{"@id":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/"},"author":{"name":"SE Daily","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8"},"headline":"Kafka Design Patterns with Gwen Shapira","datePublished":"2018-02-20T10:00:43+00:00","dateModified":"2021-11-01T08:37:17+00:00","mainEntityOfPage":{"@id":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/"},"wordCount":356,"publisher":{"@id":"https:\/\/softwareengineeringdaily.com\/#organization"},"keywords":["Analytics","Confluent","data streaming","Gwen Shapira","Kafka","log management"],"articleSection":["All Content","Data","Podcast"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/","url":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/","name":"Kafka Design Patterns with Gwen Shapira - Software Engineering Daily","isPartOf":{"@id":"https:\/\/softwareengineeringdaily.com\/#website"},"datePublished":"2018-02-20T10:00:43+00:00","dateModified":"2021-11-01T08:37:17+00:00","breadcrumb":{"@id":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/softwareengineeringdaily.com\/2018\/02\/20\/kafka-design-patterns-with-gwen-shapira\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/softwareengineeringdaily.com\/"},{"@type":"ListItem","position":2,"name":"Kafka Design Patterns with Gwen Shapira"}]},{"@type":"WebSite","@id":"https:\/\/softwareengineeringdaily.com\/#website","url":"https:\/\/softwareengineeringdaily.com\/","name":"Software Engineering Daily","description":"","publisher":{"@id":"https:\/\/softwareengineeringdaily.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/softwareengineeringdaily.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/softwareengineeringdaily.com\/#organization","name":"Software Engineering Daily","url":"https:\/\/softwareengineeringdaily.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/","url":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1","contentUrl":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2022\/01\/cropped-logo-new.png?fit=296%2C139&ssl=1","width":296,"height":139,"caption":"Software Engineering Daily"},"image":{"@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/twitter.com\/software_daily"]},{"@type":"Person","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/822f06fe7d6f895baba29a9c0a3aa6c8","name":"SE Daily","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/softwareengineeringdaily.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b92f4cf3dc4d94f73834f83e2a22a372?s=96&d=retro&r=pg","caption":"SE Daily"},"description":"The SE Daily podcast.","sameAs":["https:\/\/softwareengineeringdaily.com"],"url":"https:\/\/softwareengineeringdaily.com\/author\/erikawho\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2018\/02\/confluent-platform.png?fit=1700%2C1000&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p7GuoD-1bt","_links":{"self":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/posts\/4555"}],"collection":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/comments?post=4555"}],"version-history":[{"count":0,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/posts\/4555\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/media\/4559"}],"wp:attachment":[{"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/media?parent=4555"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/categories?post=4555"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/softwareengineeringdaily.com\/wp-json\/wp\/v2\/tags?post=4555"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}