![](\"https:\/\/i0.wp.com\/softwareengineeringdaily.com\/wp-content\/uploads\/2018\/05\/ZhenxiaoLuo.jpeg?resize=175%2C175&ssl=1\")
<\/p>\n<\/p>\n
Originally published May 24, 2018<\/em><\/p>\n When a user takes a ride on Uber, the app on the user\u2019s phone is communicating with Uber\u2019s backend infrastructure, which is writing to a database that maintains the state of that user\u2019s activity. This database is known as a transactional database or \u201cOLTP\u201d (online transaction processing). Every active user and driver and UberEATS restaurant is writing data to the transactional data store.<\/span><\/p>\n Periodically, that data is copied from the transactional data system to a different data storage system, where that data can be queried for large-scale data analysis. For example, if a data scientist at Uber wants to get the average amount of miles that a given user rode in February, that data scientist would issue a query to the analytical data cluster.<\/span><\/p>\n Uber uses the Hadoop distributed file system (HDFS) to store analytical data. On this file system, Uber has a version history of all of the company\u2019s useful historical data. Trip history, rider activity, driver activity–every data point that is in the transactional database–but in a file format that is easier to query for large scale processing. This file format is known as Parquet.<\/span><\/p>\n Data scientists, machine learning engineers, and real-time application developers all depend on the massive quantities of data that are stored in these Parquet files on Uber\u2019s HDFS cluster. To simplify the access of that data by many different clients, Uber uses Presto, an analytical query engine originally built at Facebook. <\/span><\/p>\n Presto translates SQL queries into whatever query language is necessary to access the underlying storage medium–whether that storage system is an ElasticSearch cluster, a set of Parquet files, or a relational database. Presto is useful because it simplifies the relationship between data engineers and the application developers who are building on top of the data engineering infrastructure.<\/span><\/p>\n In today\u2019s show, Zhenxiao Luo joins to give an end-to-end description of Uber\u2019s data infrastructure–from the ingest point of the OLTP database to the OLAP data storage system on HDFS, to the wide range of data systems and applications that run on top of that OLAP data.<\/span><\/p>\n Originally published May 24, 2018 When a user takes a ride on Uber, the app on the user\u2019s phone is communicating with Uber\u2019s backend infrastructure, which is writing to a database that maintains the state of that user\u2019s activity. This database is known as a transactional database or \u201cOLTP\u201d (online transaction processing). Every active user<\/p>\n","protected":false},"author":3,"featured_media":7913,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[1363,1081,2143,14],"tags":[2120,61,137,2119,343,256,40,2118],"jetpack_publicize_connections":[],"acf":[],"yoast_head":"\n<\/h2>\n","protected":false},"excerpt":{"rendered":"