WEBVTT 00:00.000 --> 00:14.560 So, welcome. My name is Soryne. I'm the CTO of a small company called OmniConvert. We run a 00:14.560 --> 00:22.400 SAS, which essentially is an experimentation platform. You can run AB tests, overlays like 00:22.400 --> 00:30.000 top-ups, surveys, or with segmentation, and of course performance tracking. For this we use 00:30.000 --> 00:38.640 events, we call them goals. I'm here today to discuss about the migration of our database. We 00:38.640 --> 00:47.360 migrated from an old version of MongoDB towards a new version of TADB. And just to have a context 00:47.360 --> 00:59.920 of our size, we, on average, run like 18, 19k RPM during the day with peaks up to 25k. And 00:59.920 --> 01:07.760 at the moment of the migration we had like 3.5 billion records in Mongo 80% of them were in the 01:07.760 --> 01:16.160 two largest collections. So, today I would like to briefly discuss about our context, why we chose 01:16.160 --> 01:25.040 TADB. And then go through the migration phase, starting from POC up until the actual project completion. 01:25.920 --> 01:32.320 And discuss about the end user perceptions as well. So, as I mentioned, we had a really old MongoDB 01:32.320 --> 01:40.000 it was a Mongo 3.4. We had lots of issues with it, but first and foremost we had terrible 01:40.080 --> 01:46.080 query performance. So, whenever running reporting, it was almost dead slow. And we had no 01:46.080 --> 01:53.920 easy way to scale out, neither in terms of multiplying the numbers of the number of servers to 01:53.920 --> 02:02.160 process nor the capacity to be stored. And of course, being sold, we didn't benefit from security 02:02.240 --> 02:12.640 patching anymore. And we weren't able to do like migration or like an upgrade to a newer version 02:12.640 --> 02:18.640 of Mongo because it was sold and there was a deadlock between the version of the database and the 02:18.640 --> 02:27.920 version of the driver. So, we were obliged to upgrade them at the same time, effectively rendering 02:28.160 --> 02:38.000 it as a migration. So, we decided to take a look at other systems as well. And we had some criteria. 02:38.560 --> 02:46.560 First and foremost, we wanted to solve our issues. So, we have better query performance, 02:46.560 --> 02:52.880 better reporting performance. Obviously, to be highly available, we wanted to solve the horizontal 02:52.880 --> 03:02.240 scalability as well, scaling out. We wanted the budget of $1,500 per month, but just for context, 03:02.240 --> 03:10.240 we're paying like $1,300 per month now. So, this was not really feasible goal. But we learned that. 03:11.680 --> 03:18.640 And very important, we are very, very small teams. So, we cannot afford to have like dedicated 03:19.200 --> 03:26.160 people maintaining the database. So, it was crucial for us to have it deployed and maintained by 03:26.160 --> 03:35.200 developers. I'm not going to get into the details, but basically from that list, TAB was the only 03:35.200 --> 03:42.640 one who picked all the boxes. Sonny presented earlier, the architecture of the TAB, so I'm not going 03:42.640 --> 03:53.040 to go into this to deep, but essentially, there's load balancer, which queries some TAB servers, 03:53.040 --> 04:01.040 which are stateless, so they don't store any data. And there are two types of storage TAB and TAB flash, 04:01.040 --> 04:10.080 sorry. TAB is row oriented while TAB flash is column oriented. And there's the placement drivers 04:10.080 --> 04:17.840 who decide where the data should sit and rebalance data for performance and also size, 04:19.120 --> 04:28.160 whenever scaling out, for example. So, before doing the actual migration, obviously we did the 04:28.160 --> 04:34.960 POC. We wanted to test two things. How fast does it run an individual query and how good does 04:34.960 --> 04:43.680 it scale in terms of loading? Load testing, sorry. I have here a very simple query and obviously 04:43.680 --> 04:50.800 for MongoDB, it was the equivalent run. It was run on one of the largest collections and as you 04:50.800 --> 04:58.560 can see, the difference is absolutely huge between the time running this simple query. This is 04:58.560 --> 05:05.040 the simple query but grouped by day, so we also have some charts showing the variance by day. 05:05.600 --> 05:13.840 And again, TAB basically crushed the MongoDB that we had. So again, this is a MongoDB 3.4, 05:13.840 --> 05:20.640 it's not the latest version. We also wanted to do like a full experiment cache. We have some 05:20.640 --> 05:26.560 predefined periods for which we cache various data like past week, past month, past year, 05:26.720 --> 05:33.520 something like that. And again, we saw more or less the same factor of increase in speed. 05:34.240 --> 05:41.120 And whenever it came to running a custom report on a custom period like runtime query, 05:42.000 --> 05:49.440 so no cache beforehand, we have even bigger of a difference here. So, this was basically a 05:49.440 --> 05:57.280 no brainer to try to switch to TAB and move forward. So, we went on to the stress testing. 05:58.720 --> 06:04.560 Before we scaled out the stress test, we wanted to determine what would be the optimal parameters 06:04.560 --> 06:12.000 for running a single container, so we can then scale this out. And we run the application which 06:12.000 --> 06:19.040 in just data in PHP, I didn't mention that. Surprisingly enough, we found out that running 06:19.440 --> 06:29.120 free workers per node, per container, sorry, was the best efficiency ratio. And we managed to 06:29.120 --> 06:37.760 have a pretty decent response time as well. So, we went on to try and run this at scale, 06:37.760 --> 06:44.240 covering like four to five times the traffic that we had on a daily basis. We ran the test with 06:44.240 --> 06:52.400 Grafana K6. And these are the results from new relics. So, basically, it shows that for a longer period 06:52.400 --> 07:02.480 of time, I think it was an hour here. We run on average of the target for the test and also having 07:03.040 --> 07:11.200 a very low latency, like 15 milliseconds or so, which is basically the same time as we have in 07:11.200 --> 07:20.320 production. So, this was a success. We decided to move on and the plan was pretty clear. We had to 07:20.320 --> 07:29.680 rewrite all reporting queries from MongoDB to my SQL to SQL, sorry, compatible. And then also 07:29.680 --> 07:38.320 the queries which inserted the data and updated some data as well. We decided to have a system that 07:38.320 --> 07:45.200 would run both of the databases in parallel. So, we are sure that we don't lose any data and 07:45.200 --> 07:53.120 the data is correct within TIDB. So, we created the flip-able switch. First, we created some scripts 07:53.120 --> 08:03.760 that migrated the data and this was done. And basically, it was creating insert batches for being 08:03.840 --> 08:13.040 run for them to be run on the import into TIDB. We did this up until a certain time stamp and 08:13.040 --> 08:21.680 then we deployed the feature which essentially has started to write in both databases. And then 08:21.680 --> 08:27.280 we, this is for phase two, we basically covered the gap in between those two time stamps. 08:27.440 --> 08:34.800 For approximately two months, we stayed at phase three which was monitoring for differences and 08:34.800 --> 08:41.440 we ironed out the queries and fixed all the bugs. And once we were confident, we basically 08:41.440 --> 08:48.640 inverted the two databases making the TIDB database, our primary database and Mongo was just 08:48.640 --> 08:55.040 kept for backup purposes. And after one additional month, we dropped that as well. This was like 08:55.440 --> 09:03.520 eight months ago. So, we were very happy with what TIDB has offered this. 09:04.400 --> 09:10.880 A few things that we learned, there are a few flags that can pretty dramatically impact performance. 09:10.880 --> 09:16.080 They're very well documented in the TIDB documentation. So, no issue there. 09:17.040 --> 09:27.360 And another thing is that joins will not go on as expected. So, at first we were so excited 09:27.360 --> 09:35.760 about the performance that we have loosely written some queries to have like multiple joins, 09:35.760 --> 09:44.160 but this did not perform very well at scale. So, we tried to rewrite and succeeded a bit afterwards. 09:45.040 --> 09:54.480 And another thing to mention is the fact that you have a multi master setup. So, whenever 09:54.480 --> 10:01.280 inserting rows, you need to be careful whenever having like a select plus insert combinations. 10:04.080 --> 10:10.320 Another thing to mention is the fact that TI flash, the column oriented storage is largely 10:10.320 --> 10:18.720 flatline in terms of its response time for a query on a large data set while the row 10:19.440 --> 10:30.640 based storage TAKV scales accordingly as expected. I'm very happy to say that this was a successful 10:30.640 --> 10:37.840 project. It was a very bold project done in three people team. And we completed it in less than 10:37.840 --> 10:48.160 six months from start to finish. We had no major issues and we also tested the highly availability 10:48.160 --> 10:57.360 of the system because we changed some servers throughout. So, it really performed very well as 10:57.360 --> 11:02.480 advertised. And also, their support was very, very prompt. Thank you, Daniel. 11:02.480 --> 11:09.120 Thank you.