WEBVTT 00:00.000 --> 00:07.000 Okay, can everybody hear me okay? 00:07.000 --> 00:09.000 Yeah, sounds like it. 00:09.000 --> 00:10.000 All right. 00:10.000 --> 00:12.000 I think we'll get started. 00:12.000 --> 00:13.000 Hi everybody. 00:13.000 --> 00:14.000 Thanks for coming. 00:14.000 --> 00:15.000 My name is Josh Lee. 00:15.000 --> 00:17.000 I am a developer at Altinity. 00:17.000 --> 00:23.000 We do hosting and services and support for Clickhouse, which is a little bit of a giveaway 00:23.000 --> 00:25.000 about what I'm going to be talking about here. 00:25.000 --> 00:30.000 The name of this talk is Tracing, a Cloud Native database, and Clickhouse is the Cloud 00:30.000 --> 00:33.000 Native database specifically that we're going to be talking about. 00:33.000 --> 00:35.000 So let's dive in. 00:35.000 --> 00:40.000 So before I can talk about Tracing, Clickhouse, we need to just level set on what 00:40.000 --> 00:42.000 Distributed Tracing is. 00:42.000 --> 00:46.000 So just really quickly, right, for anyone who may not be familiar. 00:46.000 --> 00:52.000 Distributed Tracing is a way of tracing the flow of a request through multiple services that 00:52.000 --> 00:53.000 is running on multiple hosts. 00:53.000 --> 00:59.000 So it's kind of like a stack trace, except across the network, right? 00:59.000 --> 01:02.000 And what you end up getting when you look at an individual trace, I'm sure many of you 01:02.000 --> 01:03.000 are familiar with this. 01:03.000 --> 01:07.000 You get a waterfall view, looks kind of like the waterfall view and like Chrome developers 01:07.000 --> 01:08.000 or something. 01:08.000 --> 01:12.000 And what's interesting about this is actually really the only thing that you need, like, 01:12.000 --> 01:18.000 the difference between a trace and a log, right, to get this from logs, you only need two 01:18.000 --> 01:19.000 extra pieces of information. 01:19.000 --> 01:23.000 You need the parent ID of the thing that came before it, the thing that called the 01:23.000 --> 01:24.000 thing that you're logging. 01:24.000 --> 01:26.000 And you need the end time, right? 01:26.000 --> 01:27.000 Logs usually have a start time. 01:27.000 --> 01:30.000 You include the end time, you can then calculate the duration. 01:30.000 --> 01:32.000 You include the parent ID. 01:32.000 --> 01:36.000 You get this nice tree, graph, structure, and we have tracing. 01:36.000 --> 01:37.000 Right? 01:37.000 --> 01:38.000 So it's not that complicated. 01:38.000 --> 01:40.000 It's just logging with a few extra steps. 01:40.000 --> 01:45.000 But it's very powerful in what it how it allows us to analyze distributed applications, 01:45.000 --> 01:48.000 like a distributed database, like Clickhouse. 01:48.000 --> 01:49.000 Okay. 01:49.000 --> 01:55.000 So, distributed tracing is awesome, but, you know, ten years ago, the only way to do it really 01:55.000 --> 02:00.000 was to build it yourself or to work with a vendor, like Dynatrice or something like that, 02:00.000 --> 02:01.000 right? 02:01.000 --> 02:03.000 And to open Solometry. 02:03.000 --> 02:04.000 Right? 02:04.000 --> 02:09.000 So, Open Solometry merged a couple of projects and is now sort of the fact to open source 02:09.000 --> 02:11.000 way to do distributed tracing. 02:11.000 --> 02:12.000 Along other things, right? 02:12.000 --> 02:13.000 You can use it for metrics and logs. 02:13.000 --> 02:17.000 But what the problem it really solves is distributed tracing. 02:17.000 --> 02:18.000 And it's a couple of things, right? 02:18.000 --> 02:23.000 So, Open Solometry is a format and it's a bunch of tools and libraries that sort of 02:23.000 --> 02:24.000 conform to that format. 02:24.000 --> 02:27.000 The thing that we're most interested in for this talk today is just the format. 02:27.000 --> 02:30.000 We're not going to be using that many of the tools, but just know that they're available. 02:30.000 --> 02:34.000 The part that's interesting to us, right, when we're talking about a trace, how do we actually 02:34.000 --> 02:35.000 get that parent ID, right? 02:35.000 --> 02:37.000 That thing that called the thing? 02:37.000 --> 02:41.000 And the way that we do it in Open Solometry is with the W3C trace context 02:41.000 --> 02:42.000 propagation specification. 02:42.000 --> 02:44.000 So, basically, we just have this header. 02:44.000 --> 02:46.000 And we just put the trace ID in the header. 02:46.000 --> 02:50.000 And as long as the thing that we're calling over HTTP understands how to interpret that header, 02:50.000 --> 02:53.000 it can use that trace ID when it's building its own trace spans, right? 02:53.000 --> 02:56.000 Each part of the trace is called a span. 02:56.000 --> 02:57.000 Okay. 02:57.000 --> 03:04.000 So, I mentioned, right, there's a whole bunch of stuff in Open Solometry that are tools that 03:04.000 --> 03:06.000 sort of conform to the specification. 03:06.000 --> 03:11.000 Like, you can build an entire observability stack using this. 03:11.000 --> 03:16.000 And you could even use Clickhouse as sort of the destination for some of that observability data, 03:16.000 --> 03:19.000 if you wanted to. 03:19.000 --> 03:23.000 So, let's talk about Clickhouse a little bit. 03:23.000 --> 03:24.000 What is Clickhouse? 03:24.000 --> 03:25.000 It's SQL compatible. 03:25.000 --> 03:26.000 It's an analytics database. 03:26.000 --> 03:28.000 It's really, really fast. 03:28.000 --> 03:31.000 Injusting millions of rows per second is no problem. 03:31.000 --> 03:32.000 Queryn overpetted weights of data. 03:32.000 --> 03:34.000 And under a second is no problem. 03:34.000 --> 03:39.000 And it's, I had massively scalable, but the test also claimed that. 03:39.000 --> 03:45.000 So, I'm just going to claim even more massively scalable. 03:45.000 --> 03:49.000 So, let's talk, I mentioned, right, you kind of use Clickhouse for observability. 03:49.000 --> 03:51.000 That's actually how I came into the world of Clickhouse. 03:51.000 --> 03:53.000 Working at an observability vendor. 03:53.000 --> 03:57.000 I'm only going to talk about this very briefly because tomorrow in the monitoring of observability 03:57.000 --> 03:58.000 Debt Room. 03:58.000 --> 04:01.000 I'm going to be talking about this much more in-depth using Clickhouse for observability. 04:01.000 --> 04:05.000 But kind of why it lends itself to the observability use case. 04:05.000 --> 04:07.000 Part of the reason is because of this architecture, right? 04:07.000 --> 04:12.000 So, the way Clickhouse works in part of the way that it's so scalable is when you're ingesting data. 04:12.000 --> 04:16.000 It gets written into these partitions that are, that can be pretty small, right? 04:16.000 --> 04:21.000 They can be small on memory just all of the data that came with a specific insert. 04:21.000 --> 04:27.000 And then, as that data is sitting on the disk, right? 04:27.000 --> 04:31.000 We have these little small parts that are very inefficient to read. 04:31.000 --> 04:34.000 What Clickhouse does is in the background it merges them. 04:34.000 --> 04:38.000 It keeps them ordered on disk, so that we can read them very, very fast if we're reading, 04:38.000 --> 04:41.000 using the same ordering that they were inserted with. 04:41.000 --> 04:46.000 And we eventually get much, much faster queries on our fully marched parts. 04:46.000 --> 04:51.000 So, this led itself to observability for a number of reasons. 04:51.000 --> 04:55.000 It's optimized for the right once read many use case of observability data, right? 04:55.000 --> 04:57.000 Like we want to record all of this data, it never gets updated. 04:57.000 --> 04:58.000 A log never gets updated. 04:58.000 --> 05:01.000 Eventually, we want to delete it when we don't care about it anymore. 05:01.000 --> 05:05.000 But we want to read it a lot in the meantime. 05:05.000 --> 05:10.000 Because we're a partitioning that data, usually by the time frame, 05:10.000 --> 05:14.000 it's very, very easy to clean up with things like TTL that you can just delete an entire partition 05:14.000 --> 05:16.000 when you don't care about it anymore. 05:16.000 --> 05:21.000 It's very fast, as I mentioned, it can deal with cardinality, like extremely high cardinality. 05:21.000 --> 05:24.000 There's a quote from CloudFlare. 05:24.000 --> 05:27.000 They use Clickhouse internally. 05:27.000 --> 05:32.000 And they, right, they talk about how it's with prometius low cardinality means dozens, 05:32.000 --> 05:34.000 maybe scores. 05:34.000 --> 05:38.000 With Clickhouse, low cardinality means millions. 05:38.000 --> 05:42.000 So, it's the cardinality is completely different orders of magnitude, right? 05:42.000 --> 05:44.000 In terms of what we can do it. 05:44.000 --> 05:46.000 And this is really useful at observability data, 05:46.000 --> 05:51.000 when we don't necessarily have control over the cardinality of the data that we're storing. 05:51.000 --> 05:55.000 There's a lot more, and like I said, I'm going to talk about this tomorrow, 05:55.000 --> 06:00.000 but there are a lot of integrations with Clickhouse that make it really really awesome for observability. 06:00.000 --> 06:02.000 Co-groups, signals, query, and hypergex, 06:02.000 --> 06:05.000 Christmas, these are all tools of observability tools built on Clickhouse. 06:05.000 --> 06:06.000 They're awesome. 06:06.000 --> 06:09.000 All right, so here's what we're actually here to talk about today, 06:09.000 --> 06:15.000 which is observing Clickhouse itself, not using it for our other observability data. 06:15.000 --> 06:21.000 So, one of the cool things about Clickhouse is that it's actually very observable, right? 06:21.000 --> 06:25.000 Observable in the mechanical engineering definition of the word, 06:25.000 --> 06:28.000 of being able to understand how a system is working, 06:28.000 --> 06:31.000 without changing the system without prodding it, right? 06:31.000 --> 06:35.000 Clickhouse is just reporting all of the data and recording all of the data about its internal operations, 06:35.000 --> 06:39.000 and we can just examine that and inspect that whenever we need to know what's going on. 06:39.000 --> 06:41.000 It does a lot of this in these system tables, right? 06:41.000 --> 06:46.000 We have the query log of all of the queries, the park log of the log of the partitions and the merging. 06:46.000 --> 06:49.000 The trace log, this is actually not distributed tracing, right? 06:49.000 --> 06:53.000 This is a stack trace log and can be used for profiling, 06:53.000 --> 06:58.000 and then we have this open geometry span log, which may not be in your Clickhouse cluster out of the box, 06:58.000 --> 07:01.000 but is very, very easy to turn it on and I'm going to show you how. 07:01.000 --> 07:04.000 And then there's a lot more, right? 07:04.000 --> 07:07.000 But let's talk about the architecture real quick. 07:07.000 --> 07:12.000 So, like I said, Clickhouse is very scalable. 07:12.000 --> 07:14.000 Part of how this happens is horizontally, right? 07:14.000 --> 07:16.000 So, we can parallelize the queries. 07:16.000 --> 07:21.000 Clickhouse is very, very good at deciding exactly which node need to be involved in a specific query. 07:21.000 --> 07:25.000 It's going to distribute that work to the nodes that need to be involved that might have relevant data. 07:25.000 --> 07:29.000 And then like we showed a couple of slides ago, right? 07:29.000 --> 07:35.000 The data is all stored in a column or format ordered in these mergeable partitions. 07:35.000 --> 07:39.000 And you can also use things like tiered storage, right? 07:39.000 --> 07:44.000 With like S3 and object storage so that you can, you know, have your hot data very close to the compute, 07:44.000 --> 07:50.000 ready to go and have one term storage that is cheaper, but maybe a little bit slower. 07:50.000 --> 07:54.000 And then we have clickhouse keeper or zookeeper, you can use either one, 07:54.000 --> 07:59.000 clickhouse keeper sort of the clickhouse native one, zookeeper right, of course, we're all familiar with. 07:59.000 --> 08:08.000 And this is going to handle things like consensus, replication, and all of those things that keep the clickhouse clusters coordinated 08:08.000 --> 08:11.000 and working together, right? 08:11.000 --> 08:17.000 And we were going to talk about this, how it's kind of merging in the background and improving our query efficiency over time. 08:17.000 --> 08:20.000 And then when it's actually processing a query, right? 08:20.000 --> 08:25.000 So we say we have an insert, we have these three steps, we're going to parse and plan the query. 08:25.000 --> 08:34.000 This is actually not a distributed node, but it's kind of the same idea, right? 08:34.000 --> 08:40.000 We're going to parse and plan the query, we're going to load the relevant data and then we're eventually going to format and respond. 08:40.000 --> 08:45.000 Let me show some of these steps in our tracing in a little bit. 08:45.000 --> 08:49.000 But one way that we can speed this up, make it work on even more data, right? 08:49.000 --> 08:50.000 It's to just paralyze it. 08:50.000 --> 08:56.000 We can do that inside the single node with more threads, and we can also add more nodes for example scaling. 08:56.000 --> 09:03.000 And this is what allows us to do those, you know, petabyte reads in under a few seconds or a second. 09:03.000 --> 09:13.000 Similar, similarly, right, with aggregations, except we're doing some extra steps in memory before we're responding. 09:13.000 --> 09:16.000 So, guys, this is really, really good at aggregation, right? 09:16.000 --> 09:18.000 Like, oh, that database has calmed our databases. 09:18.000 --> 09:24.000 These really lend themselves to, I don't need a specific data point, but I need to aggregate all of these things across billions of rows. 09:24.000 --> 09:31.000 One story that I like to tell is when I worked at an observability vendor, and I need to know which customers were using a specific new feature. 09:31.000 --> 09:33.000 And we hadn't instrumented it. 09:33.000 --> 09:38.000 And I asked my late engineer, hey, do we know which customers are using this feature? 09:38.000 --> 09:41.000 He was like, well, it would show up in the metadata of their spans if they're using it. 09:41.000 --> 09:50.000 So let me just query all of the spans for the last three months for all of our customers and tell you what percentage we're using this new feature. 09:50.000 --> 09:52.000 And I got my answer in about seven seconds. 09:52.000 --> 09:55.000 That's pretty cool, right? 09:56.000 --> 09:59.000 So here's some more. 09:59.000 --> 10:03.000 How does click out right on how click out is kind of doing that? 10:03.000 --> 10:13.000 Thread aggregation with hash tables and then eventually merging and giving us a result. 10:13.000 --> 10:14.000 Okay. 10:14.000 --> 10:19.000 So, I showed that architecture slide for open geometry earlier. 10:19.000 --> 10:21.000 And one of the things we saw was the SDKs, right? 10:21.000 --> 10:25.000 And we have all these awesome SDKs for 11 different programming languages in open telemetry. 10:25.000 --> 10:27.000 One of them is for C++. 10:27.000 --> 10:29.000 Clickhouse is written in C++. 10:29.000 --> 10:33.000 So it's like, oh, if we wanted to trace clickhouse, we would just install this. 10:33.000 --> 10:37.000 See, wait, how do we, which is that mean we have to make our own build of clickhouse. 10:37.000 --> 10:40.000 That sounds crazy what we could use auto instrumentation. 10:40.000 --> 10:46.000 If we were in Python or Java, but clickhouse isn't in one of those languages that has auto instrumentation with open telemetry. 10:46.000 --> 10:48.000 But like I said, it's all just built in. 10:48.000 --> 10:50.000 We just have to turn it on. 10:50.000 --> 10:54.000 Clickhouse already has open telemetry tracing built in if you turn it on. 10:54.000 --> 10:57.000 So what does that look like? 10:57.000 --> 11:00.000 Well, you turn it on. 11:00.000 --> 11:02.000 And you get something like this. 11:02.000 --> 11:03.000 Now, this is a pretty simple query, right? 11:03.000 --> 11:07.000 This is a table creation query, but just kind of showing a very basic trace. 11:07.000 --> 11:10.000 So we can see this in this first span. 11:10.000 --> 11:13.000 Right, we're receiving a network request essentially. 11:13.000 --> 11:15.000 We're using the TCP handler. 11:15.000 --> 11:16.000 And then we're processing a query. 11:16.000 --> 11:17.000 Nothing's actually happening here, right? 11:17.000 --> 11:19.000 You can see this black bars where things are actually happening. 11:19.000 --> 11:25.000 And kind of without the black bar means we're waiting for other processes or functions to complete. 11:25.000 --> 11:29.000 So then we step down into the query interpreter and we interpret the query. 11:29.000 --> 11:32.000 And the query interpreter sees that this is a table creation query. 11:32.000 --> 11:35.000 And so it does its DVL action, right? 11:35.000 --> 11:37.000 Then we return to the query. 11:37.000 --> 11:43.000 And it takes, you know, what, two and a half milliseconds for the network to respond here. 11:44.000 --> 11:45.000 So this is kind of cool. 11:45.000 --> 11:46.000 This is a single trace. 11:46.000 --> 11:48.000 Let's look at a more interesting query. 11:48.000 --> 11:51.000 I think maybe this one. 11:51.000 --> 11:53.000 Yeah, so here's a much more interesting query, right? 11:53.000 --> 11:59.000 And we can see some of the parallelization that's happening here to speed up this read query. 11:59.000 --> 12:01.000 It actually ended up returning zero results. 12:01.000 --> 12:08.000 But, right, it's really, we can see how awesome this format is. 12:08.000 --> 12:12.000 Now, this is useful in it on its own, but it's even more useful in aggregate. 12:12.000 --> 12:15.000 Because in aggregate, we can start to identify patterns, right? 12:15.000 --> 12:21.000 Like, oh, this particular sub-process is always slow on this node. 12:21.000 --> 12:25.000 What may be happening on that node that we need to look at. 12:25.000 --> 12:30.000 And so we can kind of use this to sort of optimize both our infrastructure layout. 12:30.000 --> 12:34.000 And maybe we might see something about how merging is happening. 12:34.000 --> 12:37.000 It might tell us that we need to change our queries, right? 12:37.000 --> 12:41.000 And reformat them and make them more efficient or we might need to change our schema. 12:41.000 --> 12:49.000 And make that conform better to what clickhouse can sort of spread out. 12:49.000 --> 12:53.000 These aren't slides anymore, so yeah, there we go. 12:53.000 --> 12:55.000 Okay. 12:55.000 --> 12:57.000 So how did I get that? 12:57.000 --> 12:59.000 Well, oh, that's hard to read. 12:59.000 --> 13:01.000 Okay, as you've done. 13:01.000 --> 13:04.000 So I said, it's just one flag to turn it on, right? 13:04.000 --> 13:06.000 And then we get this table. 13:06.000 --> 13:09.000 And this table is the open telemetry span log table. 13:09.000 --> 13:16.000 These are fully formed open telemetry spans ready to go ready to use in your observability tool of choice. 13:16.000 --> 13:21.000 Where we were just looking at Grafana, but you could ship these anywhere that's open telemetry compatible. 13:21.000 --> 13:26.000 To get these, all we need to do is tell Clickhouse, hey, I want you to start some traces, 13:26.000 --> 13:29.000 even if you didn't get a trace context from the thing that called you. 13:29.000 --> 13:33.000 So we can turn this on on a session, you can also step this on a profile, right? 13:33.000 --> 13:39.000 And then based on that probability, so if it's set to one, it's always going to create a new trace for every new query. 13:39.000 --> 13:43.000 If it's 0.5, half of the time, it will create a new trace. 13:43.000 --> 13:48.000 And you can, you know, reduce some of your data because if you have a lot of if you've high throughput of queries, 13:48.000 --> 13:50.000 this can generate a lot of data, right? 13:50.000 --> 13:54.000 You probably don't want this turned on for things like a profile that's doing a health check, right? 13:54.000 --> 13:56.000 That would just be kind of redundant and wasteful. 13:56.000 --> 14:02.000 But say you have a user who's like always breaking your Clickhouse cluster. 14:02.000 --> 14:05.000 You can create a profile for them that turns on all the debugging. 14:05.000 --> 14:10.000 So you know exactly what they did and can tell them not to do that again. 14:10.000 --> 14:15.000 And we can also do it for an individual from the client, right? 14:15.000 --> 14:17.000 So the client can provide a trace ID here. 14:17.000 --> 14:18.000 I'm just spoofing one. 14:18.000 --> 14:20.000 I'm like, this is just a random string that I came up with. 14:20.000 --> 14:24.000 And I'm like, hey, use this as your parent trace, right? 14:24.000 --> 14:30.000 If we were coming from an actual like Python application or a job application that's calling Clickhouse, 14:30.000 --> 14:33.000 that SDK would include that trace ID. 14:33.000 --> 14:37.000 And then that would allow us to contextualize all of this distributed stuff happening inside Clickhouse, 14:37.000 --> 14:42.000 with all of the other stuff that we had going on before we got to Clickhouse and anything downstream as well. 14:42.000 --> 14:46.000 This is the beauty of trace propagation. 14:46.000 --> 14:49.000 And then once we've got that data in that table, right? 14:49.000 --> 14:50.000 What do we actually do with it? 14:50.000 --> 14:53.000 How do we get it out into our observability systems? 14:53.000 --> 14:57.000 This query is from the documentation. 14:57.000 --> 14:59.000 It's a little bit out of date. 14:59.000 --> 15:01.000 You've zipped in up there. 15:01.000 --> 15:03.000 It's like an older trace format. 15:03.000 --> 15:07.000 I think Yeager is still technically supportive, but it's not on by default. 15:07.000 --> 15:09.000 So the documentation is out of date. 15:09.000 --> 15:11.000 We're going to work on fixing that. 15:11.000 --> 15:15.000 But what this shows is that we can use a combination of Clickhouse features, 15:15.000 --> 15:17.000 a materialized view, right? 15:17.000 --> 15:23.000 And a URL table engine, and essentially export all of our trace spans over an HTTP endpoint. 15:23.000 --> 15:25.000 I wouldn't recommend actually doing this in production. 15:25.000 --> 15:27.000 There's some issues you're going to lose. 15:27.000 --> 15:29.000 Basically, you'll lose traces when there are errors, 15:29.000 --> 15:33.000 which is when you most want those traces. 15:33.000 --> 15:37.000 And you might also create availability issues for your cluster using the URL engine 15:37.000 --> 15:41.000 if the endpoint that is trying to write to isn't highly available. 15:41.000 --> 15:45.000 So what I would recommend doing instead, like for production, 15:45.000 --> 15:47.000 is just straight, right? 15:47.000 --> 15:49.000 Make your own receiver for the open symmetry collector. 15:49.000 --> 15:51.000 Make your own program that basically just scrapes that table. 15:51.000 --> 15:55.000 And we used a view actually for it, so I can show that. 15:55.000 --> 15:57.000 Right? We created this view. 15:57.000 --> 15:59.000 Instead of Zipkin format, this is open symmetry format. 15:59.000 --> 16:01.000 You can also turn this into a materialized view. 16:01.000 --> 16:05.000 Just going to take that deal with those issues that I've talked about a second ago for production. 16:05.000 --> 16:07.000 But you can create this view, 16:07.000 --> 16:09.000 and then build something that scrapes this, 16:09.000 --> 16:11.000 and you have your trace spans. 16:11.000 --> 16:15.000 Okay. 16:15.000 --> 16:17.000 Okay. 16:17.000 --> 16:25.000 And I mentioned briefly about that we could kind of put the client side traces into clickhouse. 16:25.000 --> 16:29.000 I actually have a call to action here, because this is not complete. 16:29.000 --> 16:31.000 This is the beauty of open source, right? 16:31.000 --> 16:35.000 We do get trace propagation with these clickhouse clients. 16:35.000 --> 16:37.000 But there are clickhouse clients for many more languages, 16:37.000 --> 16:39.000 and there are community contributed ones. 16:39.000 --> 16:43.000 Most of them do not include this open symmetry trace propagation. 16:43.000 --> 16:47.000 So please, if you're using one of those client libraries, 16:47.000 --> 16:49.000 and you feel compelled to contribute, 16:49.000 --> 16:53.000 consider adding open symmetry trace context propagation. 16:53.000 --> 16:55.000 That would be awesome. 16:55.000 --> 16:57.000 Please help. 16:57.000 --> 16:59.000 Okay, real quickly some best practices, 16:59.000 --> 17:01.000 and then we'll take a few questions. 17:01.000 --> 17:03.000 So this is awesome for distributed queries. 17:03.000 --> 17:05.000 I was showing queries right now and I think we'll know it, 17:05.000 --> 17:07.000 but this is most useful when you have distributed queries 17:07.000 --> 17:09.000 running across multiple nodes, 17:09.000 --> 17:11.000 and especially when you want to identify issues 17:11.000 --> 17:13.000 that are specific to a shard or two node, 17:13.000 --> 17:17.000 you can use it to identify root causes of slowness, 17:17.000 --> 17:19.000 because problems are going to cascade 17:19.000 --> 17:23.000 and traces have the order of operations embedded in their structure. 17:23.000 --> 17:27.000 So it tells you the root clause, 17:27.000 --> 17:29.000 basically in the structure of the log itself. 17:29.000 --> 17:31.000 You don't have to figure this out. 17:31.000 --> 17:33.000 You don't have to do any investigation. 17:33.000 --> 17:36.000 And right, so I kind of mentioned the times 17:36.000 --> 17:38.000 when you would want to turn this on, 17:38.000 --> 17:40.000 especially for problematic users, 17:40.000 --> 17:43.000 problematic queries, things like that, 17:43.000 --> 17:45.000 not for routine things like health checks, 17:45.000 --> 17:47.000 and so on. 17:47.000 --> 17:49.000 Thanks. 17:49.000 --> 17:50.000 Yeah. 17:50.000 --> 17:51.000 Yeah. 17:51.000 --> 17:53.000 Thank you. 17:59.000 --> 18:00.000 Any questions? 18:00.000 --> 18:01.000 Redoison. 18:01.000 --> 18:02.000 Good job. 18:02.000 --> 18:03.000 Sorry. 18:11.000 --> 18:12.000 Hi. 18:12.000 --> 18:13.000 Hi. 18:13.000 --> 18:15.000 My question is, 18:15.000 --> 18:18.000 how clickhouse is working on the client side 18:18.000 --> 18:22.000 to provide more scalability in the number of client connections 18:22.000 --> 18:24.000 that can connect to the cluster? 18:24.000 --> 18:27.000 Because I know there's sort of like a hard limit 18:27.000 --> 18:29.000 when you install a cluster, 18:29.000 --> 18:32.000 and then you have to kind of make the changes, 18:32.000 --> 18:35.000 but is there any work for kind of 18:35.000 --> 18:39.000 make the number of clients connected to the clickhouse cluster 18:39.000 --> 18:43.000 be more like changeable in a way 18:43.000 --> 18:47.000 that you can have like thousand clients connecting 18:47.000 --> 18:50.000 to the cluster without any issues. 18:50.000 --> 18:53.000 Because yeah, nowadays it's not like very. 18:53.000 --> 18:54.000 Yeah. 18:54.000 --> 18:56.000 I know that this is not respect. 18:56.000 --> 18:58.000 I'm not aware of some of my co-workers here, 18:58.000 --> 19:00.000 so we want to stop by the Osakam booth 19:00.000 --> 19:02.000 and ask them they might be more aware of future work, 19:02.000 --> 19:04.000 but the way it stands currently, 19:04.000 --> 19:06.000 you definitely want to reduce the number of queries. 19:06.000 --> 19:08.000 You can insert a lot of data with each query, 19:08.000 --> 19:10.000 and clickhouse has a strong preference for 19:10.000 --> 19:13.000 big inserts versus many inserts. 19:14.000 --> 19:15.000 And think, right? 19:15.000 --> 19:17.000 They want to reduce the reading side, 19:17.000 --> 19:20.000 so kind of read like thousand times 19:20.000 --> 19:22.000 like using thousand connections. 19:22.000 --> 19:24.000 Oh, that's not it. 19:24.000 --> 19:27.000 So just scaling up the reading? 19:27.000 --> 19:31.000 Yeah, actually the amount of clients that will read that data 19:31.000 --> 19:33.000 because right now it's just like when you install. 19:33.000 --> 19:37.000 It's like, I think it's 100 or 200 something like that. 19:37.000 --> 19:38.000 Okay. 19:38.000 --> 19:39.000 Kind of limpets. 19:39.000 --> 19:42.000 And then if your application kind of goes there, 19:42.000 --> 19:45.000 every time you create a loop and then, 19:45.000 --> 19:47.000 like you can scale and then, 19:47.000 --> 19:48.000 these scale. 19:48.000 --> 19:50.000 Yeah. 19:50.000 --> 19:52.000 For reading, I don't know. 19:52.000 --> 19:54.000 For writing, you can use a reverse proxy, 19:54.000 --> 19:56.000 like the opens, elementary collectors are great. 19:56.000 --> 19:57.000 Reverse proxy to put in front. 19:57.000 --> 19:59.000 Up there goes cluster, maybe there's something similar 19:59.000 --> 20:03.000 that could have like a read cache on the read side. 20:03.000 --> 20:05.000 Robert, do you know them? 20:05.000 --> 20:06.000 Yeah. 20:06.000 --> 20:07.000 Okay. 20:07.000 --> 20:11.000 Come to the booth and talk to our CTO and he'd like to thank it. 20:11.000 --> 20:16.000 Any other questions? 20:16.000 --> 20:17.000 Oh. 20:25.000 --> 20:26.000 Hi. 20:26.000 --> 20:27.000 It's slightly left field question, 20:27.000 --> 20:29.000 but when you're collecting a telemetry data, 20:29.000 --> 20:32.000 do you hit any privacy or a consensus? 20:32.000 --> 20:33.000 Sorry, what? 20:33.000 --> 20:36.000 Any privacy concerns when you're collecting all the data? 20:36.000 --> 20:37.000 Absolutely. 20:37.000 --> 20:39.000 Now, in Clickhouse itself, 20:39.000 --> 20:43.000 it is going to include that query statement in the trace 20:43.000 --> 20:45.000 span, so if there's PII in that query statement, 20:45.000 --> 20:47.000 that's something that you're going to have to build 20:47.000 --> 20:49.000 some way to redact that. 20:49.000 --> 20:52.000 If there's no PII or anything that you care about in the 20:52.000 --> 20:55.000 actual query statement, it's not going to get caught up 20:55.000 --> 20:57.000 by the built-in clickhouse instrumentation, 20:57.000 --> 20:59.000 but it might get caught if you're using open 20:59.000 --> 21:00.000 telemetry auto instrumentation. 21:00.000 --> 21:03.000 So it's definitely something you'd want to build filters for. 21:03.000 --> 21:07.000 Are there any specific strategies that you have for that kind of reduction? 21:07.000 --> 21:12.000 Less familiar with the current state of the art right now in this. 21:12.000 --> 21:16.000 I do believe that Red Hat was working on an open source tool 21:16.000 --> 21:21.000 specifically for handling correlation and filtering of this type of data? 21:21.000 --> 21:22.000 Cool. Thanks. 21:22.000 --> 21:23.000 Yeah. 21:25.000 --> 21:27.000 I thought I saw a hand go up. 21:27.000 --> 21:29.000 Maybe I'm imagining. 21:29.000 --> 21:31.000 Other questions? 21:33.000 --> 21:35.000 Thank you.