Відео

Wade here. Thank you. Glad you enjoyed it.

Hi Wade. Thanks a lot for your videos. I think I have one thing that you probably could address. I read somewhere that, if you use CDC (i.e. Debezium), there is no need to store the events in the outbox table at all. It is sufficient to save and immediately delete them, because CDC only sees the changelog of the db, reacts to the insert and ignores the delete. Right?

No they flipped

Adam here. It seems you're missing some key elements: 1) A traditional RDBMS bundles processing with storage.In headless, the storage is completely separate from the processing and query. I do not know of any RDBMS systems that let you access their underlying storage (Eg: a B-tree) directly, but if there is, I would be keen to find out. 2) You don't have the risk of one query saturating your data layer and stalling other queries, like you do in an RDBMS. However, this relies on the fact that most data layers will be served via massive cloud compute storage (R2, GCS, Azure Blob, S3, etc), and not you running your own data layer on an in-house HDFS. 3) Client-server embeds business logic inside the server for the client to call. There is a tight coupling between the two. In HDA, there are no smarts in the data layer. It's just data that has been created by the upstream services. IF your server only provided raw data via a GET, and writes via a PUT/POST, but had absolutely no other functionality whatsoever, then you could equate it to a headless model. That's pretty much what Iceberg and Kafka do, with a bit of cleanup optimizations sprinkled in.

Wade here. Depends on the final implementation details of the system. They certainly could have an initial layer of basic checks that run synchronously (see other comments). This would be looking for obvious things that can be detected on a single transaction. But, the reality is that most fraud doesn't show up on a single transaction. A lot of fraud detection happens over multiple transactions. As a result, the nature of the problem dictates that they have to let through some fraudulent transactions because they can't detect it until they have seen a pattern develop. And yes, if the $$ amount is high, that could be a problem. That is why many banks have transaction limits. They have decided what level of risk they are willing to take and set the limit accordingly. There's a balance to be struck. The bank has to decide where they want to strike that balance. Would they rather have valid transactions fail (or timeout) or potentially let through more fraud? Both situations are costly. But which one costs more?

Wade here. You probably want to take some time and watch the rest of the video series here: developer.confluent.io/courses/microservices In particular, you might find the most recent video helpful (and the comments below it): developer.confluent.io/courses/microservices/case-study-asynchronous-events/ But the gist of it is that Fraud Detection takes time. You generally can't look at a single transaction and say "oh, that's fraud." You have to look at patterns of transactions over time. Eventually, you develop enough information that you can start to say "Ah, something weird is happening here" and at that point, you lock the account and prevent future transactions.

Wade here. The thing is, it may not be as simple as detect fraud or don't detect fraud. It's a balance. Yes, you obviously want to detect fraud as accurately as possible. But your bank also needs to be able to give people money when they ask for it. Financial systems, payment processors, etc, have very strict limits on how long they can take. If running fraud detection exceeds those limits, all transactions will be rejected, whether they are fraudulent or not. If that happens, the fraud detection is going to be largely irrelevant because the bank won't have any customers left. Having said that, in a real system, they can definitely have layers of fraud detection. Do the fast stuff in real-time, and leave the slower stuff for an async process.

@@WadeWaldron-Confluent alright. now i know a bit more about banking system

Adam here. I covered that here: ua-cam.com/video/dahs6jpCYQs/v-deo.html Headless is the full decoupling of data access from processing _of all forms_, providing reusable data assets for anywhere in the company - not just for analytics use cases. To emphasize - a data lake, or lakehouse, or lake-like warehouse, are all analytics constructs first and foremost. They're also predicated on the notion that you must copy all the data into the data lake's bronze layer "as-is". From there, you add schemas, formatting, and structure, leading to a silver layer (another copy). Then you can start doing work on it. The problems: 1) You don't own the data. The source breaks, your pipeline breaks, and then you're forced to react to it, determine impact / contamination, reprocess data sets, etc. (Did this for 7-8 years myself, no thanks). 2) It's stuck in your data lake. All that work you did to convert it Source->Bronze->Silver is only usable if you use the data lake. _Historically_, leading data lake providers have been happy to provide you with the best performance ONLY if you use their query engines. Using an external engine (if even compatible) would lead to far worse performance. Data lakehouse/warehouse/househouse providers were more than happy to lock you in on the data front, because they made big $$$ on it. But happily, this is starting to change due to the adoption of open-source formats that you can run yourself in house - you can see it with the growing adoption of Iceberg (Databricks bought Tabular, Iceberg cocreators - Snowflake is investing heavily in Iceberg, open sourcing their Polaris catalog). Data lake providers _could_ decide NOT to adopt these open formats, but then they risk losing their business to those who have - so the result is that most players are letting go of their control over storage so that they can adopt Iceberg/Delta/Hudi compatible workloads that they may not have had access to before. If you want a quick mental shortcut for how this is different - a headless data architecture lets you plug your data into your data lake _at the silver layer_. The data is well-formed, controlled, schematized, and has designated ownership. But you can also plug that same data into a DuckDB instance, or you can plug it into BigQuery, or Flink, or any other Iceberg-compatible / Kafka compatible consumer endpoint. The idea is that you've decoupled the creation of the data plane from "analytics-only" concerns, and instead focused on building modular reusable building blocks that can power any number of data lakes, warehouses, or swamps, in addition to operational systems.

Adam here. I'm not advocating removing bounded contexts, putting all the data into a big mixed pile, and tightly coupling all your systems together. A headless data architecture promotes the data that should be shared into an accessible format - in this version, the stream or the table. If you already have EDA with well-formed and schematized streams you're halfway there. The table component is an extension, where you take business data circulating in your streams and materializing it into an Iceberg table - but note that we didn't shove it into some data lake somewhere for just the lake to use. It remains outside the lake, and is _pluggable_ into whatever lake, warehouse, operational system, SaaS app, or client that needs it. This pluggability forgoes copying, so that you don't need to build pipelines and copy data. The gist is that you focus on building a data layer that promotes access and reuse - something that comes for free with a Kafka-based EDA, but that has historically struggled for tables due to the general approach of dumping it all in a data lake to sort it out later.

Well, the team building your architecture could abstract it below the public API. If you query data from BiqQuery, make the system do all processing on GCP and so on. However, if you're trying to join/aggregate data from different clouds, then yeah, I guess you're out of luck. Or you could make a query engine that is architecture aware and takes into account where the data is, the potential egress/ingress, etc as cost for the query planner and then try to push down as many operations as possible, so that you only send through the internet the most compact and already processed data, instead of the entire projection.

Adam here. Inter-cloud costs remain a factor, but typically I wouldn't expect to see a single query federated across clouds WITHOUT taking into consideration data locality. For example, issue separate queries for each cloud, aggregate locally, then bring those results to a single cloud for final aggregation (basically a multi-cloud Map-Reduce - thanks Hadoop!). Speed also remains a factor, as you pointed out, due to CAP theorem. There is no free lunch, so if you're going with a global, multi-cloud, distributed data layer, then yeah, you should probably invest in some tooling to prevent your users from shooting themselves in the foot with a $50k per query bill.

That's the common quote, yes. However, in this case, Danica is correctly quoting from J.R.R. Tolkien. www.goodreads.com/quotes/229-all-that-is-gold-does-not-glitter-not-all-those

Hi, Gilles from Confluent here. You can add business domain metadata via the UI or the API, or you can use our Terraform provider. Check out my video to see how to do that: ua-cam.com/video/Vg7k_3vlC3Q/v-deo.html.

Wade here. You are welcome. Glad you enjoyed it.

Wade here. Yeah, that's more or less my thinking as well. Simple rules that can be executed fast can be used for a quick sanity check on the transaction. Async events for a more detailed analysis. The key is to recognize that sometimes things need to be done synchronously, but that doesn't mean everything needs to be synchronous.

Wade here. Absolutely. Within our industry, there is a tendency to assign strong consistency to processes that aren't actually strongly consistent and don't really require it. That is a hurdle to overcome. And it's not to suggest that you can always overcome it. Sometimes, the business wants strong consistency, and you can't do anything to change their mind. However, we should avoid assuming that everything must be strongly consistent. Start with async events. Fallback to other techniques if required. And make sure the business understands the consequences. If making the operation strongly consistent results in a 10x increase in latency (as an example), is that really what the business (or customer) wants?

​@@ConfluentDevXTeam Totally agree

Wade here. Thanks. Glad you enjoyed it.

Wade here. I'm glad you got value out of it. I'm curious what aspect you found so important that you feel other orgs need to hear it.

Ephemeral means that there is no indefinite persistence of the events. It can vary from "all events are only in memory" (and therefore vulnerable to deletion upon power loss) to "events are persisted to disk, but are deleted after a short period of time "(seconds, minutes, or perhaps after consumption). For example, a consumer may only receive the messages if it is online at the time the producer writes it to the broker. A consumer coming online later is unable to access the historic events because they are no longer stored anywhere.

Wade here. Glad you enjoyed that part. Maybe it's not as controversial as I thought. When I first started doing deploy to prod first, I have to say I loved it. It definitely requires a bit of care, but it eliminates so many surprises that I would say it's worth it. I remember a project where the developers didn't do this. They spent months of crazy overtime building this thing and when they finally did deploy it to production all the wheels fell off. Turns out, it worked great in tests, but terribly against real prod data. I believe the term used was "carpet speed". Had it been deployed first to production and potentially run against production data, this could have been discovered much earlier.

Lucia here. That's a great question! Both ksqlDB and FlinkSQL can analyze data that is in Kafka topics. However, ksqlDb is built on Kafka Streams, while FlinkSQL is an API offered by Apache Flink. You can read more about FlinkSQL here: cnfl.io/4bR5ndv

Adam here. You may want to check out this video my colleague Tim made a few years back on the basics of Kafka - ua-cam.com/video/06iRM1Ghr1k/v-deo.html In terms of speed and reliability, those topics tend to get a bit more advanced. Speed needs to be broken down into latency and throughput. You can have high throughput (lots of MB/s) and low latency (< 10 mS), though the two often have a bit of a tradeoff (lots of small writes is low latency, but the writes have overhead so your throughput goes down). In terms of reliability, a lot of it has to do with things like exactly-once processing, transactions (so you don't lose data), distributed systems (if you lose a broker you don't lose data, and your system can keep working as before). Anyways, lots to unpack there. A bit more of a hands-on approach can be found here on Confluent developer.confluent.io/learn/kafka-performance/