Live Demo: Getting Started with Conduktor + Q&A – Transcript

00:05hello hey hey loan he everybody in the chat let us know where you're tuning in from while we just wait for a few other people to join we will get started in one minute

00:49okay let's let's get going uh because I think we've got a bit to get through so uh welcome to this webinar uh today we're going to look at how to get started with conductor um be useful if you are new to conductor even if you already know about conductor and maybe you want to learn a bit more about some of the features of some of our different products then we're going to give a bit of a grand tour today um couple of housekeeping rules if you're new to the

01:15live stor platform if this is the first webinar you've been to uh feel free to ask us questions we will be reactive um react um Chuck your questions in and we can either answer them as we go through or we'll come back to anything at the end so quick introductions before we get into it um I'm James I'm director of product at the conductor very happy to be joined by one of our customer success

01:41Engineers today so Lan do you want to introduce yourself hi everybody uh L and cook is my name and I've been working with conductor for the last two and a half years as a customer success engineer helping our customers uh succeed with our products nice how have things changed over the years Lo uh everything's gotten a little bit better good okay all right so on to today so as I said we're going to give a very quick intro into conductor if you're not familiar with us and then we're going to

02:10look at the different deployment mechanisms today for deploying our2 core artifacts that's the conductor console um our CFA management uh platform Kafka UI um which you may have seen before and then we will also do a worldwind tour of conductor Gateway this is our vendor agnostic Apache kafa proxy uh which you may new to you so look and I'll hand over to you thanks James uh and I suppose good

02:37afternoon good evening or good morning to everybody wherever you're coming in from um but first off you're probably well aware of the move from batch to data streaming and in that regard Kafka is the defao standard and however implementing CF can be quite challenging and maybe that's something you've seen already and one of our colleagues who has been in the ecosystem for a long time says that the s in KFAN for simple so there there's no s in CFA and obviously that you know it's that simple

03:05um but we typically see the customers that we I suppose deal with and people that we chat to in the industry are fall into one of five stages in their data and motion or Kafka Journey uh we're stage one would be early interest understanding if kfka is right for them maybe if they have the right project or not and if they need that kind of complexity um then the second stage identifying a project they understand that yes capka can be used here in the use case getting GP some of the

03:30terminology in Kafka producers consumers topics uh distributed systems overall and then moving to stage three where CFA will be in production in at least one project um once it's in production then you'll need to ensure that you have monitoring in place that you trust your monitoring that kind of things and then to stage four where cfas in production in multiple projects and then you'll need to ensure that you have your endtoend tests and integrate them with

03:55your cicd flow and then finally stage five where you have CFA as part of your central nervous system in the company where um you know you're fairly Advanced and you're starting to look at things like gith ups for CFA and ensuring that you have right governance across projects and you can understand who's done what and that the right um they're doing the right things as well so I suppose why do we bring this up well really because doesn't matter what stage you're at near Kafka Journey conductor products and solutions can help you and

04:23we're going to show you how how we can do that um I suppose it's good to know that if you we're looking at conductor solutions that we are complete and agnostic so what do we mean here well we cover the entire CF ecosystem not just the producers and consumers but CF connect ksql schem of registry um doesn't matter what you're using we can help you and then also the same on the provider side so whether you're using a manage provider or whether you're using an open source cafka we can help

04:53you and then this slide so we we'll talk later about in terms of deployments the there's two technical artifacts one is Conductor console and that would be the UI in front of your CFA which helps you understand your different CFA resources and control access for your users and then there is Conductor Gateway which is another technical artifact which would be a h proxy in between your CFA clients

05:18and Kafka itself and that provides further functionality um for your applications and then James I know is going to speak a bit more about it but things like having encryption end to end or having virtual clusters to optimize your cluster setup um but just to make youw that there's two technical artifacts and then we spoke about two technical artifacts and then commercially they're split to four

05:44different solutions so across those Solutions are conductor console governance data security and optimize and uh we're not going to suppos go line by line but you can see what's in each of those Solutions there and I guess worth pointing out Len that to get the full benefits of any of those Solutions you would warrant deploying both the console and the Gateway that we're GNA we're going to walk through today yeah absolutely

06:13absolutely um and also good to know you would be the first person to use conductors so since 2020 we have more than 100,000 users across 15,000 Global Tech teams uh across basically every industry supp we're not going to pick up anyone here but you can see a lot of well-known names and also if you're in the system integrator space we work with a lot of your colleagues as well so oh go ahead James I was just

06:40going to say moving on so we're going to walk through how you deploy both the console and the Gateway and then after each uh introduction to how you to deploy it we're going to do a quick demo quick tour of some of the features and functionality you can expect to find in them as well so uh we'll start with the console um so loer maybe you give us a quick a quick recap of of what the console is and what it does yep so console will give you the ability to explore um View and manager

07:07cap resources like your topics your schemas Etc also troubleshoot and have an observability layer so be able to see live consumers messages come in as they happen be produced to carry out some some ad hoc tests and then be able to monitor your applications infrastructure and then also give you access control to an Enterprise level so uh granularly for your users down to CFA resources and

07:32then also for them signing in through single sign on either ldap or IDC um that that's the functionality you can get in console if will show cool so in terms of deployment what's involved so there's there's two Docker images um and I suppose there's really two hard requirements so one is the console image and then a postgress database to store information related to

07:59that which would be things like your users any sort of access control your Cafe clusters that kind of stuff if you do want to make use of conductor monitoring you'll have to deploy that as a separate Docker image as well and then what happens is Conductor console speaks to your Kafka takes that information and then conductor monitoring then scrapes metrics from conductor console and shows you that information um in its own UI section and then for monitoring you'll also need to deploy S3 like storage to

08:26store information for more than two or three days um the final one there for your users there is the ability to use SSO so either oidc or ldap based so meaning supporting all identity providers uh yeah yeah as long as you're using either o or Ora awesome and the deployment options so you have three main options either

08:53going with something like dock compose or if you're using kues we have a Helm chart that you can utilize or if you're prefer to use some of the kind of managed um container services then that is also an option we have a cloudformation template to use ECS on adress but if you prefer to use something like I think it's called ACI on Azure I'm not actually sure the go equivalent but whatever way you run

09:17containers you can run contr console awesome and terraform templates coming soon as well I think yeah that's definitely a topic um I suppose a bit maybe getting into a bit more on the technical side on the console deployment if we're looking at do compose uh you can see here there's two containers running the console one on the left which is taking your cap information and then has the postgress store backing it up but then you've also

09:45got the monitoring conductor monitoring Docker image so there's a metric ends metrics endpoint which monitoring is scraping metric from and then it's storing that information um on its own UI for you and then you have your S3 like store where you're storing um the monitoring information and then the other thing I was going to say if you do set up alerts then there is specific alert manager there as well the final thing in the most recent version of

10:12conductor console in so 1.21 we have um given you the ability to export the metrics on conductor so maybe you're using something like Prometheus and graan yourself we have those endpoints available for you and you can uh basically export that to your own uh prome EMP and I think if you're just getting started and you're thinking this looks a bit scary uh then obviously within the docs. conductor. you can find the docker

10:40example complete Docker composed that's going to uh spin up all of these these services and do all the plumbing for you so if you are just looking to you know prove out the value of conductor understand if console's valuable for you then I'd recommend that as being a good place to start uh maybe a quick point on how you configure console as well Lan and so you can either use environmental variables or a yaml configuration file

11:03so whichever way you want to go that awesome moving on then so we also mentioned that we have a Helm chart and by if you're prefer to use kubernetes and by default I'll will start two pods so one console pod and one monitoring pod you don't want to use conu monitoring you can supposed turn it off by default it does start um it also provides Secrets out of the box and then supports some of the kuity specific

11:31stuff like Affinity Toleration and Custom Service against you I will Chuck the link in there to the chat as well to the public charts repository I also saw we got a we got a question from Rob which while we're at it I think we might as well uh address so Rob's asking uh is console still surfacing the key metrics so cortex is required only once needing the detailed

11:57monitoring with Prometheus key metrics yeah yeah I'm pretty sure it is Rob we can talk to you afterwards to to to confirm yeah I I think the the main thing here is um you you get the um core metrics associated with like resources right without cortex if you're just looking if you just want to look into your topics um look at the metadata associated with topics in the lists the

12:23consumer groups list the subjects then there's no dependency on the cortex image however if you do want to look at any of the metrics that are surfaced in the application monitoring um and the cluster monitoring screens um then you do need to put to deploy the conductor console cortex uh images of dependency so on to a quick demo cool uh

12:50let me share my screen and let me know that you can see it uh how's that looking there is it looking okay looks good to me cool um so we've talked about deploying uh console and maybe the benefits of it so let's we've deployed console what can you do inside it so I've signed in here H what I'm

13:16going to show first is uh once you've signed in you can set up your ca clusters so you don't have to do it through the UI you could do this either in the yo config or maybe using their public apis but just as suppose for listen purposes show you setting up a cafa cluster if I go in here actually I want go back on uh to this guy so pretty stra forward you'd add the name of your cluster and your bootstrap servers your authentication method and then you can test connection and then you do the same