Turkish subtitles - Would you believe me if I told you that there was a database out there that could continuously handle large volumes, information, scale automagically and be available to keep all continuously taking
- on data?
- Hello, my name is Jamil Spain, Developer Advocate with IBM.
- And today's topic is the answer to that Elasticsearch.
- All right. It's a great database data store, and I want to talk a little more about the some of the characteristics of it.
- We're going to compare it to a relational database management system and then talk about the ecosystem that comes with it.
- So to get started, let's talk about what is Elasticsearch exactly.
- Well, first it is distributed in nature and it is a NoSQL, JSON based data store.
- We're going to abbreviate that with the DSS there as well.
- So on the spectrum of where databases fall with postgres and SQL kind of being the most structured type of databases, put this on the out of spear past MongoDB when it comes to
- how unstructured and NoSQL it can be.
- When it comes to interacting with Elasticsearch interests, interests, interestingly enough, it's done through a RESTful API.
- So all your queries happen that way, you programmatically program all you indexes and all the stuff that you pretty much anything you need to interact with, it would be through rest URLs.
- And a lot of the major use cases for this, you know, could be you can take many different data sources from logs.
- It could be any type of metrics you have from different systems and maybe even some application trace data that comes in. And you can have one system that you can combine all
- of this. You think about data coming from all these different sources, and it'd be able to push them into JSON documents and then allow you the ability to search and get that information back in real time.
- So it sounds like a big job that it has to do.
- And certainly let's do it from our normal comparison of what a relational, what we know of from relational databases to see how that compares and how the lingo and the context changes.
- Well, we know that with relational database management systems, they are called databases.
- And in Elastic Search, these are known as indexes, or I-N -D-I-C-E-S, Indices, right?
- And also in a relational database, we have the term of tables.
- OK.
- And in this, they're going to be called, that could be called kind of index patterns.
- And some of the earlier versions there were known as types.
- All right. So now we know from our tables in relational database has many tables.
- All right. And we know the obvious second one we're going to look at is going to put both of these down as we're getting to the bottom of the screen here, rows and then columns.
- OK, let's get another marker here.
- And rows, just like we know from most NoSQL data sources are going to be as documents.
- And normally in a relational database, you know, you have tables, you the rows, individual columns.
- These are going to be called fields.
- So just a quick comparison, if you have a lot of familiarity with relational databases like MySQL, the Postgres, this is kind of a way to transition your understanding of all that and know how things kind of map together and when you start planning
- out your your structure these are things that you need to consider that how you can translate that over.
- So we know that it's a jSON based data store, you're going to interact with it with rest.
- And we're looking to get many, it's very powerful, has the capability to ingest data from many, many data sources and scale out.
- If I think about the CAP theorem concepts and we'll probably put this on an A to P for availability and petition tolerance.
- Already built in and depending on how you want to configure it, you could probably achieve some different consistency bases as well.
- But let's get move on to the whole ecosystem.
- So you hear the name Elastic Search out there, but often you will hear about this term ELK, E-L-K Stack.
- Here's how you will hear about it being referenced and I think the easiest way to break down how the stack works, let's diagram it out and then we're talk about each kind of component and the place that it fits.
- And that would be a great way to really help understand this.
- So let's put Elasticsearch, I'm going to abbreviate this E-S that's going to be kind of in the center of everything here.
- And what we're going to do the, the K is for Kibana.
- And Kibana is a web based UI.
- This will be how you actually interact with a lot of the data that Elasticsearch prepares and indexes for you to use and so you can build a dashboard.
- And you can build different widgets, or visualizations that can continuously update as well
- as data comes in on that side.
- So this can really be your main interface that you use to keep, keep, keep updating and looking at your data as it flows in.
- Now let's talk about the other source. We talked about the output.
- We have this great data store Elasticsearch, we're going to be we're visualizing things with Kibana, kind of our gateway to view our data and how things are running. Now let's talk about how data gets in, and there are two parts that I would
- like to talk about here.
- We have something called Logstash, and you also hear something called Beats.
- All right.
- So for a Logstash, think of this as well, it actually is a very open source server, side processing pipeline, and its main job is to do two things
- to take data in.
- Input data from many different sources is then going to transform that data and then you get to what we like to call so eloquently stash it somewhere.
- All right. Now the inputs can be from a variety of things.
- You can actually just put it in a format. Most of the time, you can add SDKs or things to your code or different systems, and they pushed the data into Logstash transformations, maybe to do some formatting on the data mine
- structuring before it comes in.
- If you would like and then you can output that through to stash it somewhere, and you can imagine one of the first plug ins that are there is Elastic Search.
- So let's complete our triangle here.
- We'll go from Logstash into Elastic Search and so you can continuously feed things in.
- Now we mentioned the part Beats that we're here, unlike the headphones these Beats are set up to be kind of agents on different servers.
- So say you have something in maybe in serverless or you have some files that you want to do or different, maybe something on Windows servers, so it's kind of a complementary kind of component that's very Longstash
- in nature, but it has plugins to many different other services.
- And one of its outputs is to go directly into Logstash.
- So collectively, you're kind of building this consistent pipeline that keeps going in as you visualize.
- You can kind of say you program more things to come in and continuously keep this circular nature coming and keep flowing.
- Now, this can scale up to massive amounts of information and nodes that can really already set up to be distributed in the nature and handle a variety of scenarios.
- But one great thing is there are containers available that you can set up this complete infrastructure all on your laptop to taste, test things out on a very much smaller scale and have it grow to a much
- larger scale, effectively making it a great component in your architecture to be how you visualize your data that will be in the data like that your ability.
- Thank you very much for your time.
- If you have any questions, please drop us a line below.
- And if you want to see more videos like this in the future, please like and subscribe.
Subtitles generated on my computer; they follow the video — click a line to jump there. Elasticsearch is a distributed NoSQL JSON-based data store that can automatically scale large volumes of data and be queried via a RESTful API. This video is published on youtube.com; the player here is YouTube’s. The Turkish summary and subtitles are prepared for AiPulse. Watch on YouTube AI Critique → |