Welcome, ladies and gentlemen, to Season 3 of Beers and Bites. I am your host, Jeremy Merteshaw, along with my co-host, Chris Jordan, and we're here today with our special guest Gorkham Erkand from Jozu. Welcome everybody. Good to be back. Thanks for having me. Absolutely so, as our typical process, let's start by talking about our beers that we're going to be drinking throughout the show, and we'll start with you, Gorka.

So I did a lot of research on this and decided to go with a Lager from Ontario, where I'm situated, and it's called a Fast. It is labeled as October fast Lager. I never had it before, so this is the first time that I'm having it. But October fest is upon us, so I thought it was fitting.

Excellent. Very good Chris. What do you have? Yeah, I'm not as good man.

I've got the last barrel of Middleburg Brew company and, uh, it's a double ipa like normal I.

I just just hit it hard, nice, nice. Yeah, I've got a resurgence. It's got this crazy little goat crown thing on the front. It's an ipa, it's a uh, west coast style ipa, I don't know. It's a 9%, so that should be fine. It's a good double, yeah, and we'll definitely enjoy that. So let's crack them open and say cheers, cheers, cheers everybody, and and we're off, let the games begin. So first question, gorka tell us about yourself.

And we're off. Let the games begin. So first question, Gorka, tell us about yourself. So I am the CTO of GeoZoo. Before I joined GeoZoo, I was working at Red Hat as a distinguished engineer working on developer tools, and before that I was at Nokia in Finland. Oh yeah, actually yeah.

I worked a little bit with Nokia in my past, but only in the context of the Ipso firewall from Checkpoint Wow, so that's dating me.

I was part of the now non-existent handset unit. Okay.

Nice. So tell us about Jozu. What is your company to do? What is your mission? What service do you provide?

Jozu is a company that helps enterprises adopt AI and ML into their applications. Our goal is to make it easier for the enterprises to streamline the process of getting AI and ML integrated into their DevOps DevSecOps flows, and we have an open source project that we started, started and now we're at the stage where we have released our first product, sweet, and then open source one that's kit app. That's kit apps, dot, ml. Yes, kit apps. It's a packaging, an oci artifact that is used for packaging ai and ml artifacts. Okay, so there's only a portion of the with the structure of the company's about fact that is used for packaging AI and ML artifacts. Okay, so there's only a portion of the structure of the company's about. Sorry, excuse me, is that a portion of what the company's trying to accomplish or is that just a complimentary? Yeah, it's actually like that actually comes back from the story of how we started doing Jojo. When we first started doing Jojo, we were working on something else and for that product, we had to do a lot of ML models. We had to fine-tune a lot of ML models at a very rapid speed fine-tune a lot of ML models and at a very rapid speed, and one of the things that we have noticed was the way to automate these things using the DevOps processes was not very easy.

The whole MLOps tooling space. There's actually a lot of tools out there and there's a lot of open-source tools out there, but none of them have a standard, so nothing works with each other. There are no standards. Even when you think there is something standard, there are like three, four variations of it with the same name, unfortunately. So while we were doing that, we thought that we are coming back from a developer tooling background, and then we did a lot of DevOps and DevSecOps in our lives, so we thought that, hey, you know what this is a problem. This is a problem space that we need to address.

And while doing that, one of the first things that we noticed was the packaging Data and the trained models and the code and the documentation. Those were all stored in different storages and it was very hard for someone to be able to tell oh, this was the model that this data was trained with and this was the model that this data was verified with, and so on and so forth. So we decided, hey, can we actually make this easier? Because there's something called OCI, which we have been using it for Docker images forever, and every CIC pipeline. Every DevOps engineer knows how to deal with that and every problem that you actually have today, like authentication, authorization, auditing, is actually built into an OCI registry, so you don't need to start from scratch to do that.

Can we actually do it? And we did a trial, which was the first precursor of the kitopsml, and then we thought that, hey, this may be something useful for everyone. The implementation so we did a CLI implementation. We published a specification for the OCI artifact called ModelKits and now, as part of our Jozu product line, we have Jojo Hub, which provides essentially an OCI registry with an AIML experience to customers.

So you use the acronym OCI quite a bit. Can you translate that into translate the acronym and then explain it to our viewers?

Yeah, yeah, OCI is Open Container Initiative. What it is is it's essentially comes from all the way back from Docker images. Well, we could call it well. Someone took the Docker images and turned that into a neutral specification so that other than Docker can implement registries, and also started including other kinds of artifacts like Wasm or Helm charts and, in our case, Malt.

Makes sense. It was just important that we made a distinction as we're discussing your solution, that when you say OCI, you're not referring to like Oracle Cloud Infrastructure. People say, oh my goodness, I can only run this on OCI, I can't run this on AWS or Azure. It's not really applicable contextually. So we just wanted to make that distinction.

Oracle has a cloud too. I forget about that.

Most people do yes about that. Most people do yes. But I hear Mr Ellison has got funding to build some more data centers with his own nuclear reactors, so that's going to be interesting to see.

Yeah, there's the one that just got Microsoft that was Microsoft.

Three-mile island reading game that was Microsoft. That seems risky, ominous, a little scary. Yes, russia just re-engaged Chernobyl to start further abstaining from power.

So you know, one thing is Gorky, as you think about this, is you really brought up DevSecOps and DevOps and the structure of packaging and then moving that package from research, from development, into production, right? So before we go into that, you talked about, you know, being a regular engineer, distinguished engineer, but regular as far as AI and ML. So what was that process like before? Why don't we start with that one? The life before you started working on AI and ML and then we'll move forward to what it's like after that. So before AI and ML, what was the DevOps process like and the packaging like?

So one of the things that you have on the DevOps for applications is they are very deterministic, right, you can compile the same source code with the same set of build tools and you will. 99.9% of the time you're going to get the same result, and that remaining one percent is probably npm causing something. So, um, I think that's the biggest difference when we are talking about devops processes for ai and ml. Ai and ml is is a little bit indeterministic. You can run the same code to train your AI ML model and you will always not get the same result. It's going to variate a little bit. You can't really like. There is a lot of built-in belief in the process of DevOps that these things are deterministic and repeatable. Unfortunately, with the AI ML it's not so. When you are starting to design your DevOps pipelines linear DevOps pipelines that doesn't work well with AI ML because it's indeterministic. The pipeline is essentially not really linear. It goes to training and then you know, you do evaluation and then it's another loop and it's another loop. It's the whole experimentation, verification and processes. So I think the life before AI and ML is, our first concern was, oh, can we actually build these things efficiently? And then our second concern was, oh, can we get the correct attestations and all the DevSecOps and so on and so forth? That was our second concern. Well, which should be the first concern. But now we are getting into an area where we need to be more flexible. We need workflows that actually can handle feedback as the workflow progresses and then do the next step. So I think that's the big change that I see between a DevOps workflow and an AI ML-enabled workflow, now the AI ML. That's because you're talking about changes non-deterministic Are you mostly focused on large language models you know generative AI or are you even talking like even the tensor flows and the neurons also have that non-deterministic aspect.

So we are mostly talking about the case in general. We are mostly talking about the case where you are deploying your own LLMs. Those can be open source LLMs, not the ones that you trained. You are likely to fine tune those LLMs or use some technique like RAG and also you are modifying or trading your own ML models as well. So those are the ones like, if you are using OpenAI to APIs on your application, that's not mostly a classic DevOps application flow.

So version controlling right, obviously fairly difficult when you have these big apps. That's what it should have been called. It should have been called big ass models, bam. Oh, that is kind of stupid. Rags A rag, yeah. So really, when you take a look at that model space, that's big right.

So you're basically doing like a SHA kind of do you use just a regular checksum for or do you use a SHA like an encryption? No, we what we did, okay. So when we implemented our own OCI registry, we did something so that we can calculate the checksum as we receive the file, when someone pushes the file to the registry. We want to be able to calculate the checksum of the whole file as the file comes in. So we kind of implemented our own SHA implementation where we can save the state of the algorithm at any point and then continue on. That's the data version control, like the DVC, right? So I'm going to run DVC over my basic data sides or data.

Now, what do you do with the references, the RAG? Are you also collecting all the RAGs or do you just make a list of your reference points For the RAG? So you have the LLM, right, the data modeling side, which is the generative answer, and then you have the RAG, which is the validation of fact. Is the validation of fact that data set is that very large or is that nice and small, where I can just say here it is, here's the reference, here, yeah, usually they are large, and they are very large because you know, some of these data sets are actually videos and photos and they tend to get really, really large. So, but that's just to clarify. So the RAG actually can contain video and audio and other forms, not just structural data. No, it can be anything. Yes, yeah, yeah, it can be anything. Yeah, and what we do is we also version the data together with the model, because at some point someone is going to come back and say, hey, what data did you use to train these models or test these models? Right? So it's very important for us.

Our advantages is registries are the Docker registries or OCI. Registries are content addressable storage. Therefore, we store them essentially once, but we're able to verify them multiple times. So I'm pretty curious on this. Probably you can search Rick and Jeremy, I'll guess it's hard. So so data sets okay, everybody, okay, everybody will trust me.

I think there's more fascination and misunderstanding around data sets than anything else when it comes to llm, not just the training and the creation of the data set, but the data set itself. So there's no regulations right now on data sets, correct, I mean, there's in. In other words, there's no way and I bring this up because we all know that data sets can be corrupted and poisoned. And the question is the validation of a data set, especially during the different stages of development. Is there any insurance? How do we manage a data set in the population of a data set? Yeah, so, and that's yeah. That's actually a hard problem to solve at the moment. That's why we're doing it. Yeah, like the way that we look at it is the moment that you have your data set ready.

And this data set, like the source of your data set, is very important because when you're doing training, you cannot really train out of the live data. You need to like create a snapshot of it and then use that for training, because it's expensive to train and you can't really wait for live data to come. So let's say that your data is coming from your corporate database, your customer information or something. You take a snapshot of it and then you take that snapshot. The moment that you take that snapshot, you need to protect that and verify that right. One of the things that our model kits does is because everything has a checksum. But also we sign our model kits as well, just like you would sign a Docker image. So at the places where you are storing it, there's a signature that you can verify. At the place where you're going to use it, you can use the signature to verify. So there is always an attestation of the data set. So you know that your data set has not been compromised after you created your data set.

You cannot always create the data set out of your customer or enterprise data. Sometimes you need to use datasets from somewhere else and that is where it gets really really, really hard, because someone can just go into a plugin phase and download a dataset and there is no way that you can verify where that dataset comes from. So there are, you know. I think that's going to be a large problem where you to find data sets where you can trust the data in those data sets is going to be a problem. It is a problem today. The other part of it is data sets usually do not really come with a manifest. There is a manifest, but there isn't too much of a standard added right now.

What we are trying to do on model kits in Jozu is to include an SBOM with every model kit which also includes, as far as we can go back to the source, the data set, the model and if there are any relations to it, depending on what is like, if the data set is used for training or verification that information as well and put that into an S-POM which you can attest and verify. Right, because just like a Docker image, we can sign the SOM as well so that you know where your dataset is coming from, as far as you can see, and you can use this SPOM with your existing DevSecOps tooling to see I used a dataset that I shouldn't have used in that AI model because I have an S-Farm. I can actually easily query and find it. So when we talk about this configuration, right in my brain, I have a Git repository in my head, right, I understand. This seems to be a lot more complicated than a Git repository. So you have a basically YAML infrastructure or yaml definition that explains the different components that are in the repository. So the next question I kind of have is is that you have a product line.

So we were talking about jozu. We would eventually get back to jozu. You have the, the jozu hub. Now. Is the jozu hubHub like a GitHub? Is it like a repository for media? Josuhub is not less, is not like GitHub, but it's more like Docker Hub. Okay, yeah, so it's more like Docker Hub, so you can go ahead.

Sorry, I didn't mean to cut you off. My question was you said Docker Hub, so can any company that in the world can come and submit their thing to you, and then you become the central repository for all these images yeah, yeah, they can, and they can use whatever tool they like, like cosine, to sign their model kits and include an espam in their model kits.

and what Jojo Hub does is, if your model kit is signed, it shows that this is the signature. This is how you can verify the signature. That sort of information, and if you have an SPOM, it tells you that there's an SPOM on it, and so on and so forth. So, like that sort of information is also built into the hub, so that sort of information is also built into the hub. It's very similar to what you would expect from a Docker registry with attestations and provenance support.

We're trying to do the same for AI ML.

Do you do any type of vulnerability scanning against the image? That's checked into the hub? Not yet, but that's actually in the works. So, yeah, we do want to do. We are implementing a way to run a scanner that scans the model weights and reports any kind of possible vulnerability on those. Yeah, with the.

SBOM sorry Chris. With the SBOM that gives you a list of the things that are associated with the s-bomb. Sorry, chris, just saying with the s-bomb. Even that gives you a list of the things that are associated with the image. So that should be an easy thing to do a cve look up against yeah, well, the cv database for ai models is very limited.

So that's one problem. And also the model scanner, what it does. Is it actually there? There are certain vulnerabilities in the AI serialization formats. So it basically scans those AI serialization formats and says, okay, yeah, there's something here. Like, for instance, pickle has a way of running code embedded into the AI model, code embedded into the AI model. So it basically scans that and says the pickle serialization has some code in it that will be executed if you run this. Typically, the scanner does that. As you said, the SBOM is just like the regular application images. The SBOMs are a very good way of identifying CVEs, but in this case, the CVE, the number of CVEs are limited, so it will take a little bit more time before it becomes a really useful feature for the enterprises the enterprises With KitOps.

am I able to make a um? Do you work or do you support Jupyter Notebook? Are you able to turn those into?

artifacts. Yeah, we, we do something even more interesting in my opinion. So if you have your Jupyter Notebooks and if you want to package them with your models and model kits, you can definitely do that. We actually have a code, artifact type or subtype on model kits which you can put your Jupyter notebooks into. But the other thing that we are doing at Jojo Hub is we are providing what we call microservice containers. So let's say that you have an LLM model that you have fine-tuned and you put that into a model kit and push that into a Jojo Hub. That's all your data science team needs to do, and after that, jojo Hub gives you an endpoint, a URL that you can put into your deployment YAML on Kubernetes or your Docker run and you just receive a container.

So where did you add that feature? That's an interesting one. Is that recent or it is? Well, it's about to be released. Let's say Breaking news. This is just breaking news. Yeah, I don't know what I was actually supposed to even say about it, tell about it. But yeah, we got a monologue. We have been testing that and it is a very cool feature. So it basically gives you an OpenAPI-compatible microservice for any kind of model that you have pushed as a model kit to the registry.

Yeah, microservices has always been an issue of how do you track those, you know. So that's very interesting. You say that it is and kind of the inspiration like when we started this, we talked with a lot of data scientists and one of the things that they hated doing was to create these Docker images, because that's what DevOps engineers and application engineers expect. They don't know what to do with the serialized model, and putting that into a model kit is not an issue and that's when their job is done. So after that we can provide the microservice, because that is becoming more and more standard for things like LLMs and transformers and whatnot. So I came up with this solution. It kind of happens a little bit magically because you push a model kit and then on the other side you're receiving a Docker image. But it is a pretty cool feature that at least our team has been having a lot of fun with.

So one of the other things that you know it did kind of in the same realm of things that are hard to measure and hard to control. So in the LLM especially in the LLM, as I'm running my model and I'm having my data, the LLM changes right. So are you looking at, how do you measure drift and is there a way in which you can control drift as far as against the initial model? I mean, how does drift play in the way that Joseph is looking at the world right now? So one of the things that we started doing again with our microservices is we looked at this world of LM monitoring, or model monitoring, not just LLMs but actually the ML models. Drift even more was, you know, the monitoring stacks were not what everyone is used to and we started looking at oh, can we actually do it so that we use the regular open telemetry to collect these data and then present that to a monitoring stack? So that's what our microservices comes with, so you can use an open telemetry endpoint to collect the data. At Jozo we are not really providing a way to detect the drift per se, but once you have the data in place, there are other tools that you can integrate to that stack that will help you with the drift detection. So that really brings us probably to the last part of the data model. So we have the basic data model. We talked about the RAG part.

The last piece I want to talk about is the guardrails. So, as you're developing the structure, obviously there must be a section for guardrail articulation, and is there a way to measure when the guardrails actually get hit? In other words, is there a way, when the system runs, you're monitoring the stack? Can you monitor when guardrails have to be kicked in? Well, it depends on the guardrail implementation, I assume. So that's not Interesting. They differ from different models like LAMA, they differ from Stage and stuff like that. No, it's actually the technique, the guardrail technique.

So guardrail is a concept that is that includes multiple techniques for, for uh, controlling the outputs, right, so? And some of that can be just the, the uh, the input. You know, it may be just at the input time, that it may be just cleaning up the input, or it can be that output needs to be in a certain format, in certain length, and so so forth, that sort of thing. So it depends on the technique that you are using and in some cases, yes, there are statistical techniques that are used to say this is not a good answer and let's give it another try. Actually, some of the techniques are that it's like this is not within the acceptable variation, that it needs to be, and then I'll just give it, send it back and ask again. And then there are other techniques where they actually don't like the output and actually send it to another LLM so that it can be refined. This is an interesting thing because obviously we're two security guys and maybe you can tell us how to defeat guardrails.

We have to know how to break it yeah.

Now I understand the beer part. So are there like tests for guardrails, like structured tests for how to defeat guardrails, or is answer this, like your grandma would tell me a story, something that's not really built into the tests? So there are these red teaming LLMs. So you're basically training an LLM to be a RIT team for another LLM and use that to measure your guardrails. Essentially, you're getting fascinated. I was going to joke and say why don't I just have the AI do the packaging for me? But now you're talking about so there are tools, there are AI tools to test AI. Is that what you're saying? Yeah, so there are tools, there are AI tools to test AI. Is that what you're saying? Yeah, there are RIT teaming, llm, rit teams that will test other LLMs yeah, interesting, jeremy.

Ideas are fluent, so let's take a step back Talking about Josu. Tell us about your business model and then, if I'm an LLM developer, do I make money with you or what? What is, how does that work?

you mean by all?

I'm maker you mean like, let's say, let's say I'm a father, yeah, or a mama or whatever, right, if I put my, my product into your and if I put it into your hub, do I get subscription revenue? Do you get subscription revenue? How does Josie make money and do partners make money with you?

No, we don't. We don't do. We don't. We do not charge for public repositories. We do charge on private repos of private repos. So if you want to keep a private repository on Jojo Hub, then we charge a monthly fee for that, and if you are a model provider, we do charge a fee for verifying that you are that model provider. There is a trust relationship that we need to establish. That comes with certain costs, so we do charge for that, but other than that, we do not charge for if you have a public repository, people can download from it as many times as possible and you can have as many public repositories as you want. The business model is more about enterprises and that are hosting JojoHub as well. So we do provide an on-prem version of JojoHub which will work with your own OCI registries if you have one, and if you don't have one, we'll happily give you one as well, and so we do a SaaS business, but we also do an on-prem Georgia Hub for enterprises that are trying to adopt ML and AI into their business.

Interesting. We referred to in the really early part of the episode the words distinguished engineer. I want to circle back to that. How do you become a distinguished engineer and who assigns you that designation?

I guess every company has its own way of doing it. I was a distinguished engineer at Red Hat, so the way that it works at Red Hat is there is a central distinguished engineering committee I think they have a better name than what I just articulated, but they're the central group that is compromised of representatives from different parts of the company and once or twice in every year they come together and then there are candidates that are brought into the committee. Paperwork that is done to show that you have been doing what RETAT considers a distinguished engineer level work, uh, and they decide if you are, uh, you have been fulfilling those and decide whether you are a distinguished engineer or not. So this was the last. Engineer is a very special level at Red Hat, so it's.

Does that mean you're super focused on like one application or one business niche, or that you have the knowledge across the entire spectrum of Red Hat's bazillion product lines?

So well? That's a very good question and it's been repeatedly asked inside Red Hat as well. There are some distinguished engineers at Red Hat or probably anywhere else that are very proficient at one area and they are focused on one area. They probably invented some of the things that we are using today. And then there are some other distinguished engineers that are more generalists, but they have done really large impacts on the business. That doesn't mean that they are not technically capable. They are certainly technically capable, but they are more generalists that they are able to do business-wide impacts as well. So there is a mixture of it in my opinion, and in my case I was developer tools. I have been doing developer tools for 20 years, so I was kind of in that narrower lane. But there were distinguished engineers at Red Hat which have been more generalists as well.

I think that between the Lambas and the Geminis and stuff like that, there's been a lot of progress on these other models. And that does bring me to an interesting question about Jozu. All these different models are out there, all these programs that are out there. How do you handle so many variations in your platform? Is it just a configuration change, or do you favor one over the other, because I noticed, like there is a comment like you're better off with a Mac, with the kit ops, than other operating systems. You mean for different models. Yeah, like you have the llamLAMAs I just brought up, like you have your LLAMA, you have your AI. It's actually a little bit more as well.

So what we do with Jojo is we have a whole pipeline process behind the scenes where we are working with models and downloading them and so on and so forth. And the other thing that we are trying to do is there are different serialization formats. For instance, there's a format called ggml or ggov files, which some of the LLM vendors publish in, but not many. And then there is another model called ONNX, a model with serialization format called ONNX that Microsoft definitely loves more than anything, and they publish their models on ONNX. And then there are safe tensors and so on and so forth, and the enterprises actually have their choices. Some of them favor ONNX and the other ones favor SafeDecs and so on.

So when we are packaging models, we actually package them in a way that we can provide three different variants of the models. So it gets a little bit more complicated than that as well, than that as well. And then the other thing that we do is sometimes we do quantizations with these models so that if precision is not really you don't really need precision you can get maybe smaller resource usage for the models and almost get as good results. So there is a whole process that we have built behind the scenes that kind of takes these models and massages them if needed. We don't really have a favorite. I do like the GIGA format because of its practicality, but other than that, I mean, it's not really. It's not really a preference for the technology itself.

Well, this has been great. Gorkham, I'd really like to thank you for joining us on the show. I think that we've all learned a lot about what you've done in your past, what Jozu has to offer the marketplace, and we learned a lot about AI, and I want to thank all of our new coffee sponsors who are going to sign up after this, and thank you to everyone who watches our show. Make sure you like there's a button over here and subscribe there's a button over here somewhere and we'll see you on the next show. Thank you.