- Welcome everyone, this is the Digit Podcast.
- I’m László Személyi, managing partner of Future-NOW, and my partner in crime is Zoltán Szekér, founder and managing director of OD&IT and Sailing Hangar. Hi everyone, good day!
- The goal of the Digit Podcast is to help those who are not satisfied by superficial articles and catchy slogans get closer to today’s defining trends in digitalization and information technology.
- We’re going to dig deep into things here.
- Over the past two years, almost every conference has featured the same sentence: AI is changing everything, and in fact maybe within 3 years. Yet I’m hearing the exact opposite more and more often from experienced representatives of the profession. For example, in connection with cybersecurity, that the logic of attacks has not changed, only their speed and volume have, and the fact that today almost anyone can become a cyber attacker, it is so easy to start down this slippery slope. Today we’re going to talk about which of the two is true, and elsewhere as well. And what does it even mean for an artificial intelligence to be trustworthy. Our guest is Tamás Tusor, head of information security services at Tigra Zrt. Welcome to the Digit Podcast studio.
- I warmly greet the listeners, and greetings to you as well. Please allow me to clarify that title a little.
- Well there you go, I’ve already missed the first point.
- It’s not that big a mistake, what happened. Actually, the situation is that within Tigra several business units operate, several larger business units, and the company is mainly engaged in development. Judit Alföldi leads our business unit, this IT security services business unit. We basically provide services to external partners. And Judit asked me to take on the honorable position of being the professional lead of this business unit, so in fact the correct wording is that I am the professional lead of the IT security business unit.
- Congratulations, and thank you very much for coming.
- Thanks for the clarification, and also for taking on the challenge here in the studio. Now, you’re really not the first expert from the Tigra team to sit down with us, and I think that says something about how much the company has to say on these topics in cybersecurity. I’d start with international trends. The application security community, OWASP, could perhaps be called a reader. Last November it released the first version of its AI testing guide — note, the first version last November. In it they no longer talked about safe AI, but about trustworthiness, trustworthy AI. What interests me is whether, with this slight difference in wording, they were admitting that cybersecurity in the classical sense is not even really applicable to these systems, or whether the terminology is simply changing but in fact we are talking about the same thing.
- I think the answer is yes and no, which would be a good canned response.
- I think it’s roughly about the fact that our thinking about security in general is undergoing a paradigm shift. Not only in the field of AI security, but I could say the same about IT security, and likewise about OT, that is, industrial security. We can no longer think only in terms of these separate closed areas independently; rather, with a paradigm shift we are now stepping up a level in this, and, to use a somewhat esoteric word, as they say, we need to examine the issue of security with a holistic approach, not just area by area. Because these are very closely interconnected. By now our networks and systems are so interconnected, increasingly with AI systems as well, that these systems have a very large impact on one another, and their security also has a particularly large impact on one another. The relationship between trustworthiness and security can be described as follows. These AI systems are basically becoming increasingly complex, of ever greater complexity. It is increasingly difficult for us to see through exactly how decision-making takes place inside the interior of an AI, and because of this we now say that these systems do not operate deterministically. Previously, we thought about security in such a way that we set some security level using our usual methods; this security level can be maintained more or less stably; vulnerabilities come up from time to time, we can detect those vulnerabilities, we can handle them, fix them, or mitigate them, or simply accept the risk arising from them, but the point is that after a while we always return to the same security level, or we may even be able to improve it. AI, with this non-deterministic mode of operation, has introduced uncertainty into this whole picture. We cannot always clearly say whether it will react the same way to the same inputs. Because of this, we now examine these systems not only from a technological and security perspective, but also from the perspective of trustworthiness. Which, incidentally, is nothing new under the sun, because trustworthiness has always been a key issue in information technology, and there is already a fairly mature science of how we can examine trustworthiness.
- I think now we should take a bit of a step back. We’ve jumped ahead — practically this is the end of the conversation — to how we examine an AI. Because we’re at “State Treasury 69” and “ABC 1-2-3.” That’s where we start, and then this is an extremely important institution, one that from a national-economy perspective must comply with at least 3–4 standards and can be mapped to a risk management standard, where patch management did not work, alerts were coming into the Security Operations Center, but nobody dealt with them. And by the way, if we think about the DORA-style password policies, then what of these was actually implemented? How could we achieve the kind of risk-based thinking that is needed so that an ABC123 is not in the system. Let’s not talk about AI yet, not just yet.
- Yes, actually with my colleague Ákos Solymos, who was also your guest recently, we talked several times about a similar topic, and when it came to AI we always ended up concluding that, basically, IT security still isn’t in order. So, we still haven’t solved IT security either, that isn’t in place, it’s in the same kind of state you were just talking about, where at a state treasury patch management, and completely trivial things, hardcoded passwords, and things like that still come up. Of course, making mistakes is understandable, but the fact that these things are still with us. Letting AI into this kind of environment, and then expecting AI Security to solve these things afterward, seems like a windmill fight to me. So in general, and I have a strong opinion about this, we are in trouble with AI because Security is not in order. And that’s what needs to be put in order.
- I feel a little, forgive me, that you are in the same situation, and I say this as someone who is not an IT security or cybersecurity expert, so I’m looking at you from the outside, like Semmelweis was in his own time, saying hands need to be washed, and then how do you explain it sixty more different ways, because they still don’t wash them, in fact they even want to throw you out the window for what you’re forcing the colleague into.
- I think, and here, since after all we are colleagues in this sense, I usually say how many difficult moments we experience on the leadership side when you try to explain this handwashing thing. And they say it’s fine like this, it’s been working this way for ten years. So let me give a precise example. Sunday phone call, they’ve been hacked. I tell them, let’s go through it, filter it out, shut it down, change the password, the usual protective options, then the colleague said okay, I’ll call you back when I’m done. In the meantime he calls at least four other people. All four people give him the same advice, then after that he calms down and thinks no major problem really happened, but technically he changed nothing, and then after that, when I say at least there is antivirus protection on the machine, he says there isn’t. Because there hasn’t been any for ten years, and ten years ago he switched to Mac because that’s good, and there are no such attacks. That’s where we’re starting from, and I wanted to convey that back to you.
- Yes, it’s a very deeply ingrained basic assumption that the weakest link in security is always the human being. I would challenge that a little bit, I’ve already seen the opposite of it, where the human can be the strongest line of defense. In places where the proper technology is not there, the proper resources are not there, but the colleagues — especially, in fact, in a small company there would be a real chance for this because there are so few of them — could be trained to such a level. It would be much easier to train them to a level where they are independently resilient on their own against a very large part of attacks. And I have already seen an organization like that. Actually, I’ve seen it even in a larger organization, where the colleagues’ awareness, their security awareness, was such that I was present at that company as an external expert. In principle, with a guest card I was always supposed to have a host accompanying me in the corridors, but because of the nature of my work at the time I would always sort of wander off, and then I really would drift around after my colleagues, and almost every time after 10–15 minutes someone would come over saying I must surely have gotten lost and couldn’t find my way back to my escort, so if I told them the person’s name they would take me back to them, and they regularly escorted me back. For a very long time.
- This Mr. Naworks at the terminal thing is amusing, with someone arriving and being escorted back, but technically I once carried a server out of a large company simply by saying I came from the company and that I had to take the server away, and specifically, take the receipt into the thingamajig, and they helped me carry it out. I put it in the trunk, then I even waited another half hour, because by then I had a meeting with the CEO, and nobody really made much of an issue out of the fact that the company’s central customer registry was missing, because I had managed to take that server away. Production kept going, but the fact that for the office staff — let’s be clear — things like Excel had stopped working, they were so used to it, and they had already become so resistant to something not working, that basically nobody even noticed. And how can we get from this way of thinking — I run the company from a Gmail address, with a Google Calendar linked to that Gmail and an Excel file that may or may not even be from an official source — to any kind of awareness? I know, and now I’d like you to scale this, because they always say that you only deal with large companies. But this is an example of a small company where they literally handle 7 billion in turnover, there are four of them, with two computers. And why do we think that a small company and a large company are so very different?
- A small company differs in that in a small company it is much simpler to maintain the human stock — now that sounds quite grim — but to maintain and develop the human stock is much simpler. There you can truly tailor it to the individual; personalized education is very fashionable in education these days anyway, which works really well, but in IT security awareness training I think customizability is especially important. And this training has to work not just as an I tell you and you understand kind of thing, but it has to be reinforced with active elements, even practical tests and role-playing or situational exercises, all the way until this resilience becomes a skill in the colleagues. I think this can work very well at smaller companies. And then with this we can also replace a great many technical controls. Obviously, by the way, this would not stand up in an audit, because there the auditor will expect the actual presence of technical controls, let there be this kind of strict firewall rule, that kind of strict one, let everything have spam filtering and endpoint protection, of course those should be there. But a human can manage the shortcomings of these things very well. Unfortunately, this is usually the very last thing they develop. It seems much simpler to buy a firewall, sometimes even to build out a whole stack without actually putting any people into it, but they build the technological part. In a great many places I see that the SOC is present and is screaming bloody murder when the problems come, but nobody deals with it.
- There are no people.
- We already mentioned one such example earlier. Now let me be a little bit of the devil’s advocate, because these are good answers, so I definitely don’t want to fall into the trap of nitpicking them. But beyond the fact that at many companies we still haven’t solved the problem of Gmail, passwords and backups, and security awareness. The AI solution waiting to be integrated — or already integrated — is also already here. It contains prompts, it may be reliable in some ways, but in any case it is definitely not deterministic, and some kind of answer has to be given to that too. And then a tricky question came to mind, namely that I’m now going to package this for you, because after listening to you for ten minutes, I’ll be able to package this for you in a way that gives a meaningful result — what are the vendors saying, and what should we think about it? Now without naming names. Because for a great many manufacturers, their communication, their materials, their presentations, and practically their value proposition as well are absolutely crawling with buzzwords, full of this flaunting of immense power and knowledge, as if they weren’t even speaking to the same customer as you are, the ones we’ve already talked about quite a few times earlier. Is it really useful that manufacturers show up with all kinds of flashy materials and all kinds of hyper, bombproof solutions, while maybe they should be starting with the simplest things. Does what manufacturers are offering in this area help solve your task. Or do you have to solve the task on top of having to manage these manufacturers?
- For me, this is basically a difficulty, because what manufacturers are obviously striving for is jumping on the hype train and happily waving their hair in the wind.
- They don’t even bring the substance, right—is that how you say it?
- Well, that depends.
- Yes.
- More or less, yes. But the point is that obviously this is what customers are demanding: now everything is full of AI, we fill every lot with it, everybody wants AI at home, and for the company too, and everywhere, and I want to optimize and improve all processes with AI, etc. Obviously there arises a need to somehow rein this in, and then surely AI-supported defensive tools will be the solutions for this, so manufacturers too are, one way or another, fighting tooth and nail to be able to slap the AI badge on their products. Then of course afterward they manage to generate a lot of buzz and advertising and try to hammer it home on every platform that their stuff is boosted with AI, but at the end of the day I think these tools are not worthless at all, not in the least, but they are not valuable because AI was put into them, they are valuable because they bring in that security which was already a known security solution. So they bring in and use the same controls, the very same ones. As for building entirely AI-based defense systems that make autonomous decisions and actively intervene, I think we are still far from being able to allow that safely. I’m not saying that some people aren’t experimenting with this, but I think it is fundamentally irresponsible to build automatic intervention into our systems in this way by entrusting it to AI. So in my opinion it is useful, because in the end those security solutions do reach companies, it’s just that now there is a new marketing tool, which we call AI, and that is what carries these tools in.
- What I find interesting is, say, here’s a firewall, I build AI into it. AI hallucinates. Now, in the firewall, isn’t this AI going to hallucinate? So do you understand what my fear is, that there is a product we know hallucinates, I build it in, this is the excellent “can I eat this type of mushroom?” Sure, you can eat it. Oh no, I died, this was a poisonous mushroom. Oh, sorry, you’re right. And so this kind of AI in a firewall, according to the dream book, may not necessarily mean anything good. What’s your view on that?
- There are methods for mitigating AI hallucinations. They cannot filter them out completely, but there are various tools with which this can be kept at quite a low level, yet the decision-making itself still does not become something we can interpret deterministically. So it may be that for the same type of attack, once it will react by saying no problem, I blocked the IP of the one who wanted to attack, and it’s solved, while in another case it will say we’re under terrible attack, I’m shutting down my server. And then that is immediately a different quality of decision, because it has disrupted my business operations. Whereas the previous one probably did not. That is why I think automations of this type must not be allowed without human interaction. This human-in-the-loop methodology is what would prevent this. Obviously above a certain intensity we cannot put this in place, so you can’t have 35,000 analysts standing next to a high-traffic server, looking at every single line of code every second and deciding whether this can go or cannot go. We have to find the right priority, but I think entrusting the decision purely to AI is not yet possible, though we have had solutions for this for a long time. So these attacks, vulnerabilities, and risks that AI has brought in are not new. Each of them has its counterpart and its proper handling method already from the earlier world of IT security. It is the volume and the intensity of the whole thing that really makes this problem so big, and that is what we have to be able to handle now. And that means that the IT security issues we have not solved up to now must now be solved, and much more strictly, much more comprehensively, if we are to withstand these big waves.
- Well now I’m going to hold you to your word. You said there is nothing new under the sun, these vulnerabilities already existed. I think there is, for example, a new one that did not exist before, and it stems precisely from the same architectural issue that hallucination stems from, namely prompt injection, which is about the fact that if I place certain words or certain instructions well in a prompt, then I can induce that tool, that AI tool, to engage in malicious activity. In fact, I think there is an even more advanced version of this, indirect prompt injection. In fact, when the user doesn’t even need to access it directly, doesn’t even need to encounter that interface, but some hidden instruction from an ingested source, in a document, on a website, an instruction hidden in white letters, can also trigger this effect. Now this, just like hallucination, cannot be eliminated completely either, because then I have practically stopped the operation. But for this we do need some new solution, or to build in some new security element, don’t we?
- Yes, and also no. My opinion remains that there is nothing new under the sun. At the end of the day this is an injection attack, which we already know: SQL injection, prompt injection, same thing.
- Not the same thing, very similar. Code injection is also very similar to this. An alloy of these is practically what we now know as prompt injection. But the solution is the same. So if you take a pentester who already has routine in uncovering various vulnerabilities and working out how to mitigate them. If you give a pentester the task, “here’s another injection, what should be done about it,” then even without necessarily interpreting exactly what it is, they will definitely blurt out that input validation has to be done, data filtering and data sanitization have to be carried out, and this has to be applied to all data that comes from the client. What is painful about prompt injection is that, in the case of prompts, code does not separate from data, because practically speaking, if you look at it from here, then everything is just data, and if you look at it from there, then everything is just code. Because of this, large language models themselves in isolation, so if you specifically look at the model core itself, cannot defend against this. It is impossible, because they cannot tell the difference, because for them these are all just statistical features in a language. The solution to this is that we have to build a layer around it that can filter the input. And obviously our solution will be that we have to pre-filter everything we give to the AI. There can be many kinds of solutions for this, but none of them is really good.
- Unfortunately.
- They all hurt somewhere?
- They all hurt somewhere, if you think about the fact that there is this trio of, say, security, convenience, and functionality. Out of these three, you cannot maintain all of them in parallel.
- You can only choose one.
- You really choose one, and that will pull focus away from the other two. If security is key, then you place your model, the AI part of your whole AI application, very far from the user, and you put a lot of layers in between so that data can only reach it in a very controlled way, preferably already heavily pre-filtered. But with that, the system loses a great deal of its flexibility. And if it does not matter, because I hear this from many people, that “it doesn’t matter to me, security doesn’t count for me, what matters to me is having a working system very quickly,” then the user, or in some cases the organization, may decide that they will weigh the risk, but here too it is important that this whole thing has to be done on a risk basis. So if they decide that they are willing to take this risk, then they take it. That is not my business. As a consultant, I tell them, I call their attention to the fact that this is an enormous risk, sometimes such a great risk that it is hard even to assess, because it may have effects beyond them, but if they take that risk, then who am I, I cannot do anything to prevent it. But the point is that the right balance has to be found within this trio.
- That was a very important sentence here. In security, who am I? That is the question it raises. Who am I? In a way that, in an organization, finance is there as a decision-maker, it can introduce it and kill it and say yes to it. HR is there as a decision-maker, it can kill it, introduce it, say yes to it, and think about it, résumés. Résumés such that if one gets in, through that the attacker, this is my favorite, and I am going to name him, sorry, the ethical hacker colleague is called Matek Camillo, and in his presentation he likes to conjure up a PDF containing a Doom game. So we send a PDF into the organization, apparently a CV, while meanwhile the attacker is playing Doom inside the server, and this is actually one of the better cases. And here, with this question of who am I. You phrased it in such a way that as long as there is nothing new under the sun, and as long as the basic IT questions are not in order. Now, could you help with saying what maybe five or ten things are that are mandatory, when the answer to the question is not “who am I”?
- Who am I? I ask because as an IT security person I am not a determining factor in the company in the sense that, no, the company does not exist because of me. I do not produce, I do not run the business, so there are very few companies that exist because of IT security.
- No, no. So what I wanted to say by this, and this is very important, is that we always move into defense mode.
- Yes, yes.
- Cybersecurity’s job is not to defend. We have to protect the system. I just want to ask: from what? Because every single time we run into a management discussion where they think we are sitting around uselessly and spending the money, while they themselves do not know how their own business works. We, as cybersecurity professionals, are supposedly required to know how to plan a budget for them for cybersecurity. At a time when, by the way, I would say that the cybersecurity cost should be, say, 1.0% alongside all annual after-tax positive results of any kind. I am asking for this to be spent. That is my only sentence. So I am with you, I think we are doing the same thing, and what I would like to bring in here is that awareness that okay, they do not invite us to the board meeting, and they cut the budget, my favorite example being when they cut the generator. Here too, security has several layers, so physical, organizational, etc. They shut down the generator, but a power outage is going to occur, and I put the generator back line by line, component by component, and then we are still trying to prevent a power outage with it. So how should we try to make a proposal to operational management, or to strategy or vision or anything, that in Cyber Security you should weigh these things like this on a risk basis for the sake of business continuity.
- My short answer would be that IT risks, and basically technology risks, need to be translated into business risks, and that is the language the decision-makers on the business side understand. This almost always works. It is not that trivial, because it requires exactly what you said: we are not necessarily the ones who understand business risk, we mostly understand technology risk, but I think that within an organization you can always find it. The larger the organization, the easier it is to find those transformer people who are able to switch communication between the two worlds. By the way, for example AI can also be quite good for this, I am just noting that quietly, but what I usually suggest are the same basic principles that also apply in the living world now. We cannot achieve it, it is practically impossible to implement zero trust perfectly, but we have to strive in that direction. Let us reduce permissions, keep these under continuous control, pay attention to endpoint protection, pay attention to our leaked passwords, train our colleagues, we have already discussed this in the previous sections, these are the solutions. But proportionate to risk, we cannot sell it to the business areas that they should sacrifice not 1% but 30% of the budget, because it is pointless. The company does not exist to implement IT security, it exists to manufacture screws.
- And how can, for example, a four-person company choose backups for itself. So when we say defense in depth, then within that depth, as part of multilayered protection arranged in depth, there should be, say, a backup. Because that is our farthest and most important line of defense, right, that we should be able to recover; is that important from a business perspective?
- I think that at such a small company it is especially important that they can no longer afford to waste their already small headcount on having one of them be a security person. So I find it hard to imagine that they would want to solve this well in-house. If so, then someone puts on that hat and tries to carry this out. I can say the same here: they need to analyze why they want backups, whether they even need backups at all. I met a design office that said it did not matter to them at all. The only backup they need is enough to make sure the blueprints they are drawing are not lost, but from the moment they hand them over to the client, they have nothing to do with them, they are not interested. They do not even care if someone comes in and reads them. Nothing matters. So their only requirement in terms of availability is that during those, I do not know, few hours while they are drawing that drawing, I do not know how long it takes, never mind, apologies to the architects if they take this personally, I am not trivializing it, so during those hours while they are working on that piece, it should not be lost. That is all. So it may very well be that they do not need it. That is why I say protection has to be proportionate to risk. The other extreme is also possible, where they work with super-secret data and there are business secrets involved, then again I think an expert should be brought in who can tell them what the appropriate protection is at their level. I would not like to state some universally perfect solution now.
- No, no, no, absolutely not. Rather, what I wanted was what you answered: that they should try to understand their own risks, whether it poses a problem if, for two hours while they are drawing the important drawing, their computer does not work.
- That is a business question. So that is what I was trying to get at a little.
- A four-person company has to think this through in the same way, I completely agree. We are at halftime. At times like this we usually interrupt the increasingly dense thread of the professional discussion, and in fact we usually ask a completely unexpected question, a kind of personal thread, so that we can diverge from the questions we asked our previous Tigra guests; this time I brought up summer jobs. What was your first summer job? Did it pay well, what did you spend it on, why did you go do summer work?
- Yes, my first summer job was very exciting. I remember that it paid well. As for what I spent it on, I cannot say exactly anymore. I went because basically I did not come from a well-off family, and summer was anyway a kind of dead period, because we did not vacation all summer long, so I had a lot of time, I was bored, I must have been around 11 or 12, and I did not get an unlimited amount of pocket money unfortunately, so this was the period when I could pull myself together financially a bit, and then it lasted me for the whole year. That was roughly what it was used for. The first job that could really be called a summer job was that I got into a bricklayers’ crew. It was an eight-man bricklayers’ crew, I must have been around 11, and the special thing about this crew was that at age 11 I was the only member of this crew who had not spent at least five years in prison.
- Aha, right in the thick of life.
- Yes, yes. It was actually an incredibly good experience, and I cannot say anything negative about it. I learned a lot of things; jokingly, the first thing I usually mention is that I learned how to open a beer with anything, absolutely anything. That was the first thing they taught me; when they did not think I could carry the cement bag, I should open their beers for them, and then later they involved me in other jobs too. But what was truly valuable in this was that many people would say these were not good people, since they had been prison inmates, one of them had spent twenty-something years in prison. Obviously, it could have been because of a fairly serious crime. Still, what I had to come to know and see there was that these people—I basically can’t say about them that they’re bad people, they are not bad people—they did stupid things, they did bad things, they made huge mistakes, but that still doesn’t stop them from being human beings, and anyway by now they mostly live decently, as best they can, but these were mistakes, they did not truly become bad people because of them; I’m not saying the things they did were wonderful, because they weren’t, because obviously they committed crimes, and that is bad. But there is always a human being behind it. And well, they made mistakes. This happens very often in the profession too, by the way, people make mistakes.
- Zoli? Summer job.
- Oh yes, I worked as a lifeguard. This question is interesting from this perspective now, and the way you compare it, because the other day. As you know, I’m from the Balaton area, and the other day on one of the forums a dear tourist raised the point that in a second-level storm warning why does he have to leave, since he paid the 2,600-forint entrance fee, and that he is entitled to get that back. Obviously I’m not the right person to answer that, probably Sándor Bagyó is, and I value the work of lifeguards tremendously, and I wrote there an empirical experience of just how awful it is to have to tell the family sobbing behind you, while someone is drowning in 40 centimeters of water, that unfortunately they have just lost their loved one. And then various trolls and attacks came in on this, saying the cobbler should stick to his last, and Hujjajt too, why is he talking about being a lifeguard? Well then, let me give my résumé here now: I did work as a lifeguard. I think for roughly six years, and after all that I also helped the work of the Hungarian child rescue service for seven years. I think we can learn very important moral things from this about how difficult it is to remain human in certain situations. Laci?
- I smiled broadly when Tamás started the story, because our first summer job was the same, I was a mason’s assistant too, only at a much older age, I went in the summer when I was 17. I remember exactly why. Just the same, financially our family wasn’t exactly rolling in it; let it just be noted about the family story that even years before the story we had to sell the little Polski Fiat because it was expensive to maintain. So, in that situation I wanted a mobile phone, I don’t even know, well, this was the ’90s, so not a smartphone.
- A good Nokia 30-30.
- A mobile, and I think it was actually an Ericsson that I wanted to buy then, and so I was saving up for that, and then I went, and in the morning I got on my bike and biked from Fót over to the outer part of Dunakeszi. I didn’t have a crew, there was an older man who was a mason, and I was able to hire on alongside him; actually I found him through one of my teachers. What I took away from that period, besides the fact that you can improve your physique, though I do have my limits, referring to the garbage bag, was that the old man had a catchphrase. That’s where I learned that such a thing exists, that there is such a thing as a catchphrase. We work from morning till evening, and when you hear the same thing for the twenty-fifth time, then something is already not right there. The old man kept saying, poor man suffers, suffers. That’s from Madách’s The Tragedy of Man, so in its original context it’s actually a wonderful sentence. During masonry work, by the twenty-fifth time it isn’t, and there I had to realize that at 17, as a teenager, there are choices about what kind of life I want to live, and it doesn’t depend on what the circumstances are like, it is decided here in your head. So besides the fact that I did get a mobile phone, I think this experience was worth more, looking back. But back to the last, since the lifeguard thing was actually the last too, only the troll didn’t know it. Attacking side. As you know, there is a defending side; now this is half an hour tougher.
- Thrones.
- Attackers, yes. The average length of malicious code, by the way, is supposedly 125 lines; I don’t know where that comes from, but it’s very short, while a modern software can have code running into several million lines. So there’s this kind of imbalance, and as a layperson it’s quite a sweaty read at first. But isn’t it precisely the shortness that actually helps in this, and when we talk about agents now in connection with AI, then with these attacking agents is there perhaps some new trend that changes these previously, say, easily identifiable short codes.
- This imbalance really does exist, and it’s actually very strange, but this strange imbalance is an advantage for the defending side, because our operating systems, our defensive tools, our firewalls, and everything are enormously robust. These endpoint protection tools too can run into the millions, or even tens of millions. The attacking code necessarily has to be short, because it has to slip through mail systems, slip through firewalls, traffic monitoring, and everything. In many cases it cannot contain real binaries, but instead has to operate only by quasi scripting the existing tools, and that is why it has to be very short. So if the defender finds a file and starts analyzing it, they can very quickly analyze that given malware, that given malicious code, from top to bottom, get to know how it works, and very quickly respond to it and mitigate the effect of that malicious code. This works really well until we put the AI idea into the whole thing. And we might think that because its operation is of enormous complexity, the code it produces itself will also be very large, very complex, but it will not be successful. Because if it does not obey the same rules that must be obeyed as an attacker—remain small, remain unnoticed, remain under the radar—then it fails. That is why AI too is forced to write short, substantive code, which is just as easy to analyze, even with AI support from the other side. Because of this, it does not really become extra attacking potential. The volume, however, changes here too, so they can generate far more such attacking code, and there is nothing left, because every attacking code will be unique, so on a signature basis, that is, based on what the attacking code looks like, we cannot identify that this is attacking code, but must definitely analyze it based on behavior. If you allow me, in the meantime I’d bring in something related to this, very closely related to this, which I don’t know if you know, that there are access brokers who only take attacks as far as Initial Access, that is, obtaining access, getting in.
- Getting behind the gate.
- Yes, they carry out getting behind the gate, and then they put that access up for sale, and then other hacker teams come in, and they’ll do the rest of the work. It was roughly nine hours, this was in 2022. From Mandiner’s recently published study, we can see that by now this time has shrunk to 22 seconds, 22 seconds. Obviously because the whole thing is well automated, and the exchange happens immediately. So from this point of view, I think we should be more worried about the presence of AI and agents than about the fact that they can write really good attack code.
- I’ve had this on my mind for a long time, that the hacker is the service, so nothing really protects against that. Laci?
- There is definitely no level of security spending that reduces this to zero. What occurred to me is that it might be worth digging a little more into this SME-versus-large-enterprise dilemma from the attacker’s point of view, because large enterprises really do belong among the more prepared clients—there are exceptions, but in many places they at least have greater resources—and so a kind of jargon has developed there: there’s red teaming, which in Hungarian I think should be called attack simulation, they build testing into the development process, they use retrieval-augmented AI systems based on their own documents, so they really do have this kind of toolkit and technical vocabulary. But what is it from all this that trickles down, say in Hungarian terms, into the life of an SME? And incidentally, are we in Hungary behind or ahead here compared with other countries on the global market? Because there are things in which Hungary is technologically damn near the forefront, and I’d be very interested to know where exactly we stand in SME cyber defense.
- Well, you asked a lot of things at once, I mean a lot came to mind for me about what you said.
- That was the goal?
- I’ll start from the end. I am convinced that the reason news like this isn’t swarming all over the place is not because our defenses are actually so strong, but unfortunately because we are not in the crosshairs. So here in Hungary, we are not really a target in major international warfare. I’m not saying attacks don’t come here, but if we were really being heavily attacked, we would fall very quickly, at many smaller and larger companies alike. So that’s our good fortune, that we are not in the crosshairs.
- I’d add to that a little, let’s say I have a fairly strict view: we’re not the target, we’re a stepping stone.
- Yes.
- And that’s a bigger problem.
- So whoever comes in through us may not want to attack us, but instead looks for the one they do want to attack, and gets the information from us, while yes, they haven’t really attacked us.
- Preferably they try to do it in such a way that we don’t even notice, otherwise this vulnerability will be mitigated globally. Yes, they say the same thing about startups, that Hungary is a test market—well, apparently that can be said in defense as well.
- Well, and the other side is the same too. So technically, who is inside on what machine, as you said.
- So we don’t know that; it may have been opened long ago, they may have been inside for a long time, and its only task may be to run a watcher in the background, a kind of sentinel, which continuously tosses information outward from the background, but not to such an extent that, say, divergent data traffic appears on a firewall. And you know, this is where the issue comes in again that there is no 100% protection, period. Compliance isn’t there to provide 100% protection either, but then here is an example like the State Treasury, where there was rock-hard and strict compliance, so then let someone please explain to me that checkbox-compliance part. At what point in time, where was it true and where was it not true—but that can be rhetorical too, and if you want to add to it, gladly.
- I can’t really add much to that. Thanks for bringing in that part, that we are only a stepping stone. Yes, that’s how I see it too, and that’s actually where I wanted to get to, and I’ll steer the conversation a little in another direction on this question, because the fact that we are a stepping stone also means that since, incidentally, we are not very strong not only in defense but also in detection, it may very well be that those malware strains, even those attacker groups we are afraid of, have been living in our systems with us for years, but since we are not the ultimate target, with us they only hide and conceal themselves and wait and watch and collect data, gather information, because the same user will also appear in the energy company’s system, but will also be a user of another large system. And then they can move across neatly, because we know user habits, they will use the same username and password. That’s guaranteed, but the point is that in many cases we don’t even know they’re there, so they can hide for years without us noticing. And coming back to your original question, where the attacker side stands with these agentic tools and AI-supported tools. What I consider our great good fortune is that as the complexity of tasks increases, this certain context window that AI models require grows too, and the larger this context becomes, the greater the need would be for this context window to grow, and after a while it can’t grow anymore because it becomes insanely expensive. So we all know those stories too where some development demand accidentally ran wild at a software company, and then it generated token costs in the tens or hundreds of millions—I don’t know, I no longer remember the exact amount—but enormous token costs, and attackers can’t bear that cost to just any extent either. So they have to make compromises. The compromise is that unreliability grows in inverse proportion to complexity, and here I would refer back to the very beginning, to the reliability of foods. The system becomes much more unreliable. Up to a certain point the attacker doesn’t care about this, because they don’t have constraints as strict as, say, I do—for example, my pentester colleagues get goosebumps even at the thought of just letting an AI agent loose somewhere to do pentesting, because it is completely uncontrollable what it will do. And attackers don’t have restrictions like that; it doesn’t hurt them if it goes a little off course and life bombs the neighboring company too.
- Big deal.
- Big deal, yes.
- Yes, it doesn’t matter to them. But their efficiency is not good in this respect, because either it has to be kept very simple, in which case they can’t solve very complex, complicated tasks with it, or the whole thing spins out of control and then even they can no longer see through it, the attacking teams can’t see through it either, and then they’re trying things blindly, and either it works or it doesn’t, but then there is a high risk of exposure.
- War—then let’s talk about weapons. In the Vietnam War, it was the M16 versus the Kalashnikov, right? Those two were up against each other, and the highly reliable M16 would unfortunately always break down, because it operated with such tight tolerances, whereas the opponent’s somewhat imprecise machine gun, with its wide spread but high hit rate, served them well. And in what you’re saying, I see the exact same thing now. So the hacker group itself also operates on a risk basis, since it says it needs to generate revenue, it needs to generate unlimited revenue without regulation, ergo it can spend as much on it as it wants, because in terms of return metrics, if with this it doesn’t find that company but finds another five, then it has its revenue. And I kind of see, and I don’t know whether I’m seeing this correctly, that basically what happens is that—and we’ve had them as a guest, so I’ll say this boldly—they go onto Opter, pull the company information, look at what company value the company itself represents, and then launch an attack anyway. And they launch the same attack against companies with a similar character and profile as well, and if, say, only six out of ten work out, then that is already more than profitable, and they’ve achieved the ROI, and at that service the finance director is happily applauding at the ROI calculation. What do you think?
- Absolutely, yes, that’s exactly how it works. I think that obviously, once again, we can go back to the basics and talk about what motivates a given attacker, and there can be many kinds of motivation. In most cases it’s economic gain, in some cases reputation, but I think that’s not a strong enough motive or motivating factor. And then, of course, there are the various hacker groups funded by different states and governments for other purposes, generally supported by power ambitions or some kind of national interest. And I think that especially hacker groups funded by states and governments are under great pressure to attack that specific target. That’s why they get everything they need for it, that’s why they generally have very large resources, and they can deliberately marshal major AI capacity for carrying out a given attack, but again I come back somewhat to the thought that there is nothing new under the sun, they used to do this in the past too. I suddenly wanted to mention creating a deepfake video. Today that is obviously trivial; a 12-year-old kid can very quickly find the tool with which they can make a deepfake video from this podcast episode too, of any one of us.
- That’s true.
- So verifying the virtual space has become the hardest task on one side.
- That’s right.
- And now they can even upload it to the AI authority’s side as if this were an NDP video.
- Yes, yes. But there were already deepfake videos years ago, it’s just that back then they were inaccessible to the general public. Now this has become much closer and more accessible, but the point is that these APT groups, which are maintained and paid by governments, have the potential to make good use of these AI tools. We can’t really defend against them in any other way than by creating a very robust defensive environment. And by observing as precisely as possible and developing in as much detail as possible all kinds of IT security or cybersecurity principles that have already been repeated many times before.
- You said that this is a calibration, how much we put into checks and security steps when adopting an AI function for use, and that is inversely proportional to how much risk we will have in it. And how do you feel—generally speaking—is this now already well calibrated at companies, meaning they are building in neither too many nor too few control points? The more control points there are, the slower the whole thing becomes, and the more that advantage AI would provide—being fast and automated—practically disappears, or does this calibration generally still tip in one direction or the other?
- I think this calibration still tips, and not just because they don’t find these weights properly, but because companies don’t use AI properly, because right now we think that if here is a hammer, then everything is a nail.
- And we want to hammer in everything with AI. So when, for example, we talk about an agentic network, I listened to many of your episodes, but I don’t know how far you’ve gone into it, I haven’t listened to all of them yet, and I don’t know whether—
- We haven’t gone into instant agents. Precisely because, a little bit—
- We were waiting for you.
- Here in the Cybersecurity section we were definitely waiting for you, while on the practical implementation side we somehow see in this agenting that starry-eyed shepherd boys have appeared in it, in the sense that everyone has some kind of solution, and in reality each AI, with its own connector, can link up a game and then put the emails in order, but technically automating an entire company in such a way that I don’t know how my own processes work is, I’d say, a bold thing.
- Yes, for the definition of an agent you’d need to know what the process is.
- That’s exactly where I wanted to end up too, namely that on the one hand, when a company gets to the point of saying, I really want AI, so let’s introduce it, let’s do it. Very often not only is their IT security not in order, but their own processes aren’t in place either, and they expect AI to create good processes for them, but it doesn’t do that; that is one thing I think is very important in this matter, and the other is that with these agentic networks, I think there is a huge misunderstanding in people’s minds, namely that an agentic network is not in itself an AI network, but an agentic network is practically the old-school microservice topology already well known to everyone in IT, where small units have small independent tasks.
- And these—I’m not talking about AI now. I’m talking about having a small unit that independently implements something.
- Yes, except that in a microservice it is always precisely for the same input.
- So it provides the same service, whereas with AI there is a bit of this indeterministic fluctuation as to what it will do.
- That’s right, and precisely for that reason we don’t need to force AI into all of our agents; instead, let’s do it properly. First let’s clarify what processes we have. Let’s figure out what steps are needed, what agents are needed for them to work, but let’s not think in terms of AI agents at first—agents that iterate through things properly in an algorithmic way, go through an Excel spreadsheet thoroughly, row by row, it doesn’t need a script for it, it goes through it and extracts the data from it, and hands it over to a next agent. These can work nicely, in a closed way, really well, they can be tested to death, for the same input they will always give the same output, and for those tasks that would be excessively complicated, say a data conversion—say converting something from an audio file into an Excel spreadsheet—let’s give only that narrow task, well defined, to an AI agent, and that’s how we get our agenting network, an AI application, where AI is present in the right place helping our work, but not harming our reliability, and especially not security. And if we think about it this way, then we can hide the AI agent very well within this agentic network in such a way that it has as little direct interaction with the outside world as possible. That way we don’t take on such a big risk with it.
- And you don’t need to build so many security gates around it that it becomes very slow.
- Or rather, the agentic network itself will be the security gate around it. And by the way, if I interpreted it correctly, then we’re now approaching the end of our conversation, so if you allow me, I would still say two more thoughts about AI, which we’ll just leave hanging in the air for now, because I don’t think we’ll be able to unpack them now, but one of them is that I strongly believe and think—and soon I won’t be the only one in this—that an AI boom is coming soon which will put a bit of an end to this current state of ours where everything is always AI and we really want to hammer in even the nail with AI. And I think the solution to this current awkward situation is likely to come from somewhere quite different from what we expect. I think it’s energy hunger that will put a stop to this. There are older people who have great fun playing with AI, getting it to generate poems about how to cook chicken soup and I don’t know what else, so they use it for things like that, and they don’t see the tremendous amount of energy it burns up doing this, the whole thing just seems funny, but in the same way, those solutions where we implement every single agent with AI completely unnecessarily are so extremely wasteful that this is unsustainable. On top of that, this complexity that all of this AI brings into the way we operate is, again, unsustainable. That’s one important idea, so this cannot keep working like this. And the other thing, which is more of a somewhat futurological thought, is that all of this that I’ve been talking about so far, and what I believe—that there is nothing new under the sun right now, everything is the same, please, there’s nothing to see here, let’s move on, let’s just do our job well— all of that gets thrown into parentheses if it happens that AI takes a leap to the next level. Right now we talk about AIs, but mostly what we’re talking about are language models. When artificial general intelligence arrives, they say that it already has the creativity, problem-solving ability, intelligence, and vast knowledge of a real person. That is a completely different level, and if this artificial general intelligence arrives—there isn’t anything like that yet. So by definition, nothing like that exists yet. Everyone would like to be the first to be that.
- Theirs is the first.
- They get the first flag in connection with this, but nothing like this exists yet. I think the period will soon begin when people start shouting that they’ve done it, but for now I don’t think we should believe it yet. It will be very spectacular when it arrives. It won’t be like this Mythos mess we’ve had recently, or the runaway agents; it will be much more spectacular than that. On top of that, it will be very spectacular because the time when it appears brings much closer the next, even more dangerous phase, which is the so-called superintelligence. This may sound a bit like a tinfoil-hat thing, but all AI researchers agree that this is the direction development is heading in. Superintelligence already has roughly the total knowledge and problem-solving capacity of all humanity. That is an enormous scale; we won’t be able to do anything with that. So these levels—even artificial general intelligence really puts what I was talking about earlier into parentheses; then there will be new forms of attack that we still cannot clearly see now.
- And then they connect this with robotics too.
- Exactly.
- I think we’ll have plenty to talk about next year and in two years as well, but even today I’d be happy to close the day, because I think we achieved our goal: not to form opinions about the IT security implications of AI based only on marketing materials. Tamás, thank you very much for coming to join us.
- Thank you very much as well for the opportunity.
- Thanks for coming.
- It was great.
