Artificial intelligence is no longer confined to answering questions, writing emails, or generating images. With the help of AI agents, organizations can now analyze confidential data, operate software, and complete various tasks without direct human intervention. But as AI's capabilities grow, so does the risk of relying on it blindly. Microsoft Chief Executive Officer Satya Nadella has warned everyone not to trust AI without question, saying it could become a source of internal risk. He also stressed the importance of building effective safeguards to keep AI operations in check while putting the technology to use.

In a recent post on the social media platform X, Nadella said that even if an AI model appears safe, it should not be fully trusted. Extra caution is needed especially when AI agents gain access to an organization's sensitive information and can carry out important tasks on their own. In the post, Nadella referred to artificial intelligence as 'super intelligence.'

According to Nadella, relying on the security assurances given by AI model developers is not enough. Even if a model is built to be safe, there is no guarantee it will behave as expected in every situation. The model could make unexpected decisions, or its safety mechanisms could be compromised. While AI can provide analysis and recommendations, determining which data it may use and what kind of work it may perform must be handled through a separate system of controls.

Nadella also cautioned against using one AI model to verify the work of another. In his view, without an independent verification arrangement, one model may fail to detect the errors of another. As a result, even using multiple models may not prevent wrong decisions or security risks. For this reason, systems must be developed to monitor AI behavior, limit the permissions granted to it, and ensure it cannot cross defined boundaries. In addition, if an AI system fails or its security is breached, that information should be disclosed quickly so that other organizations can learn from such incidents and improve their own protective measures.

The use of AI agents is currently growing across organizations. These agents are being assigned tasks such as data analysis, software management, and completing various jobs, saving time and labor while accelerating work. However, the question of what could happen if an AI makes a wrong decision or takes an unwanted step has also become important. When a bug appears in conventional software, the cause can usually be identified by examining the relevant code. But it is not always easy to understand why an AI model gave a particular answer or took a specific action. As a result, investigating the reasoning behind a model's decisions and managing potential risks can become difficult.

Source: India Today