According to Gartner, by 2025, 80% of enterprises will have integrated AI-powered assistants into their daily operations. These tools are expected to boost productivity by 40%, with tasks like meeting scheduling, note-taking, and even decision-making being automated. However, this rapid growth also poses a potential risk for data security and confidentiality breaches if not handled correctly.

The Otter.ai Incident: When AI Fails to Respect Boundaries

One such case involved Alex Bilzerian, a researcher and engineer, who discovered an unexpected problem with the transcription service Otter.ai. After a Zoom meeting with venture capital investors, Bilzerian received a transcript that included post-meeting conversations where investors discussed sensitive topics, including their firm’s strategic failures. Bilzerian was not in the meeting when these discussions took place. Shocked by the breach, he chose to cancel the deal, showing how AI mishaps can have real-world consequences.

This incident highlights one of the main concerns with AI assistants: they don’t always know when to stop recording or what information should remain private. Otter.ai clarified that users can control settings around transcript sharing, but this issue reveals a gap in user knowledge and the AI’s lack of discretion.

A Statista survey from 2023 indicated that 53% of employees felt uncomfortable with AI handling sensitive information in the workplace. Furthermore, 45% believed AI assistants could inadvertently lead to data breaches, highlighting the need for stronger controls and more transparency from AI vendors.

The Growing Use of AI in Corporate Settings

This incident is part of a larger trend where companies are rapidly integrating AI tools into their operations. From Salesforce’s Agentforce to Microsoft’s Copilot and Google’s Gemini, AI is increasingly embedded in workplace software. These AI assistants can manage meetings, summarize conversations, transcribe discussions, and even provide daily recaps. But as AI gains more access to our work, it becomes clear that these tools don’t have the nuanced understanding of discretion that human assistants do.

• Microsoft Copilot: Integrated into Microsoft 365, Copilot helps employees draft emails, create documents, analyze data, and more. While the assistant increases efficiency, it also has access to potentially sensitive company information, raising privacy concerns.
• Salesforce Agentforce: This AI-powered tool is designed to assist sales and customer service agents. It automates common customer interactions and analyzes sales data, but its access to confidential sales pipelines and proprietary information makes it vulnerable to misuse.
• Google Gemini: Google's suite of AI-powered tools offers similar functions, including summarizing documents and generating reports. However, with large amounts of data passing through these systems, the potential for leaks is ever-present.

The Privacy Concern: AI at Work

Privacy advocates, like Naomi Brockwell, are raising the alarm about the invasiveness of AI tools. Brockwell points out that while AI offers immense convenience, it also records vast amounts of data, often without users fully grasping the implications. This constant recording and the subsequent transcription of private conversations open the door to significant privacy violations.

For example, Isaac Naor, a software designer, shared a story about receiving a transcript from a Zoom meeting that included a private conversation where his colleague muted herself to discuss him. This highlights how AI can inadvertently capture private, sensitive moments, potentially creating uncomfortable situations for employees.

The problem lies in AI's inability to "read the room." While these tools are designed to be efficient and capture everything, they often lack the discernment to know when something should not be recorded or shared. In many cases, users are unaware of the settings that govern how these AI tools operate, leading to unintended information leaks.

AI's Role in Shaping Work Culture

AI assistants aren’t just tools for productivity—they’re also reshaping how we interact at work. A survey by Pew Research in 2022 found that 65% of workers in AI-assisted environments felt that the technology made their jobs easier, but 48% worried about the technology capturing sensitive or private information.

Rob Bezdjian, the owner of an events business, shared an instance where a meeting with investors became tense due to the presence of Otter’s AI transcription service. The investors insisted on recording the conversation, making Bezdjian uncomfortable sharing proprietary ideas. As a result, the deal fell through.

AI's tendency to record everything can also lead to more serious consequences. OtterPilot, for example, can record, transcribe, and even summarize meetings. While users are notified when a recording is in process, some may not realize that AI tools like Otter can also collect screenshots, text, and images from virtual meetings, as well as other user-provided data. This data can be shared with third parties, such as AI services or even law enforcement, raising significant concerns about privacy and security.

The Responsibility: Companies vs. Users

Despite the potential risks, companies that develop and deploy AI tools argue that users have control over their settings. Otter.ai responded to the incident involving Bilzerian by noting that users can change, update, or stop sharing permissions at any time. They also provide guides on how to adjust these settings. Similarly, Zoom encourages users to review settings to prevent unwanted sharing.

However, placing the responsibility solely on users is problematic, according to Hatim Rahman, an associate professor at Northwestern University’s Kellogg School of Management. He argues that companies should be doing more to prevent such issues. For example, AI tools could be designed with more friction—such as asking for confirmation before sharing transcripts with attendees who left a meeting early.

Rahman believes that while users should familiarize themselves with the technology, companies need to take a more proactive approach to ensure these tools don’t lead to unintended consequences. This is especially important given that many decision-makers who implement AI tools may not fully understand the privacy risks involved.

AI and Corporate Accountability

The risks posed by AI tools extend beyond individual users to entire organizations. Will Andre, a cybersecurity consultant, recalls a time when AI mistakenly saved a video meeting where his bosses were discussing layoffs to the company’s public server. The consequences could have been disastrous, but Andre chose not to act on the information.

A Deloitte report in 2023 estimated that 56% of companies using AI assistants have faced privacy issues or leaks due to improper configurations or misuse of these tools. The same report emphasized that companies need to develop more stringent AI governance policies and educate their workforce on potential risks.

Conclusion

AI assistants are revolutionizing the workplace, offering unparalleled convenience and efficiency. However, the risks they pose, from privacy breaches to unintended information sharing, cannot be ignored. As these tools become more ingrained in our work lives, companies must take a proactive role in ensuring that AI is used responsibly, and employees must be vigilant about how their data is being handled. The key to harnessing the power of AI lies in understanding its limitations and ensuring that discretion and privacy remain at the forefront of its deployment.

Source: AI assistants are blabbing our embarrassing work secrets