By Nir Kshetri
Ever since the start of the pandemic, more and more public school students are using laptops, tablets or similar devices issued by their schools.
The percentage of teachers who reported their schools had provided their students with such devices doubled from 43% before the pandemic to 86% during the pandemic, a September 2021 report shows.
In one sense, it might be tempting to celebrate how schools are doing more to keep their students digitally connected during the pandemic. The problem is, schools are not just providing kids with computers to keep up with their schoolwork. Instead – in a trend that could easily be described as Orwellian – the vast majority of schools are also using those devices to keep tabs on what students are doing in their personal lives.
Indeed, 80% of teachers and 77% of high school students reported that their schools had installed artificial intelligence-based surveillance software on these devices to monitor students’ online activities and what is stored in the computer.
This student surveillance is taking place – at taxpayer expense – in cities and school communities throughout the United States.
For instance, in the Minneapolis school district, school officials paid over $355,000 to use tools provided by student surveillance company Gaggle until 2023. Three-quarters of incidents reported – that is, cases where the system flagged students’ online activity – took place outside school hours.
In Baltimore, where the public school system uses the GoGuardian surveillance app, police officers are sent to children’s homes when the system detects students typing keywords related to self-harm.
Safety versus privacy
Vendors claim these tools keep students safe from self-harm or online activities that could lead to trouble. However, privacy groups and news outlets have raised questions about those claims.
Vendors often refuse to reveal how their artificial intelligence programs were trained and the type of data used to train them.
Privacy advocates fear these tools may harm students by criminalizing mental health problems and deterring free expression.
As a researcher who studies privacy and security issues in various settings, I know that intrusive surveillance techniques cause emotional and psychological harm to students, disproportionately penalize minority students and weaken online security.
Artificial intelligence not intelligent enough
Even the most advanced artificial intelligence lacks the ability to understand human language and context. This is why student surveillance systems pick up a lot of false positives instead of real problems.
In some cases, these surveillance programs have flagged students discussing music deemed suspicious and even students talking about the novel “To Kill a Mockingbird.”
Harm to students
When students know they are being monitored, they are less likely to share true thoughts online and are more careful about what they search. This can discourage vulnerable groups, such as students with mental health issues, from getting needed services.
When students know that their every move and everything read and written is watched, they are also less likely to develop into adults with a high level of self-confidence. In general, surveillance has a negative impact on students’ ability to act and use analytical reasoning. It also hinders the development of the skills and mindset needed to exercise their rights.
More adverse impact on minorities
U.S. schools disproportionately discipline minority students. African American students’ chances of being suspended are more than three times higher than that of their white peers.
After evaluating flagged content, vendors report any concerns to school officials, who take disciplinary actions on a case-by-case basis. The lack of oversight in schools’ use of these tools could lead to further harm for minority students.
The situation is worsened by the fact that Black and Hispanic students rely more on school devices than their white peers do. This in turn makes minority students more likely to be monitored and exposes them to greater risk of some sort of intervention.
When both minority students and their white peers are monitored, the former group is more likely to be penalized because the training data used in developing artificial intelligence programs often fails to include enough minorities. Artificial intelligence programs are more likely to flag languages written and spoken by such groups. This is due to the underrepresentation of languages written and spoken by minorities in the datasets used to train such programs and the lack of diversity of people working in this field.
Leading AI models are 50% more likely to flag tweets written by African Americans as “offensive” that those written by others. They are 2.2 times more likely to flag tweets written in African American slang.
These tools also affect sexual and gender minorities more adversely. Gaggle has reportedly flagged “gay,” “lesbian” and other LGBTQ-related terms because they are associated with pornography, even though the terms are often used to describe one’s identity.
Increased security risk
These surveillance systems also increase students’ cybersecurity risks. First, to comprehensively monitor students’ activities, surveillance vendors compel students to install a set of certificates known as root certificates. As the highest-level security certificate installed in a device, a root certificate functions as a “master certificate” to determine the entire system’s security. One drawback is that these certificates compromise cybersecurity checks that are built into these devices.
Gaggle, which scans digital files of more than 5 million students each year, installs such certificates. This tactic of installing certificates is similar to the approach that authoritarian regimes, such as the Kazakhstani government, use to monitor and control their citizens and that cybercriminals use to lure victims to infected websites.
Second, surveillance system vendors use insecure systems that hackers can exploit. In March 2021, computer security software company McAfee found several vulnerabilities in student monitoring system vendor Netop’s Vision Pro Education software. For instance, Netop did not encrypt communications between teachers and students to block unauthorized access.
The software was used by over 9,000 schools worldwide to monitor millions of students. The vulnerability allowed hackers to gain control over webcams and microphones in students’ computers.
Finally, personal information of students that is stored by the vendors is susceptible to breaches. In July 2020, criminals stole 444,000 students’ personal data – including names, email addresses, home addresses, phone numbers and passwords – by hacking online proctoring service ProctorU. This data was then leaked online.
Schools would do well to look more closely at the harm being caused by their surveillance of students and to question whether they actually make students more safe – or less.
Nir Kshetri is Professor of Management at the University of North Carolina-Greensboro.
The Conversation arose out of deep-seated concerns for the fading quality of our public discourse and recognition of the vital role that academic experts could play in the public arena. Information has always been essential to democracy. It’s a societal good, like clean water. But many now find it difficult to put their trust in the media and experts who have spent years researching a topic. Instead, they listen to those who have the loudest voices. Those uninformed views are amplified by social media networks that reward those who spark outrage instead of insight or thoughtful discussion. The Conversation seeks to be part of the solution to this problem, to raise up the voices of true experts and to make their knowledge available to everyone. The Conversation publishes nightly at 9 p.m. on FlaglerLive.
“Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.” Benjamin Franklin.
Everyone. . . read Orwell’s 1984! Read it again and again! Since it is banned, that is even more reason to read it. This spying is “BIG BROTHER” personified!!! Very very dangerous!!!!
Welcome to communist China, rather America. Control, control, control is the name of the communist America we now live. I miss the America I grew up in. One by one, your freedoms are being taken away. Wake up America while we are somewhat still free.
Brian Ford says
Professor Kshetri is all wrong in his analysis and conclusions regarding monitoring minor students use of district supplied computing equipment and the Internet. He is often citing examples from organizations were computing users are adults and were the organization’s have defined policies regarding the acceptable use of the equipment and Internet access. Kshetri should be pointing out the failure by parents to monitor their own children’s use of computing equipment (especially smartphones) and the Internet. This is not a free speech issue. These are our minor children. These are children (right here in Flagler County) that are being accused of committing ‘crimes’ where the laws they are breaking were often written before the advent of Internet. These are children whose parents should be monitoring their use of computing equipment and the Internet. When a school district supplies computing equipment that can access the Internet it should be the responsibility of that district; working with parents, to make sure that children are educated in the proper use of these tools and in doing so do not come into any harm. Harm that might be something like being profiled and groomed by an Internet predator; but also harm that the child can do to themselves by sending a message or making an inappropriate post. When a parent provides a smartphone or computer and Internet access to their child that responsibility of knowing what their child is looking at; who that child is communicating with ; and what their child is saying should fall squarely on those parents shoulders.
Ray W. says
Since two persons holding different beliefs can both be correct (or incorrect) about their interpretations of an issue, and since one person can be right and wrong at the same time about an issue he promotes, given the possibility that many issues can have multiple correct (or incorrect) outcomes, I hope Brian Ford can spot the irony he creates in his first sentence of his comment when he asserts that the author is “all wrong” in the author’s “analysis and conclusions.” I am not defending or denying the author’s points, as the article is presented as an opinion about a complex issue that has been debated since long before the founding of our nation. I am merely pointing out that Brian Ford greatly weakens whatever points he attempts to make by starting his comment using an all-or-nothing perspective. I learned a long time ago that some people live in a perfect or bad world and others live in a good/better/best, bad/worse/worst world. I repeatedly attempt to live in the latter world, and try to avoid life in a perfect or bad world. Those who occupy a perfect or bad world create a vision that often results in a perspective that allows for only one possible state: I am totally right and anyone who opposes my view is totally wrong. While I do not claim expertise in AI-driven apps used to monitor schoolchildren’s uses of school-issued electronic devices to do research on the internet, it appears to me that the author’s opinion piece has multiple correct, yet debatable, conclusions based on the author’s analysis of the overall issue. Whether the author is more correct or less correct in his points than Brian Ford is debatable, but I assert that Brian Ford might be well-served if he were to consider and eventually accept the idea that he may not be totally correct in the scope of his assertions. It is not a bad thing to remain skeptical of one’s viewpoint, even after publishing that viewpoint. This is how we are able to change our minds after thinking things through. Indeed, the scientific method is based on skepticism of accepted truths. All that being said, I do want to point out that it seems to me that it might be fair to argue that those who occupy street corners in Flagler Beach while waving flags bearing obscene messages about our president are “all wrong.” Oy vey!