<thead id="t1bbr"><dfn id="t1bbr"></dfn></thead>

    <nobr id="t1bbr"><cite id="t1bbr"><menuitem id="t1bbr"></menuitem></cite></nobr>

    <form id="t1bbr"></form>

      <em id="t1bbr"></em>

      18210244181 | 登錄 注冊
      AI Is Learning to Identify Toxic Online Content
      發布時間:2022年05月01日     郭曉陽 譯  
      來源: 英語世界
      字號 簡體 繁體 打印

      AI Is Learning to Identify Toxic Online Content


      Machine-learning systems could help flag hateful, threatening or offensive language.


      By Laura Hanu et al.



      Social platforms large and small are struggling to keep their communities safe from hate speech, extremist content, harassment and misinformation. One solution might be AI: developing algorithms to detect and alert us to toxic and inflammatory comments and flag them for removal. But such systems face big challenges.


      The prevalence of hateful or offensive language online has been growing rapidly in recent years. Social media platforms, relying on thousands of human reviewers, are struggling to moderate the ever-increasing volume of harmful content. In 2019, it was reported that Facebook moderators are at risk of suffering from PTSD as a result of repeated exposure to such distressing content. Outsourcing this work to machine learning can help manage the rising volumes of harmful content. Indeed, many tech giants have been incorporating algorithms into their content moderation1 for years.

      近年來,網上的仇恨言論或攻擊性語言激增。社交媒體平臺依靠數千名人工審核員,難以審核持續增長的有害內容。據報道,2019年, 臉書公司的審核員由于反復接觸此類令人痛苦的內容,面臨罹患創傷后應激障礙的風險。把這項工作交由機器學習完成,有助于解決有害內容數量不斷攀升的問題。事實上,近年來,許多大型科技公司已經把算法集成到內容審核中。

      One such example is Google’s Jigsaw2, a company focusing on making the internet safer. In 2017, it helped create Conversation AI, a collaborative research project aiming to detect toxic comments online. However, a tool produced by that project, called Perspective, faced substantial criticism. One common complaint was that it created a general “toxicity score” that wasn’t flexible enough to serve the varying needs of different platforms. Some Web sites, for instance, might require detection of threats but not profanity, while others might have the opposite requirements.

      谷歌旗下的Jigsaw公司即為一例。Jigsaw是一家專注于提升互聯網安全性的公司。2017年, 它幫助創建了Conversation AI。這是一個旨在檢測網上惡意評論的合作研究項目。然而,這個項目推出的一款名為Perspective的工具卻遭到廣泛批評。一條常見的投訴意見是,此工具生成的綜合“惡意評分”不夠靈活,無法滿足不同平臺的各種需求。例如,有些網站可能需要檢測威脅言論,而非不雅語言,而另一些網站的需求可能正好相反。

      Another issue was that the algorithm learned to conflate toxic comments with nontoxic comments that contained words related to gender, sexual orientation, religion or disability. For example, one user reported that simple neutral sentences such as “I am a gay black woman” or “I am a woman who is deaf ” resulted in high toxicity scores, while “I am a man” resulted in a low score.



      Following these concerns, the Conversation AI team invited developers to train their own toxicity-detection algorithms and enter them into three competitions (one per year) hosted on Kaggle, a Google subsidiary known for its community of machine learning practitioners, public data sets and challenges. To help train the AI models, Conversation AI released two public data sets containing over one million toxic and nontoxic comments from Wikipedia and a service called Civil Comments. Some comments were seen by many more than 10 annotators (up to thousands), due to sampling and strategies used to enforce rater accuracy.

      為回應這些關切,Conversation AI團隊邀請開發者訓練自己的惡意檢測算法,并參加在Kaggle平臺舉辦的三項算法競賽(每年一項)——Kaggle是谷歌公司的子公司,以旗下的機器學習從業者社區、公共數據集和挑戰賽而聞名。為幫助訓練人工智能模型,Conversation AI公布了兩個公共數據集——包含一百余萬條來自維基百科的惡意和非惡意評論,以及一個名為“文明評論”的服務。由于采樣和為加強評分者準確率所采用的策略等原因,部分評論由遠超十名(最多數千名)的注釋者審閱。

      The goal of the first Jigsaw challenge was to build a multilabel toxic comment classification model with labels such as “toxic”, “severe toxic”, “threat”, “insult”, “obscene”, and “identity hate”. The second and third challenges focused on more specific limitations of their API: minimizing unintended bias towards pre-defined identity groups and training multilingual models on English-only data.

      Jigsaw公司第一個挑戰的目標是創建一個多標簽惡意評論分類模型,其標簽包含“惡意”“嚴重惡意”“威脅”“侮辱”“淫穢”“身份仇恨”等。第二及第三個挑戰則專注于解決更加具體的API 限制:最大限度減少對預定義身份群體的無意識偏見,以及訓練純英語數據的多語言模型。

      Our team at Unitary, a contentmoderation AI company, took inspiration from the best Kaggle solutions and released three different models corresponding to each of the three Jigsaw challenges. While the top Kaggle solutions for each challenge use model ensembles, which average the scores of multiple trained models, we obtained a similar performance with only one model per challenge.


      While these models perform well in a lot of cases, it is important to also note their limitations. First, these models will work well on examples that are similar to the data they have been trained on. But they are likely to fail if faced with unfamiliar examples of toxic language.


      Furthermore, we noticed that the inclusion of insults or profanity in a text comment will almost always result in a high toxicity score, regardless of the intent or tone of the author. As an example, the sentence “I am tired of writing this stupid essay” will give a toxicity score of 99.7 percent, while removing the word “stupid” will change the score to 0.05 percent.


      Lastly, all three models are still likely to exhibit some bias, which can pose ethical concerns when used off-the-shelf3 to moderate content.


      Although there has been considerable progress on automatic detection of toxic speech, we still have a long way to go until models can capture the actual, nuanced, meaning behind our language—beyond the simple memorization of particular words or phrases. Of course, investing in better and more representative datasets would yield incremental improvements, but we must go a step further and begin to interpret data in context, a crucial part of understanding online behavior. A seemingly benign text post on social media accompanied by racist symbolism in an image or video would be easily missed if we only looked at the text. We know that lack of context can often be the cause of our own human misjudgments. If AI is to stand a chance of replacing manual effort on a large scale, it is imperative that we give our models the full picture.





      1. content moderation 內容審核,是基于圖像、文本、視頻的檢測技術,可自動檢測涉黃、廣告、涉政、涉暴、涉及敏感人物等內容,對用戶上傳的圖片、文字、視頻進行內容審核,幫助客戶降低業務違規風險。


      2. 由谷歌建立的一家技術孵化公司(其前身為谷歌智庫部門Google Ideas),主要負責創建技術工具來減少并遏制線上虛假信息、騷擾以及其他問題。


      3. off the shelf(產品)現成的,不需定制的。文中充當副詞,用作狀語。