{"id":116167,"date":"2025-05-30T11:15:19","date_gmt":"2025-05-30T15:15:19","guid":{"rendered":"https:\/\/www.freethink.com\/?post_type=ftm_article&p=116167"},"modified":"2025-05-30T11:15:20","modified_gmt":"2025-05-30T15:15:20","slug":"vegetative-electron-microscopy","status":"publish","type":"ftm_article","link":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy","title":{"rendered":"AI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not"},"content":{"rendered":"\n

\u201cVegetative electron microscopy\u201d is nonsense. Say it to a scientist, and you\u2019ll likely get a blank stare in response, and yet several scientific papers<\/a> have included the phrase over the past few years.<\/p>\n\n\n\n

A Russian chemist first spotted it<\/a> in a since-retracted paper published in the journal Environmental Science and Pollution Research in 2022. They relayed the finding to Alexander Magazinov, a software engineer, who also seeks out scientific fraud in papers. <\/p>\n\n\n\n

After doing some digging, Magazinov identified the error that likely led to the birth of vegetative electron microscopy: When two papers from the 1950s were digitized, two columns of text may have been read together incorrectly, the \u201cvegetative\u201d from one column smushed up with the \u201celectron microscopy\u201d in the other.<\/p>\n\n\n\n

He speculated that the digitized paper was then included in the datasets used to train large language models (LLMs) \u2014 AIs capable of \u201cunderstanding\u201d and generating human-like text. When researchers then used the models to help them write their papers, the nonsense phrase made it into the texts.<\/p>\n\n\n\n

A team of researchers at the Queensland University of Technology (QUT) has seemingly proven his theory right \u2014 and their discovery has implications that extend far beyond the margins of obscure scientific journals.<\/p>\n\n\n\n

Nonsense in, nonsense out<\/h3>\n\n\n\n

To understand how the QUT team traced vegetative electron microscopy to LLM training data, you first need to understand how these models work.<\/p>\n\n\n\n

The most important thing to remember is that, despite seeming smart, these AIs don\u2019t actually know<\/em> anything. They don\u2019t have common sense and can\u2019t think, at least not in the way we can. <\/p>\n\n\n\n

What they can<\/em> do is predict what \u201ctoken\u201d is most likely to come next in a response to a prompt. Tokens are units of language \u2014 usually words, but not necessarily \u2014 and LLMs learn to string them together in ways that make sense to us by analyzing and identifying patterns in enormous datasets of training material.<\/p>\n\n\n\n

\u201c<\/em>At the simplest level\u2026[an LLM] looks at the previous tokens that it\u2019s generated, and then it outputs the next most probable token,\u201d Shing-hon Lau, manager of the AI Security team at the Carnegie Mellon University Software Engineering Institute<\/a>, who was not involved in the QUT research, told Freethink. <\/p>\n\n\n\n

\u201c<\/em>Really, it\u2019s a next-token generator,\u201d he continued. \u201cThat\u2019s all an LLM is at the end of the day.\u201d<\/p>\n\n\n\n

\"A
Bacteriological Reviews \/ The Conversation<\/figcaption>
Excerpts from the scanned papers show how “vegetative electron microscopy” was introduced into AI training data. \n<\/div><\/div><\/figure>\n\n\n\n

To get their LLMs to be masters of token prediction, developers train the models on massive amounts of human-generated text. In some cases, they scour the web for new training material, but often they acquire datasets already compiled by someone else.<\/p>\n\n\n\n

To figure out whether vegetative electron microscopy was included in AI training data, the QUT team prompted various LLMs with bits of text from the two 1950s papers to see if the models would provide responses containing the nonsense phrase. If a model did, this would indicate that it had seen the phrase during training.<\/p>\n\n\n\n

The researchers discovered the phrase started showing up in responses generated by OpenAI\u2019s GPT 3.0, which was released in 2020.<\/p>\n\n\n\n

\u201c<\/em>That doesn\u2019t happen with older models,\u201d Kevin Witzenberger, a research fellow at QUT\u2019s GenAI lab<\/a>, told Freethink. \u201cIt only happens with GPT 3.0, so we know kind of when this must have happened.\u201d<\/p>\n\n\n\n

By looking into the training data for GPT 3.0, the QUT team determined that vegetative electron microscopy likely entered the \u201cmind\u201d of the LLM through Common Crawl<\/a>, a collection of HTML text for billions of websites. Because this data is freely accessible<\/a>, many AI companies use it for training, which could explain why the phrase was also included in outputs generated by Anthropic\u2019s Claude 3.5.<\/p>\n\n\n\n

Digital fossils<\/h3>\n\n\n\n

The QUT researchers have coined a new term for output oddities like vegetative electron microscopy: \u201cdigital fossils.\u201d <\/p>\n\n\n\n

Much like a fossil becomes one with the ground beneath our feet, digital fossils become embedded in the data sets that ground LLMs like ChatGPT and Claude, the team wrote in The Conversation<\/a>. The term is also appropriate given that digital fossils are relics of the past, at least in AI terms.<\/p>\n\n\n\n

\u201cA \u2018digital fossil\u2019 is one of the many problematic things that can be produced with the operational logics of LLMs [the processes that shape the models],\u201d said Witzenberger, adding that \u201cthere are lots of other problematic or nonsense words that can get into an LLM.\u201d<\/p>\n\n\n\n

These words can be embedded into LLMs through large datasets, like Common Crawl, or more targeted training data, such as Wikipedia, Reddit, and research papers, Stefan Baack<\/a>, who researches AI training datasets at the Mozilla Foundation\u2019s Campaigns and Policy Team and who was not involved with the QUT team\u2019s work, told Freethink. <\/p>\n\n\n\n

The latter kind of data is often used because developers believe it can make their models better in specific ways \u2014 Reddit posts may help an AI generate responses that are more conversational, for example \u2014 but it can lead to other kinds of output oddities.<\/p>\n\n\n\n

Case in point: Remember when Google\u2019s AI was encouraging you to put glue on your pizza? 404 Media<\/a> traced that back to a Reddit post. The AI\u2019s suggestion that you eat at least one rock every day also came from Reddit.<\/p>\n\n\n\n

\u201cWe’re going to see it cause more and more problems in the future.”<\/p>Fran\u00e7ois Chollet<\/cite><\/blockquote><\/figure>\n\n\n\n

Rather than being caused by scanning errors or misspellings, these nonsense outputs are nonsensical by design<\/em>: humans wrote them as jokes, but because LLMs don\u2019t actually think, they can\u2019t tell when something is said in jest and thus present the tokens with a straight face.<\/p>\n\n\n\n

\u201cWhile these models usually do understand context, it is not how we as humans make sense of context,\u201d said Witzenberger.<\/p>\n\n\n\n

An LLM trained on any of this flawed data might then be used to generate brand new data that can be used to train newer models. This \u201csynthetic data<\/a>\u201d can further embed digital fossils and other nonsense into the AIs and make it harder to identify their source.<\/p>\n\n\n\n

\u201cThis is definitely a real phenomenon,\u201d Fran\u00e7ois Chollet<\/a>, an AI researcher uninvolved with the QUT team, told Freethink. \u201cWe’re going to see it cause more and more problems in the future as future datasets will increasingly incorporate LLM-generated text.\u201d<\/p>\n\n\n\n

Training transparency<\/h3>\n\n\n\n

The technique the QUT team used to identify the source of vegetative electron microscopy could help us find the origins of not only more digital fossils but also other flawed text, like the nonsense that made it from Reddit into Google\u2019s AI.<\/p>\n\n\n\n

By inputting prompts and looking to see at what point an AI model begins to output a questionable token as the most likely result, researchers can estimate when the problem took hold and work to remedy it.<\/p>\n\n\n\n

The issue is that they can only search for the sources of flawed outputs that they know exist. The method works as a reaction to a problem \u2014 it cannot help us get out ahead of digital fossils and other output issues, and right now, we have no idea how much nonsense may become part of our knowledge base because it\u2019s injected into, repeated, and eventually solidified by AI. <\/p>\n\n\n\n

\n

“You can only have so many amusing glitches before you have a problem.”<\/p>\nShing-hon Lau<\/cite><\/blockquote>\n\n\n\n

Knowing a problem exists isn\u2019t enough, either \u2014 we also need to have an understanding of how an LLM was trained in order to find the root of an issue, like the digitization mistake that caused AIs to think vegetative electron microscopy is a thing.<\/p>\n\n\n\n

However, according to Alex Hanna, director of research at the Distributed AI Research Institute (DAIR)<\/a>, AI companies are often hesitant to share information on how they train their models due to concerns about staying competitive in the market and protecting their proprietary methods \u2014 the QUT was only able to trace vegetative electron microscopy to Common Crawl because OpenAI explicitly mentioned using the dataset in its paper on GPT 3.0.<\/p>\n\n\n\n

Ultimately, vegetative electron microscopy was a relatively silly digital fossil with limited impact, and as far as we know, no one actually put glue on their pizza because an AI told them to, but a buildup of digital fossils and other problematic outputs in these ever-more-trusted tools could eventually be a serious problem, the researchers Freethink spoke with agreed.<\/p>\n\n\n\n

Take cybersecurity, which Carnegie Mellon\u2019s Shing-hon Lau studies. A nonsense piece of code could do nothing, acting like junk DNA<\/a> \u2014 or it could be a weakness that bad actors can exploit. If a digital fossil in code goes undetected by humans, a language model may repeat the mistake over and over again.\u00a0<\/p>\n\n\n\n

\u201cNow you\u2019ve introduced this very peculiar bug and vulnerability into everything that that LLM helped to write,\u201d Lau told Freethink. \u201cRight now, this is an amusing glitch, right? But you can only have so many amusing glitches before you have a problem.\u201d<\/p>\n\n\n\n

We\u2019d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at tips@freethink.com<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

The appearance of a nonsense term in multiple scientific papers has exposed how flawed data in training sets can become embedded in AIs.<\/p>\n","protected":false},"author":25,"featured_media":116171,"template":"","ftm_taxonomy_fields":[46],"ftm_taxonomy_challenges":[],"ftm_taxonomy_statuses":[36],"ftm_taxonomy_hidden_tags":[],"class_list":["post-116167","ftm_article","type-ftm_article","status-publish","has-post-thumbnail","hentry","ftm_taxonomy_fields-ai","ftm_taxonomy_statuses-featured"],"acf":[],"apple_news_notices":[],"yoast_head":"\nAI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not<\/title>\n<meta name=\"description\" content=\"The appearance of a nonsense term in multiple scientific papers has exposed how flawed data in training sets can become embedded in AIs.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"AI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not\" \/>\n<meta property=\"og:description\" content=\"The appearance of a nonsense term in multiple scientific papers has exposed how flawed data in training sets can become embedded in AIs.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy\" \/>\n<meta property=\"og:site_name\" content=\"Freethink\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-30T15:15:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.freethink.com\/wp-content\/uploads\/2025\/05\/vegetative-electron-mcroscopy-thumb-2.jpg?resize=1200,630\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:description\" content=\"A single digitization error reveals how flawed data can quietly embed itself into powerful AI models.\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"AI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not","description":"The appearance of a nonsense term in multiple scientific papers has exposed how flawed data in training sets can become embedded in AIs.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy","og_locale":"en_US","og_type":"article","og_title":"AI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not","og_description":"The appearance of a nonsense term in multiple scientific papers has exposed how flawed data in training sets can become embedded in AIs.","og_url":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy","og_site_name":"Freethink","article_modified_time":"2025-05-30T15:15:20+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.freethink.com\/wp-content\/uploads\/2025\/05\/vegetative-electron-mcroscopy-thumb-2.jpg?resize=1200,630","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_description":"A single digitization error reveals how flawed data can quietly embed itself into powerful AI models.","twitter_misc":{"Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy#article","isPartOf":{"@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy"},"author":{"name":"kristinhouser","@id":"https:\/\/www.freethink.com\/#\/schema\/person\/e45bf79276f6c14454ee4e1dfa7aca8c"},"headline":"AI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not","datePublished":"2025-05-30T15:15:19+00:00","dateModified":"2025-05-30T15:15:20+00:00","mainEntityOfPage":{"@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy"},"wordCount":1519,"publisher":{"@id":"https:\/\/www.freethink.com\/#organization"},"image":{"@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy#primaryimage"},"thumbnailUrl":"https:\/\/www.freethink.com\/wp-content\/uploads\/2025\/05\/vegetative-electron-mcroscopy-thumb-2.jpg?quality=75","inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy","url":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy","name":"AI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not","isPartOf":{"@id":"https:\/\/www.freethink.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy#primaryimage"},"image":{"@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy#primaryimage"},"thumbnailUrl":"https:\/\/www.freethink.com\/wp-content\/uploads\/2025\/05\/vegetative-electron-mcroscopy-thumb-2.jpg?quality=75","datePublished":"2025-05-30T15:15:19+00:00","dateModified":"2025-05-30T15:15:20+00:00","description":"The appearance of a nonsense term in multiple scientific papers has exposed how flawed data in training sets can become embedded in AIs.","breadcrumb":{"@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy#primaryimage","url":"https:\/\/www.freethink.com\/wp-content\/uploads\/2025\/05\/vegetative-electron-mcroscopy-thumb-2.jpg?quality=75","contentUrl":"https:\/\/www.freethink.com\/wp-content\/uploads\/2025\/05\/vegetative-electron-mcroscopy-thumb-2.jpg?quality=75","width":2500,"height":1406,"caption":"Mariia Shalabaieva \/ Unsplash \/ Freethink"},{"@type":"BreadcrumbList","@id":"https:\/\/www.freethink.com\/artificial-intelligence\/vegetative-electron-microscopy#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Articles","item":"https:\/\/www.freethink.com\/articles"},{"@type":"ListItem","position":2,"name":"AI thinks \u201cvegetative electron microscopy\u201d is real \u2014 it\u2019s not"}]},{"@type":"WebSite","@id":"https:\/\/www.freethink.com\/#website","url":"https:\/\/www.freethink.com\/","name":"Freethink","description":"Move the world","publisher":{"@id":"https:\/\/www.freethink.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.freethink.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.freethink.com\/#organization","name":"Freethink Media","url":"https:\/\/www.freethink.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.freethink.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.freethink.com\/wp-content\/uploads\/2021\/06\/logo.svg","contentUrl":"https:\/\/www.freethink.com\/wp-content\/uploads\/2021\/06\/logo.svg","width":651,"height":124,"caption":"Freethink Media"},"image":{"@id":"https:\/\/www.freethink.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.freethink.com\/#\/schema\/person\/e45bf79276f6c14454ee4e1dfa7aca8c","name":"kristinhouser","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.freethink.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/ff88759e0ed195de655c7703310050f17b921ae4fc276d7eb5930cddafa694f9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ff88759e0ed195de655c7703310050f17b921ae4fc276d7eb5930cddafa694f9?s=96&d=mm&r=g","caption":"kristinhouser"},"url":"https:\/\/www.freethink.com\/author\/kristinhouser"}]}},"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/ftm_article\/116167","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/ftm_article"}],"about":[{"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/types\/ftm_article"}],"author":[{"embeddable":true,"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/users\/25"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/media\/116171"}],"wp:attachment":[{"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/media?parent=116167"}],"wp:term":[{"taxonomy":"ftm_taxonomy_fields","embeddable":true,"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/ftm_taxonomy_fields?post=116167"},{"taxonomy":"ftm_taxonomy_challenges","embeddable":true,"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/ftm_taxonomy_challenges?post=116167"},{"taxonomy":"ftm_taxonomy_statuses","embeddable":true,"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/ftm_taxonomy_statuses?post=116167"},{"taxonomy":"ftm_taxonomy_hidden_tags","embeddable":true,"href":"https:\/\/www.freethink.com\/wp-json\/wp\/v2\/ftm_taxonomy_hidden_tags?post=116167"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}