What Changed in Engineering Writing After Generative AI Chat Assistants: Evidence from 1.8 Million Articles

Journal: Journal of Informetrics

Taner Cokyasar Email: tcokyasar@tamu.edu Affiliation: Department of Engineering Technology and Industrial Distribution, Texas A&M University, College Station, 77843, TX, USA Corresponding author: Corresponding author.

Abstract

This study asks what changed in engineering articles after conversational language models became publicly available on November 30, 2022, and develops methods for measuring such change in a field without a uniform abstract index. It assembled 1,839,962 articles published from 2015 to 2025 in 256 engineering journals, tracked twelve measures of wording, style, and article structure, and compared each with the projection of its own pre-2023 trend. Building the monthly series showed that records dated only by year fall on January 1 and create false January peaks unless removed. In abstracts, 31 words associated with language model output appeared 82% more often than projected (55.4 against 27.0 per 10,000 words in 2025), while four control words of the same formal register fell 16%. Words that soften a claim, such as may, fell 13%, words that strengthen one, such as clearly, rose 20%, and long words rose 11%. Author counts, reference lists, page counts, and publication volume stayed within 4% of trend. The change appears in all fourteen subject categories, and false release dates, dropping any one journal or word, and word lists chosen from the data support it. A random sample of 12,866 full texts from the same corpus, including subscription articles and weighted back to it, shows the same words rising 99% in article bodies. Statements disclosing AI writing assistance, measured apart from mentions of language models, went from none to 3.7% in 2025, four in five of them at one publisher. The design is descriptive, not causal.

Keywords: 

scholarly writing , generative artificial intelligence , large language models , bibliometrics , interrupted time series , engineering

1 Introduction

On November 30, 2022, a conversational language model became freely available to anyone with a web browser (OpenAI, 2022). Within months such models were listed as coauthors (Stokel-Walker, 2023), editors were reporting machine-written submissions (Conroy, 2023), publishers had issued disclosure policies (Committee on Publication Ethics, 2023), and a substantial share of surveyed researchers said they used the tools (Van Noorden and Perkel, 2023). Much of what is claimed about the effect on academic writing, however, rests on anecdote, on small hand-coded samples, or on detection tools whose error rates are poorly known and that misjudge authors writing in an additional language (Liang et al., 2023). This paper asks a narrower question: what measurably changed in the engineering research literature after that date?

Engineering is a useful field for this analysis, for three reasons. It is large, so monthly series are precise. It spans subject areas with different writing conventions, so a change can be checked across fields. It has also received less attention than two other fields: biomedicine, where PubMed collects the abstracts of the literature in one index, and computer science, where many papers appear in full on the arXiv preprint server. Engineering has no such source of abstracts or full texts, so this study had to assemble both, and that shapes its design.

Beyond its findings for engineering, the paper makes three contributions to how such change is measured, each of which applies to any field. First, it combines two sources. Monthly series are built from the corpus, meaning every research article that the study’s journals published from 2015 to 2025, as described by open bibliographic records. A sample of full texts is then weighted back to the corpus, so that results from abstracts can be checked against the body of the same articles in a field without a uniform abstract index. Second, it shows that records carrying only a publication year, which bibliographic sources assign to January 1, produce false January peaks in monthly series, and it removes them. Third, it separates statements that an AI tool was used to write an article from mentions of language models as a subject or a citation; counting the latter overstates disclosure.

The study measures twelve things, in three groups, and the results follow the same order. Each was chosen because the tools could plausibly change it, and each is introduced here with the change it was meant to detect.

Four come from article abstracts. The first is how often words associated with language model output appear (marker words, such as delve, intricate, showcase, and underscore): if authors draft or edit with these tools, the words the models over-use should become more common. The marker words come mainly from the words that Kobak et al. (2025) and Liang et al. (2024) found language models over-use; Section 4 lists all 31 and their sources. The second is how often authors soften a claim with qualifying words, such as may and might, and how often they strengthen one with emphatic words, such as clearly and substantially; text drafted or polished by a model may state its claims more firmly, with fewer qualifiers and more emphasis. The third is how long and how varied the words are, which would rise if the tools replace plain words with longer and less common ones. The fourth is readability, which could move either way: the tools are often used to make prose clearer, but longer words make it harder to read. Readability also stands in for grammatical quality, which cannot be measured on published articles because copyediting removes most errors before publication.

Five come from the bibliographic record. If the tools make writing faster, more articles could be published, so publication volume could rise; if they let one person do work that once needed several, the number of authors and institutions per article could fall; and if they make the literature easier to search and summarize, reference lists could grow. Title form (length, and whether a colon splits the title in two) is included because language models favor two-part titles (Sebo et al., 2026; Igras-Cybulska and Cybulski, 2026). The share of articles that are open access is not expected to respond to writing tools. It is tracked because a change in how articles are published alters which publishers make up the literature, and such a shift could be mistaken for a change in writing.

Three require the full text and are measured on a sample. Marker words in the body of the article test whether a change seen in abstracts, short texts written to a fixed brief, extends to the article itself. Explicit statements that an AI tool was used to write the article, which publishing-ethics guidance began to call for in 2023 (Committee on Publication Ethics, 2023), show how much of that use authors acknowledge. How the article is assembled (figures, tables, equations, and length) would change if the tools let authors produce and present more analysis in the same time, giving longer articles with more figures and tables.

Each measure is compared with the projection of its own pre-2023 trend rather than with its past average, because scientific writing drifts steadily and a before-and-after difference would mistake that drift for change.

The study’s design is descriptive. For each measure, the study fits the path the measure followed from January 2015 to November 2022 and extends that path beyond the release date; the departure from trend is the gap between what was then observed and this projection. A departure shows that something changed, but it cannot be attributed to a cause, because there is no control group: no comparable body of writing that the tools left untouched. The paper also makes no claim about any individual article. Every measure is a frequency over many articles and cannot be used to label one.

2 Background

The first scholarly response to public chat assistants concerned authorship. Editorials held that a tool cannot take responsibility for a work and so cannot be an author (Stokel-Walker, 2023; Thorp, 2023), position statements settled on disclosure plus human accountability (Committee on Publication Ethics, 2023), and a literature on acceptable use followed (Hosseini et al., 2024; Haman and Školník, 2024; Dergaa et al., 2023). In the indexed literature, the tools came to be acknowledged or cited rather than listed as authors (Gorraiz, 2025). This work set the terms of the debate but measured little; it rests on editorial experience, surveys, and demonstrations that machine-written abstracts can pass human inspection (Else, 2023).

For the first measure, marker words, two approaches exist. Classifying individual texts as machine written is unreliable at scale, because detectors misclassify competent non-native academic prose at high rates (Liang et al., 2023). Measuring aggregate change avoids that problem. Kobak et al. (2025) compared the frequency of every word in millions of PubMed abstracts with a projection of that word’s own earlier trend and found that specific words became abruptly more common after 2022. Two studies by Liang and colleagues estimated, from the whole vocabulary, the share of model-modified text in scientific papers (Liang et al., 2025) and in conference peer reviews (Liang et al., 2024). In both approaches the words are an output of the analysis, not an input. Kousha and Thelwall (2026) took the other route, as this study does: they tracked a fixed set of such words across several bibliographic databases and in full texts, and found their use still rising through 2025. Hubarava and Ineichen (2026) applied an interrupted time series to preprint abstracts and titles in several fields and found longer words, lower readability, and accelerating use of colons in abstracts after 2022. Bao et al. (2025) reported the same pattern in more than 800,000 arXiv abstracts: words that language models prefer became more common, lexical complexity rose, and readability fell, most of all in computer science. Alsudais (2025) found readability falling across arXiv categories after the release, and Alani and Jansen (2026), comparing a fixed list of marker words with a control vocabulary in biomedical abstracts, found that marker use more than doubled while the control vocabulary declined slightly. An earlier study of fabricated papers showed that machine-mediated text leaves lexical traces that predate 2022 (Cabanac et al., 2021).

Disclosure has been measured directly only recently. He and Bu (2026) classified the AI policies of 5,114 journals and estimated AI-assisted writing in 5.2 million abstracts published from 2021 to mid-2025. About 70% of the journals had a policy, most of them requiring disclosure, yet estimated use rose about as fast in journals with a policy as in those without one, and of about 75,000 full texts published from 2023, 76 (0.1%) disclosed the use of an AI writing tool.

This study differs from that work in five ways. The first is the field. The studies above draw on PubMed, on preprint servers whose texts are uniformly available, or on whole databases divided into broad areas, and none of them reports engineering journals separately; in He and Bu (2026), for example, engineering is part of the physical sciences. The second is the comparison. Several of these studies compare the years before and after the release (Kousha and Thelwall, 2026; Bao et al., 2025; Alsudais, 2025), which can mistake steady drift for change. This study, like Kobak et al. (2025) and Hubarava and Ineichen (2026), compares each measure with the projection of its own trend, and it adds control words of the same register, false release dates, and an adjustment for the pandemic months. The third is breadth. Most earlier studies measure wording or style alone; this study places wording beside the bibliographic record of the same articles (authors, institutions, references, pages, publication volume, titles, and open access), which shows whether a change in wording came with a change in how research is produced and published. The fourth is the full-text sample. Earlier full-text analyses use open-access collections (Kousha and Thelwall, 2026) or the copies a script could download (He and Bu, 2026), which over-represent open-access publishers. This study draws articles at random from all the articles of each journal and year, including subscription articles, and weights them back to the corpus. The fifth is the disclosure measure, which separates a statement that a tool was used on the text from a mention of language models as a topic, and which was checked by reading every stored document it flags. The study also documents a data problem that monthly designs share: records that carry only a publication year fall on January 1 and create false peaks.

The other measures have been studied as long-run trends rather than in relation to a specific date. Qualifying and emphatic language are established categories in the study of academic writing (Hyland, 1998), and readability formulas have a long history (Flesch, 1948). Author counts and reference lists have grown for decades, and the literature itself grows several percent per year (Bornmann and Mutz, 2015), which is why a simple before-and-after comparison cannot be read directly. The segmented (interrupted time series) regression used here (Wagner et al., 2002; Lopez Bernal et al., 2016) applies the same projection logic to every measure, with standard errors that allow for correlation between neighboring months (Newey and West, 1987).

3 Data

As defined in the introduction, the corpus is the set of all research articles published from January 2015 to December 2025 by a fixed list of engineering journals, together with each article’s bibliographic record and, where available, its abstract. The window ends in 2025 because records for 2026 were still incomplete when the data were collected. The year left out is likely to matter. In a Wiley survey of 2,430 researchers in August 2025, 84% reported using AI tools in their work, up from 57% a year earlier (Wiley, 2025), and in a survey of more than 10,000 researchers in March 2026, nearly half reported frequent use of AI for gathering information and editing (Springer Nature, 2026). The tools also improved quickly: in one year, frontier models gained about 30 percentage points on Humanity’s Last Exam, a benchmark of difficult expert questions, and one reached gold-medal level at the 2025 International Mathematical Olympiad (Stanford Institute for Human-Centered Artificial Intelligence, 2026). The results here therefore describe the first three years after the release, not the state of writing in 2026.

The journals come from Google Scholar Metrics (Google Scholar, 2025), which lists the twenty journals with the highest h5-index in each subject category. Using a published ranking means any reader can rebuild the list. Fourteen engineering categories were used: general engineering and computer science; food science and technology; manufacturing and machinery; materials engineering; mechanical engineering; sustainable energy; computer networks and wireless communication; game theory and decision science; transportation; automation and control theory; power engineering; structural engineering; civil engineering; and operations research. Each listed journal was matched to its OpenAlex record by title. Conference proceedings were removed, and a journal listed in two categories was kept in the one where it ranks higher, leaving 256 journals. The time series use the 229 of these journals that published at least twenty articles in every year of the window, so that a journal’s founding does not look like a change in the literature.

All English-language works these journals published in the window that OpenAlex (Priem et al., 2022) classified as articles were collected in September 2026, giving 1,839,962 articles; OpenAlex’s article type excludes editorials, letters, and errata but includes some reviews. OpenAlex has become a common source for such studies, although its metadata on languages, document types, and affiliations remain incomplete (Forchino and Torres-Salinas, 2026). Abstracts come from OpenAlex; where OpenAlex has none, the abstract indexed by Semantic Scholar was used when available. About 8.5% of the articles in the 229 journals carry January 1 as their publication date because only the year is recorded; they are kept in annual figures but left out of monthly series, where they would otherwise pile into January.

The four abstract measures need abstract text, which publishers deposit unevenly (Table 1). Three publishers, Elsevier, Springer, and the Royal Society of Chemistry, deposit few abstracts openly and together hold 46% of the articles. Overall, 68.0% of the more than 1.8 million articles have an abstract of at least forty words, the minimum used here, and 14.4% of those abstracts come from Semantic Scholar.

4 Methods

Every measure is treated the same way. Its monthly values from 2015 to 2022 define a trend, the trend is projected forward, and the result is the gap between what was observed from 2023 onward and that projection.

The model has the following terms. Time is counted in months: t is the number of months since January 2015, so t=0 is January 2015, t=12 is January 2016, and so on, and the value of a measure in month t is yt. The last month before the release, November 2022, is denoted T0. The indicator Dt=1 for months after T0 and zero, otherwise. The indicator Ct=1 during the acute phase of the COVID-19 pandemic, March 2020 to June 2021, and zero otherwise. Research and publishing ran under unusual conditions in those months: editorial decisions came faster (Sun et al., 2023) and patterns of collaboration shifted (Ni and Wang, 2025). Some series moved temporarily, for example references per article (Figure 5(c)). The indicator lets the level of each measure shift during those sixteen months without bending the trend fitted to the rest of 2015 to 2022; it is zero after June 2021 and so plays no part in the projection. Separately, m denotes the month of the year, from m=1 for January to m=12 for December, and is used only for seasonal patterns: for m from 2 to 12, the indicator Sm,t=1 when month t falls in month m of its year (for example, S2,t=1 in every February), and zero otherwise. These indicators absorb regular seasonal patterns in publishing, with January as the reference month. The coefficients are β0 to β4 and γ2 to γ12, and εt is the error. The model is

Here β1 is the monthly trend before the release, β2 an immediate change in level, and β3 a change in that trend afterward. Articles published from December 2022 to February 2023 were written before the release and are left out of the estimation. The projection is Equation 1 with β2 and β3 set to zero, and the reported departure is the mean observed value from March 2023 to December 2025 divided by the mean projected value over the same months, minus one. Standard errors allow for correlation across twelve neighboring months (Newey and West, 1987).

Frequencies are pooled within each month: the number of occurrences of the words in question in all abstracts published that month, divided by the total number of words in those abstracts, times 10,000.

The marker list holds 31 words. Some words were listed with inflected forms, and each word is counted together with its listed forms, so that underscore, underscores, and underscoring are one word in every table and figure. The other forms counted are aligning, fostering, harnessing, intricacies, leveraging, meticulously, seamlessly, showcasing, and unveiling. The words are additionally, align, commendable, comprehensive, consequently, crucial, cutting-edge, delve, elevate, foster, furthermore, harness, innovative, intricate, landscape, leverage, meticulous, multifaceted, notably, noteworthy, nuanced, paradigm, pivotal, realm, robust, seamless, showcase, significant, tapestry, underscore, and unveil. Twenty-four come from published lists of words that language models over-use: the excess words that Kobak et al. (2025) found in biomedical abstracts and the adjectives and adverbs that Liang et al. (2024) report for peer reviews. Seven, cutting-edge, furthermore, landscape, paradigm, robust, significant, and tapestry, appear in neither source and were added by the author. Neither source used a fixed list as its method; both found these words by searching the whole vocabulary. Using their findings as a fixed list in another field is therefore this study’s choice, and the result is also checked with lists chosen from the data, described with the results.

Four further words, moreover, vital, holistic, and state-of-the-art, serve as controls. They belong to the same formal register as the marker words but do not appear in published lists of model-favored words. They are counted separately and are not part of the marker measure. If marker words rose only because academic prose in general became more formal, the controls would rise too.

For the full vocabulary, every word is grouped with its inflected forms using a standard English stemming algorithm, so that, for example, enhance, enhanced, and enhancing count as one word. For each word w and year k, let fw,k be its frequency per million words in a sample that takes at most 400 abstracts from each journal in each year, so that no journal’s subject matter dominates. A straight line fitted to fw,k over 2015 to 2022 gives projected values f^w,k for later years. The excess ratio of word w is the observed mean over 2023 to 2025 divided by the projected mean over the same years,

A value of one means the word followed its own trend; two means it appeared twice as often as the trend implied. Words occurring fewer than 400 times in the sample are left out, and ratios are reported only when the projection is at least one per million words, because a near-zero denominator makes any ratio meaningless.

The remaining abstract measures use every word of the abstract, not the marker list. Qualifying words are words that soften a claim, such as may, might, and suggests; emphatic words strengthen one, such as clearly, substantially, and significantly. Both follow Hyland (1998) and are counted per 10,000 words. Word length is the share of words with nine or more letters. Word variety is the number of distinct words in each block of 100 consecutive words, averaged over the blocks of an abstract; counting within fixed-size blocks keeps longer abstracts from looking less varied merely because they are longer. Readability is the Flesch-Kincaid grade level (Flesch, 1948), which rises with syllables per word and with words per sentence. The abstracts supplied by OpenAlex keep word order but lose punctuation, so sentence length cannot be measured; it is set to 22 words for every abstract. The grade therefore reflects syllables per word only, and only its change over time is interpreted.

The bibliographic measures come directly from each article’s record: the numbers of authors, institutions, and references, the page count, the number of articles published per month, whether the article is open access, and two features of the title, its length in words and whether it contains a colon. A colon splits a title into a main title and a subtitle; such titles are a long-studied feature of academic writing (Lewison and Hartley, 2005; Hartley, 2007), and language models produce them more often than human authors do (Sebo et al., 2026; Igras-Cybulska and Cybulski, 2026).

5 Results

5.1 Marker words

Marker words departed from their trend more than any other measure. From 2015 to 2022 their frequency rose slowly, from 22.3 to 25.9 per 10,000 words, and the annual values stayed within 2.5% of the fitted trend (Table 2). From 2023 the two separated: by 2025 the observed frequency was 55.4 against a projection of 27.0. Averaged over March 2023 to December 2025, marker words appeared 81.8% more often than projected. Over the same months the four control words appeared 15.6% less often than their projection.

Figure 1 shows the monthly series. The rise was gradual rather than immediate: the series left the projection band in mid-2023 and climbed through 2024, as expected when writing, review, and publication take months. Accordingly, the model finds a clear change in trend (p<0.001) but no statistically significant jump at the release date (p=0.14). The figure therefore shows the observed series and the projection only. The month-to-month pattern is regular: across calendar months the pre-2023 averages range only from 23.2 to 24.1 per 10,000 words once records dated only by year are excluded; including them would raise every January by 5% over the other months.

The series peaked in June 2025 at 58.4 and declined to 51.5 by December. The decline is not in the data’s composition: mean abstract length (189.4 and 189.5 words) and the share of abstracts taken from Semantic Scholar (5.0% and 4.5%) were flat between the first and second halves of 2025. It is in the words. From the first to the second half of 2025, additionally fell 28%, crucial 21%, innovative 21%, and showcase 34%, while robust rose 32% and paradigm 78%. The mix of favored words is shifting, which is consistent with changes in the tools or in how authors use them, though this study cannot separate the two. A similar pattern appeared in arXiv abstracts, where words publicly identified as typical of language models, such as delve, fell soon after they were publicized while others kept rising, which Geng and Trotta (2025) attribute to authors adapting how they use the tools. Emphatic words kept rising and qualifying words kept falling over the same months.

A rise in a chosen list is weak evidence on its own; the question is whether it is confined to the expected words. Table 3 shows that it largely is. Underscore appeared 24.0 times as often as projected and nine other marker words between 2.9 and 5.8 times, while the controls stayed between 0.81 and 1.14. The boundary is graded: common connectives in the list such as furthermore (1.16) and consequently (1.15) rose only modestly. The word most associated with machine-written prose in popular accounts, delve, is rare in engineering abstracts: it rose from 0.7 to 8.1 occurrences per million words, with a projection below one per million, so it has no ratio.

Figure 2 places the marker and control words within the whole vocabulary. Most marker words lie above the bulk of other words, and the controls lie at one, with ordinary words. The marker words are not a separate cluster, however: many other words also rose, so the list captures part of a broader shift rather than all of it.

The change reached the typical article. Pooling all abstracts from all journals and years on each side of the release (Table 4), the share of abstracts with no marker word at all fell from 67.0% to 45.6%, the median rose from zero to 42.0 per 10,000 words, and every upper percentile rose by more than half.

5.2 Robustness of the marker-word result

Table 5 reports five kinds of check. First, the model was refitted with a false release date in November 2018, 2019, or 2020, using only data from before the real release; the departures are 4% to 7%, which reflects slight curvature in the earlier trend and is a small fraction of the 82% found at the real date. Second, removing each of the 228 journals with abstract text in turn (one of the 229 has none) leaves the estimate between 79.4% and 83.1%, and removing each of the 31 marker words in turn leaves it between 77.3% and 97.2%. Third, the sample was varied: giving each journal a fixed weight in every month, so that changes in which journals publish most cannot drive the result, gives 72.4%; using only journals whose abstracts are nearly complete, only abstracts from OpenAlex, all 256 journals, or the records dated only by year keeps the estimate between 80.4% and 83.9%. Fourth, dropping the level-change term from Equation 1 gives 68.9%. Fifth, the word list was varied. The seven words added by the author are common in engineering prose for reasons unrelated to language models (significant alone accounts for 27.7% of marker occurrences before the release, furthermore for 20.6%, and robust for 11.4%), and together they make up 62.6% of those occurrences. They dilute the measure rather than drive it: the 24 words taken from the published sources depart by 130.1% from their trend, and the seven added words by 47.3%. In addition, two word lists were chosen from the data by a fixed rule: words in use in all fourteen categories before 2023 that rose above their own projection in at least twelve of them, excluding function words. The held-out list was chosen using only abstracts published before April 2024 and evaluated only on later months; the disjoint list shares no word with the marker or control lists. Both depart far more than the marker list, as expected for words selected because they rose, but the held-out list shows that words chosen before the evaluation period kept rising afterward.

A change in what these journals publish would be expected to concentrate in some fields. Instead, all fourteen categories depart upward and significantly (Figure 3), from 38.4% in operations research to 96.2% in general engineering and computer science. Fields that share little in topic, authors, or publisher moved in the same direction at the same time.

Splitting the series by the country of the corresponding author shows both groups rising: 57.5% for authors based in countries where English is the main language of research and 87.7% for others. Country is a poor stand-in for an author’s language background, so this split is descriptive only.

5.3 Qualifying and emphatic words, word length, and readability

The other abstract measures moved together with marker words (Table 6, Figure 4). Qualifying words, the usual way scientific writing signals uncertainty, fell 13.0% below projection (Figure 4(a)), while emphatic words rose 20.1% (Figure 4(b)). Words became longer: the share of words with nine or more letters rose 10.8% (Figure 4(c)), word variety 4.3% (Figure 4(d)), and the readability grade 5.9% (Figure 4(e)). Abstracts became slightly shorter than their rising trend implied (−4.0%; Figure 4(f)).

5.4 Bibliographic measures

If the literature had changed broadly, the bibliographic measures would move with the textual ones. Most did not (Table 6, Figure 5). Authors per article (Figure 5(b)), institutions per article, references (Figure 5(c)), and pages (Figure 5(d)) all stayed within 4% of trend, so the changes anticipated in Section 1 did not appear: author counts did not fall, and reference lists did not grow. Monthly publication volume was about 70% higher in 2023–2025 than in 2015–2022, but it grew along its existing trend (Figure 5(a)); its departure of −3.7% is not significant (p=0.81). This measure counts published articles only: a surge in submissions offset by more rejections would look the same, and submission data are not public. Two measures moved by much more. The share of titles with a colon rose 23.9% above trend (Figure 5(e)), in line with the preference of language models for two-part titles noted in Section 4, and open access grew more slowly than projected (−21.2%; Figure 5(f)). Most of that gap reflects a change in which publishers the articles come from. MDPI, all of whose articles are open access, fell from 29.0% of the articles in 2022 to 19.4% in 2025, while Elsevier, where 25.7% of articles were open access in 2025, rose from 40.4% to 46.9%. Had each publisher kept its 2022 share of the articles, the open-access share in 2025 would have been 48.7% rather than 42.4%, close to its 2022 level of 48.3%. Open-access status is also often recorded some time after publication, which adds to the gap for the newest articles.

The sharp fall in references per article at the end of 2025 (Figure 5(c)) comes from incomplete records rather than shorter reference lists. Records with no reference list rose from at most 1.8% of articles in each month from January to August 2025 to 6.1% in December, and records with a list averaged 48.9 references in December against 55.4 to 57.2 earlier in the year. The newest records are still being completed, which is why 2026 is excluded. Ending the series in August 2025 moves the references departure from −3.4% to −1.7%, although the change in trend is then borderline (p=0.049, against p=0.08 for the full series).

5.5 Evidence from article bodies

An abstract is a few hundred words written to a fixed brief, so a change there need not describe the article. To test whether the change reaches the body of the article, the study measured the full text of a random sample of articles drawn from the same corpus. The design called for 10,000 articles. Within each year, every journal’s share of the sample matched its share of that year’s articles, and the years from 2023 onward were sampled at a higher rate, about 0.9% of their articles against 0.2% to 0.4% for 2015 to 2022, because the after-release side covers three years against eight. Within each journal-year, articles were drawn at random. Many drawn articles could not be obtained, because publishers block automated downloads and most articles are behind a subscription. A journal-year left short was refilled with further random draws from its articles not yet drawn, and then with open-access articles drawn at random from the same journal and year, which make up 18.2% of the sample. Every article obtained and measured was kept, so the sample holds 12,866 articles, 5,396 published before the release and 7,470 from March 2023 onward, from 250 of the corpus’s 256 journals and 29 of its 31 publishers. The other six journals, which together hold 0.5% of the corpus, have no article in the sample. Three entered the corpus only after the sample was drawn, when errors in matching journal names were corrected; one has fewer than 20 articles in every year, too few to receive an allocation; and for two, none of the drawn articles could be obtained. Legal open copies were located through Unpaywall and downloaded by a script that identifies itself, follows each site’s robots.txt rules, and does not request sites that refuse automated access; 1,330 free-to-read articles without a reuse license were measured in memory and not kept, and 1,091 articles were manually downloaded by the author in a web browser. Most automated copies come from repositories and are often the accepted manuscript rather than the published version. Before measurement each document was checked: the article’s title had to appear on its first two pages, and supplementary files, documents whose text could not be extracted, front matter, and documents under 500 words were set aside. These checks removed 1.0% of the documents obtained before the release and 0.9% of those obtained after. Every estimate in Table 7 weights each article by its journal-year’s count in the corpus divided by its count in the sample. The weights undo both the higher sampling rate after 2022 and the uneven filling of journal-years, so the comparison describes the literature rather than the sample; the intervals come from resampling journals.

Marker words rose 98.6% in article bodies, against 81.8% in abstracts, while the change in control words cannot be distinguished from zero (+4.5%). Body text, about forty times longer than an abstract and collected by an entirely separate process, moved the same way. The result holds within journals: comparing each journal with itself, among the 191 journals with at least five articles on each side, gives +94.9%. Leaving out the open-access substitutes gives +104.3%, and keeping only the 2,750 publisher versions gives +122.9%, so neither the substitutes nor the repository copies produce the result. Raising the title-match and English-share thresholds of the quality checks moves the estimate by less than half a point (+98.6% to +98.7%). Emphatic words and long words rose in bodies as in abstracts. Qualifying words did not fall in bodies (+2.1%, interval including zero), so that change appears confined to abstracts. Bodies became longer and carried slightly more figures and tables, in the direction expected if the tools let authors present more analysis; the change in equations is not distinguishable from zero.

A disclosure statement is the one measure that needs no inference about style: it records that an article states, outside its reference list, that an AI tool was used on its text. Three forms count: the standard declaration “during the preparation of this work the authors used [tool] in order to [purpose],” a declaration heading followed by a statement of use, and any sentence that states use as done, names a tool, and names a purpose concerning the text, such as its language, grammar, or readability. Statements of non-use and quoted journal policy do not count, and every stored document the rule flags was read to confirm it. No article published from 2015 to 2022 carries such a statement; one in 2023 does, and the share reaches 1.1% of articles in 2024 and 3.7% in 2025. These shares count disclosed use only and understate even that: disclosure is voluntary at many journals, and a statement worded outside the three forms is missed. They are therefore a lower bound on use. Disclosure is also concentrated in one publisher. In 2025, 7.0% of the sampled Elsevier articles carry a statement, against 2.2% at MDPI, 1.4% at IEEE and at Wiley, and none at the American Chemical Society or the Royal Society of Chemistry; 79 of the 98 disclosing articles are Elsevier’s, and without Elsevier the weighted share in 2025 is 1.3%. Elsevier asks authors to place a separate declaration on the use of generative AI in the manuscript (Elsevier, 2026), so the share largely records where authors are asked to declare. The same reason likely explains why the share is far above the 0.1% that He and Bu (2026) found in full texts published from 2023: their copies came from download links in OpenAlex, which lead mostly to open-access versions, whereas this sample includes subscription articles, among them Elsevier’s. Separately, the share of articles whose body mentions a language model at all, mostly as a subject of the research, rose from 0.2% to 3.7%.

6 Discussion

Four features support reading the result as a change in wording. The departure follows eight years in which the series stayed close to its trend. It is specific: the control words, from the same register, did not rise, which argues against a general shift toward formal prose or an artifact of the method. It does not depend on the method or on any one journal: false release dates produce small departures and no single journal moves the estimate by more than a few points. Last, it is confined to wording: how research is organized and credited, as seen in authors, references, and pages, did not change.

The design shows a departure from trend, not its cause. Editorial or template changes across publishers, shifts in the mix of research, and the growth of non-native English authorship would all leave similar traces; the control words make some of these less likely but do not rule them out. The release date marks the start of broad public use of such tools rather than of any single product. The result also describes a literature, not any paper in it: a shift in average frequency is compatible with a few authors changing a great deal or many changing a little.

The direction and size agree with Kobak et al. (2025) for biomedical abstracts. Two literatures with different conventions, publishers, and data showing a similar change is more convincing than either alone. Engineering differs in one respect: without a uniform public abstract index, any text-based result rests on the machine-readable part of the literature, which here under-represents the largest publisher.

The findings have three implications for measurement and policy. First, monthly series built from bibliographic records need care with dates: records that carry only a year are common at some publishers and, left in, create January peaks that resemble a seasonal pattern in writing. Second, counting mentions of language models in full text measures research about the tools, not use of them; a disclosure measure must require a statement of use, and must set aside statements of non-use, which journals increasingly print, and quoted journal policy. Even so measured, disclosure is a lower bound, because many journals do not require it. Third, screening individual manuscripts for the words studied here would be easy to evade, would penalize writers whose style is naturally formal, and would fall hardest on non-native English writers, whereas tracking a whole body of writing is cheap and can inform journal policy without judging any one paper.

7 Limitations

Publication dates stand in for writing dates, which are unavailable, so the measured change lags and smooths any change in practice. OpenAlex tends to record the earliest date it finds for a work (Kousha and Thelwall, 2026), which brings the dates closer to writing but does not close the gap. The abstract measures rest on the two-thirds of articles with a usable abstract, which under-represent the largest publisher; if wording changed differently there, the direction of the resulting bias is unknown. The full-text measures rest on a sample of about 0.7% of the corpus, drawn at random within journal-years but completed in part with open-access articles, and most copies obtained automatically are repository versions, often the accepted manuscript rather than the published text; restricted to publisher versions, the marker estimate is larger, not smaller. The projection extends each series’ own history and cannot separate this change from others occurring at the same time. The marker list is the author’s choice, built mostly from studies of other fields; the control words and the lists chosen from the data address that choice but do not remove it. Readability uses an assumed sentence length, and all measures assume English text.

8 Conclusion

Engineering abstracts changed in word choice after the end of 2022, and in a specific way. Words associated with language model output rose 82% above their own trend, while control words of the same register did not rise. Qualifying words fell, emphatic and longer words rose, and the median abstract went from using none of the marker words to using several. How articles are assembled stayed close to trend. In the full text of a sample of articles, the same words rose by a similar amount, and statements disclosing the use of AI writing tools went from none to 3.7% of articles in 2025. The change appears in every subject category. What this study cannot establish is the cause. Natural next steps are full-text measurement at larger scale, dating by submission where publishers provide it, and the same analysis in other fields that lack a uniform abstract index.

CRediT authorship contribution statement

Taner Cokyasar: Conceptualization, Methodology, Software, Data curation, Formal analysis, Investigation, Visualization, Writing – original draft, Writing – review & editing.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

The author declares that he has no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The replication package contains the code that builds the corpus and runs every analysis, the journal list with every exclusion and correction, the identifiers of all 1,839,962 articles, the record of the full-text sample with its measurements, and the computed results. It is available at https://github.com/tcokyasar44/What_changed_in_engineering_writing_after_AI, and the version used here is archived on Zenodo (Cokyasar, 2026). Bibliographic data come from OpenAlex (Priem et al., 2022), which places its data in the public domain. Abstract and article text is not redistributed; the code retrieves it again. The analysis is deterministic for a fixed data snapshot.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

During the preparation of this work the author used Claude (Anthropic), with ChatGPT (OpenAI) and Gemini (Google) as cross-checks, in order to write and check the data collection and analysis software, draft parts of the manuscript, and verify the references. After using these tools, the author reviewed and edited the content and takes full responsibility for the content of the published article. No language model read, summarized, or judged any article in the study: every measurement comes from deterministic counting software, and rerunning it on the same data returns the same numbers.

References

• Alani and Jansen (2026) Alani, N.H.S., Jansen, B.J., 2026. Are scientists starting to sound like ChatGPT? Stylistic diffusion in biomedical abstracts in the LLM era. Machine Learning and Knowledge Extraction 8, 223. doi:10.3390/make8080223.
• Alsudais (2025) Alsudais, A., 2025. Exploring the change in scientific readability following the release of ChatGPT. Journal of Informetrics 19, 101679. doi:10.1016/j.joi.2025.101679.
• Bao et al. (2025) Bao, T., Zhao, Y., Mao, J., Zhang, C., 2025. Examining linguistic shifts in academic writing before and after the launch of ChatGPT: A study on preprint papers. Scientometrics 130, 3597–3627. doi:10.1007/s11192-025-05341-y.
• Bornmann and Mutz (2015) Bornmann, L., Mutz, R., 2015. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the Association for Information Science and Technology 66, 2215–2222. doi:10.1002/asi.23329.
• Cabanac et al. (2021) Cabanac, G., Labbé, C., Magazinov, A., 2021. Tortured phrases: A dubious writing style emerging in science. evidence of critical issues affecting established journals. arXiv (Cornell University). doi:10.48550/arxiv.2107.06751. preprint.
• Cokyasar (2026) Cokyasar, T., 2026. Replication package: What changed in engineering writing after generative AI chat assistants. Zenodo. doi:10.5281/zenodo.24048694. software, version 1.0.7.
• Committee on Publication Ethics (2023) Committee on Publication Ethics, 2023. Authorship and AI tools: COPE position statement. https://publicationethics.org/cope-position-statements/ai-author.
• Conroy (2023) Conroy, G., 2023. Scientific sleuths spot dishonest ChatGPT use in papers. Nature doi:10.1038/d41586-023-02477-w.
• Dergaa et al. (2023) Dergaa, I., Chamari, K., Zmijewski, P., Ben Saad, H., 2023. From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport 40, 615–622. doi:10.5114/biolsport.2023.125623.
• Else (2023) Else, H., 2023. Abstracts written by ChatGPT fool scientists. Nature 613, 423–423. doi:10.1038/d41586-023-00056-7.
• Elsevier (2026) Elsevier, 2026. Generative AI policies for journals. https://www.elsevier.com/about/policies-and-standards/generative-ai-policies-for-journals. Accessed November 8, 2026.
• Flesch (1948) Flesch, R., 1948. A new readability yardstick. Journal of Applied Psychology 32, 221–233. doi:10.1037/h0057532.
• Forchino and Torres-Salinas (2026) Forchino, M.V., Torres-Salinas, D., 2026. The OpenAlex database in review: Evaluating its applications, capabilities, and limitations. Journal of Informetrics 20, 101800. doi:10.1016/j.joi.2026.101800.
• Geng and Trotta (2025) Geng, M., Trotta, R., 2025. Human-LLM coevolution: Evidence from academic writing, in: Findings of the Association for Computational Linguistics: ACL 2025, Association for Computational Linguistics, Vienna, Austria. pp. 12689–12696. doi:10.18653/v1/2025.findings-acl.657.
• Google Scholar (2025) Google Scholar, 2025. Google Scholar Metrics: Engineering and computer science. https://scholar.google.com/citations?view_op=top_venues&hl=en.
• Gorraiz (2025) Gorraiz, J., 2025. Acknowledging the new invisible colleague: Addressing the recognition of Open AI contributions in scientific publishing. Journal of Informetrics 19, 101642. doi:10.1016/j.joi.2025.101642.
• Haman and Školník (2024) Haman, M., Školník, M., 2024. Using ChatGPT to conduct a literature review. Accountability in Research 31, 1244–1246. doi:10.1080/08989621.2023.2185514.
• Hartley (2007) Hartley, J., 2007. Planning that title: Practices and preferences for titles with colons in academic articles. Library & Information Science Research 29, 553–568. doi:10.1016/j.lisr.2007.05.002.
• He and Bu (2026) He, Y., Bu, Y., 2026. Academic journals’ AI policies fail to curb the surge in AI-assisted academic writing. Proceedings of the National Academy of Sciences 123, e2526734123. doi:10.1073/pnas.2526734123.
• Hosseini et al. (2024) Hosseini, M., Rasmussen, L.M., Resnik, D.B., 2024. Using AI to write scholarly publications. Accountability in Research 31, 715–723. doi:10.1080/08989621.2023.2168535.
• Hubarava and Ineichen (2026) Hubarava, H., Ineichen, B.V., 2026. Accelerated drift of scientific prose in the LLM era: Evidence from an interrupted time-series analysis. arXiv (Cornell University). doi:10.48550/arXiv.2610.26008. preprint.
• Hyland (1998) Hyland, K., 1998. Boosting, hedging and the negotiation of academic knowledge. Text – Interdisciplinary Journal for the Study of Discourse 18. doi:10.1515/text.1.1998.18.3.349.
• Igras-Cybulska and Cybulski (2026) Igras-Cybulska, M., Cybulski, A., 2026. Did LLMs colonize scientific titles? Fifty years of titular colonicity. Research Square. doi:10.21203/rs.3.rs-11137993/v1. preprint.
• Kobak et al. (2025) Kobak, D., González-Márquez, R., Horvát, E.A., Lause, J., 2025. Delving into LLM-assisted writing in biomedical publications through excess vocabulary. Science Advances 11. doi:10.1126/sciadv.adt3813.
• Kousha and Thelwall (2026) Kousha, K., Thelwall, M., 2026. How much are LLMs changing the language of academic papers after ChatGPT? A multi-database and full text analysis. Scientometrics doi:10.1007/s11192-026-05601-5.
• Lewison and Hartley (2005) Lewison, G., Hartley, J., 2005. What’s in a title? Numbers of words and the presence of colons. Scientometrics 63, 341–356. doi:10.1007/s11192-005-0216-0.
• Liang et al. (2024) Liang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z.X., McFarland, D.A., Zou, J., 2024. Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews. arXiv (Cornell University). doi:10.48550/arxiv.2403.07183. preprint.
• Liang et al. (2023) Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., Zou, J., 2023. GPT detectors are biased against non-native English writers. Patterns 4, 100779. doi:10.1016/j.patter.2023.100779.
• Liang et al. (2025) Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., Yang, D., Potts, C., Manning, C.D., Zou, J., 2025. Quantifying large language model usage in scientific papers. Nature Human Behaviour 9, 2599–2609. doi:10.1038/s41562-025-02273-8.
• Lopez Bernal et al. (2016) Lopez Bernal, J., Cummins, S., Gasparrini, A., 2016. Interrupted time series regression for the evaluation of public health interventions: a tutorial. International Journal of Epidemiology doi:10.1093/ije/dyw098.
• Newey and West (1987) Newey, W.K., West, K.D., 1987. A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55, 703. doi:10.2307/1913610.
• Ni and Wang (2025) Ni, R., Wang, J., 2025. Navigating disruptions: The effects of the pandemic on scientific collaboration and research novelty in Hong Kong. Journal of Informetrics 19, 101656. doi:10.1016/j.joi.2025.101656.
• OpenAI (2022) OpenAI, 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt.
• Priem et al. (2022) Priem, J.R., Piwowar, H., Barrett, J., Demes, K., McCafferty, N., Meyer, C., Orr, R., Portenoy, J., 2022. OpenAlex snapshot. arXiv (Cornell University). doi:10.48550/arxiv.2205.01833. preprint.
• Sebo et al. (2026) Sebo, P., Nie, B., Wang, T., 2026. How ChatGPT writes scientific titles in medical research: Structural and content differences compared to human authors. Journal of the Medical Library Association 114, 222–237. doi:10.5195/jmla.2026.2266.
• Springer Nature (2026) Springer Nature, 2026. The state of AI use among researchers. https://group.springernature.com/br/group/media/press-releases/early-career-researchers-lead-shift-towards-ai-first-discovery/52988132. White paper with Fudan University and the Shanghai Academy of AI for Science; survey of 10,480 researchers fielded in March 2026.
• Stanford Institute for Human-Centered Artificial Intelligence (2026) Stanford Institute for Human-Centered Artificial Intelligence, 2026. The 2026 AI index report. https://hai.stanford.edu/ai-index/2026-ai-index-report.
• Stokel-Walker (2023) Stokel-Walker, C., 2023. ChatGPT listed as author on research papers: many scientists disapprove. Nature 613, 620–621. doi:10.1038/d41586-023-00107-z.
• Sun et al. (2023) Sun, Z., Liu, S., Li, Y., Ma, C., 2023. Expedited editorial decision in COVID-19 pandemic. Journal of Informetrics 17, 101382. doi:10.1016/j.joi.2023.101382.
• Thorp (2023) Thorp, H.H., 2023. ChatGPT is fun, but not an author. Science 379, 313–313. doi:10.1126/science.adg7879.
• Van Noorden and Perkel (2023) Van Noorden, R., Perkel, J.M., 2023. AI and science: what 1,600 researchers think. Nature 621, 672–675. doi:10.1038/d41586-023-02980-0.
• Wagner et al. (2002) Wagner, A.K., Soumerai, S.B., Zhang, F., Ross-Degnan, D., 2002. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 27, 299–309. doi:10.1046/j.1365-2710.2002.00430.x.
• Wiley (2025) Wiley, 2025. AI adoption jumps to 84% among researchers as expectations undergo significant “reality check”. https://newsroom.wiley.com/press-releases/press-release-details/2025/AI-Adoption-Jumps-to-84-Among-Researchers-as-Expectations-Undergo-Significant-Reality-Check/default.aspx. Press release on the ExplanAItions survey of 2,430 researchers.

注意转载授权 · 阅读原文