Enterprise Information Integration (2024)

Web Design and DevelopmentStandards and ProtocolsEnterprise Information IntegrationTechnologyDatabasesData AdministrationHuman ResourcesWorkflowMaster Data ManagementData CentersIntranetSearchingInformation Retrieval

AI News by Category

Check blogSoftwareRead news

3 days ago

Luka

Web Design and DevelopmentStandards and ProtocolsEnterprise Information IntegrationTechnology

Oxylabs introduces the first AI copilot for public web data collection

Oxylabs Launches OxyCopilot to Revolutionize Web Data Collection

September 27, 2024 6:44:58PM

AZoRobotics.com

OxylabsSep 27 2024

Oxylabs, a premium web intelligence collection platform, announced the launch of a new AI-driven solution, OxyCopilot, that is designed to help its users save time and money spent on complex web data acquisition tasks. The industry's first artificial intelligence (AI) assistant for scraping is a part of Oxylabs' Web Scraper API, an all-in-one public web data collection platform.

Built on a combination of AI and proprietary Oxylabs' technology, OxyCopilot will enable both web scraping professionals and those who have little experience in web data collection to generate instant parsing instructions and requests for the Web Scraper API using nothing more than a URL and natural language prompts.

"In Summer, together with Censuswide, we carried out a wide survey of developers and web scraping practitioners in the UK and US. The data showed that 74% of businesses faced an increasing demand for public web data over the last year. Unfortunately, building the necessary infrastructure and maintaining data parsers remains a heavy challenge for many companies -- a proper parsing process alone can take up to 40 development hours per week for the tech teams. With OxyCopilot, we aimed to help our clients collect the data they need more easily", says Julius Cerniauskas, CEO at Oxylabs.

According to Cerniauskas, the newly-released AI assistant helps to level the playing field. Previously, the barrier to entry for smaller companies was tough because they needed to hire a team of web scraping professionals that are hard to find on the market. By saving development hours spent on repetitive web scraping tasks, businesses can devote more attention to data quality, analysis, and innovation.

Moreover, using Oxylabs' unified web scraping platform, they don't need to maintain costly infrastructure, such as servers -- a challenge that 61% of professionals identify as the most pressing when collecting data on a large scale.

"We always aspired to be industry leaders through innovation, and taking a step further in the field of AI was a natural business decision for us. OxyCopilot is unique in its design, and we are currently patenting technological implementations behind it. Most importantly, it functions as a part of a broader platform that includes other AI-powered solutions, from proxy management to web unblocking. With the help of AI machine learning (ML), we are moving towards automating the entire public web data collection process", adds Cerniauskas.

Oxylabs holds over 100 patents globally, part of them covering technologies that have AI and ML implementations. In 2023, the company obtained ISO/IEC 27001:2017 standard for excellence in information security management.

You can read more about building the industry's first web scraping AI copilot here.

Source:

Oxylabs

View original article

September 27, 2024 6:44:58PM

AZoRobotics.com

The Tie Launches Polkadot Ecosystem Dashboard on The Tie Terminal - Decrypt

Bill Laboon, Director of Education at Web3 Foundation, on Polkadot's 1000 Referendums, Centralized Points in Decentralized Polkadot, and Sprinkling Blockchain Dust | Ep. 369

September 13, 2024 6:41:08PM

cryptonews.com

Cryptonews Podcast host Matt Zahab sat down for an exclusive interview with Bill Laboon, the Director of Education and Governance Initiatives at the Web3 Foundation, a not-for-profit entity supporting the Polkadot ecosystem.

Laboon discussed everything Polkadot and various aspects of its ecosystem, the real competitors to Polkadot and the Web3 Foundation, and how the Foundation and Parity turned out to be centralized points in a decentralized project.

He talked about the community reaching a whopping 1,000 referendums and crypto needing more end-user-focused apps if it's to go mainstream.

Will the Real Competitor Please Stand Up

Web3 Foundation is a not-for-profit company, meaning it doesn't aim to make money. It has funds that the team uses for day-to-day operating expenses.

Instead, the Foundation was founded to bring about the decentralized internet, a world where people own their own data, Laboon told Matt.

One of the many ways they're accomplishing this is by supporting the Polkadot ecosystem.

To help shepherd Polkadot, the Foundation provides education, outreach, and grants. It also does coordination and interacts with the community.

These are not Polkadot's competitors, Laboon said. "We're working towards the same goal in different ways, and we do a lot of interoperability with them."

Actually, argued the director, "our competitor really is the Web2 world. We have different ways of going about our way of winning."

Moreover, while Web3 Foundation focuses on Polkadot, it always had a "broader mandate."

Therefore, they have provided funding to people building other projects and researching other ecosystems, which doesn't directly benefit Polkadot but is more generalized.

Anyone can issue a referendum. The large community of DOT holders makes all decisions by voting.

The ecosystem is "entirely decentralized," Laboon said. "The control is entirely in the hands of the DOT holders," including any upgrades, updates, or spending from the treasury.

Polkadot has its OpenGov governance system where one DOT equals one vote.

"It's just a very direct democracy. And this has been very challenging," Laboon remarked.

Importantly, there is no dependency on developers to implement referendums.

Instead, users issue referendums in code, which will execute following a successful vote - and "can actually change the rules of the blockchain, which I think is really neat."

"There's no way to stop it," Laboon added.

Web3 Foundation does look through referenda, double-checking if something seems malicious.

They interact with governance teams and answer people's questions, but they prefer not to share opinions unless something is dangerous.

Furthermore, "we've figured out what the community does and what they're interested in versus what's actually good for the long-term health of the network," the director noted. "It's been a very interesting experience in democracy."

Even if the Foundation expresses a positive opinion on a referendum, the community can vote it down. They recently did just that, arguing that an upgrade should run longer on the testnet.

This allowed more time for the wallets to understand the upgrade, given that Polkadot is not a single blockchain but a system of 51 blockchains (aka parachains).

"Three weeks later, [the referendum] was reissued, and the network was upgraded," Laboon said.

Sprinkling a Bit of Blockchain Dust Doesn't Make It a Blockchain App

The crypto market needs useful and user-facing apps, Laboon argued.

We have plenty of infrastructure in the space and many people "building really cool tech for the developers."

"What we don't always have is people that are building things that are useful for the end user on top of that stuff that the nerds are building," the director pointed out.

And this is necessary for the process of adoption.

Developers often promise "smart contracts and immutability," creating "unstoppable" applications, but then they go and code their smart contracts to actually be stoppable, argued Laboon.

There is a lot of dependence on the teams building projects. This makes the project vulnerable and is generally an issue.

This vulnerability is not only in the form of devs abandoning a project; it could also be something much more malicious.

Therefore, projects need teams that will ensure they're "actually creating unstoppable applications and not something that somebody sprinkled a little blockchain dust on and said, okay, well, now it's a blockchain application."

As explained above, Polkadot is unstoppable. Even if Parity, the main engineering team behind Polkadot, were to just up and leave, it would not be the project's end. Polkadot would be able to find another team, Laboon claimed.

Centralized Points in the Decentralized Ecosystem

Polkadot established the Decentralized Futures grant program last year, allocating $20 million and 5 million DOT to kickstart economically independent, active participants in the Polkadot ecosystem.

"The basic outline," Laboon said, is "we give them cake, they build good products."

Last year, the team came to a major realization. Web3 Foundation and Parity were actually "centralized points in the ecosystem we wanted to decentralize."

Centralized entities within a system, no matter how far they distance themselves from decision-making, still maintain "soft power in the ecosystem."

Therefore, to turn themselves into decentralized elements, they established various teams.

These include marketing, business development, education, documentation, investments, and more.

These are small teams with great products, which needed funding to start. When receiving the grant from the program, they agree to several milestones, getting a portion of the grant with each milestone reached.

"So far, it's been great. We've had some really good successes coming out of that," Laboon commented.

They had 250 applications and gave out exactly 20% (50 grants).

The Decentralized Futures program is now over, but "we may have another one in the future," he concluded.

Laboon is the author of two books: 'A Friendly Introduction to Software Testing,' an undergraduate textbook, and 'Strength in Numbers: A Novel of Cryptocurrency,' a near-future novel set in a world where cryptocurrency has eliminated traditional money.

View original article

September 13, 2024 6:41:08PM

cryptonews.com

Customers Unify Critical Data and Processes with New Oracle Primavera Unifier Capabilities

Customers Unify Critical Data and Processes with New Oracle Primavera Unifier Capabilities - CRN - India

August 7, 2024 8:39:03AM

CRN - India

A core part of the Oracle Smart Construction Platform, customers continue to rely on Primavera Unifier to digitise and automate capital planning, project execution, and facilities management, to help improve predictability and reliability. To speed execution and efficiency, Oracle has released the Primavera Unifier Accelerator base. With 65 new preconfigured business processes, this new configuration helps customers save time and can eliminate costly process design and custom reporting. When used as a starting point, the Unifier Accelerator has helped early-adopted users expedite implementation time by up to 75%1.

With the new Unifier Accelerator configuration, which includes 250 out-of-the-box reports and dashboards, Primavera Unifier helps organisations better manage cash flow, forecasts, contracts, scope changes, project administration including daily reports, RFI's, and more.

"Northwell Health strives to lead and innovate, not only in the care we provide, but also in how we build and maintain our healthcare facilities," said Ian Jablonski, Director of Strategy for Systems, Data, and Services for Facilities Services, Northwell Health. "Oracle Primavera Unifier has been a key component, supporting the unification of our processes and data on our journey toward maximising efficiency across more than 900 Northwell facilities through optimised processes, system standardisation, data integration, and predictive analytics."

Extensions of Unifier Accelerator process-specific capabilities to help improve compliance and outcomes include NEC4 contract management with preconfigured contract templates and reports. Additionally, Earned Value Management, Progress Measurement, Advanced Work Packaging, and Facilities Management are other capabilities supported by Primavera Unifier and the Unifier Accelerator. The connected architecture can help unify cost, contract, change, schedule milestone, workflow, and document management processes within a single solution. Many of these processes are now supported by the newly updated Primavera Unifier mobile app, including location-based services that can instantly tag on-site inspections to groups and manage issues based on geography more easily.

Connected AI

With integrations to Oracle Primavera P6 and Oracle Primavera Cloud scheduling solutions, organisations using the Unifier Accelerator base configuration can unite all their processes and data in applications across their Oracle Engineering and Construction portfolio, including Oracle Construction Intelligence Cloud. This enables them to leverage built-in predictive analytics and AI insights to better detect and reduce project risk. For example, customers can leverage the Safety Risk Forecast, a custom predictive model that enables users to reduce safety risk by proactively identifying high-risk projects through an incident-warning system. This information enables organisations to take actions that can prevent incidents and losses before they occur.

"The Primavera Unifier Accelerator is a natural new addition from Oracle Construction and Engineering," Vernon Harley, project & portfolio management solutions lead, Accenture. "Given the many cross-industry relevant business processes that are required to drive core project controls and given their learnings from many past implementations of Primavera Unifier, it makes perfect sense for Oracle to offer this configuration package. It will greatly reduce implementation time for clients who need core project controls business processes to be available in production for the end users more quickly."

Integrating systems to unify data

Leveraging Oracle Integration Cloud (OIC) middleware, organisations can also connect Primavera Unifier to its other Oracle and third-party business systems, including enterprise resource planning (ERP) and enterprise asset management (EAM) systems, more consistently, reducing integration time and risk. Users can additionally streamline integrations with mature RESTful Primavera Unifier APIs that support Oracle and third-party middleware solutions with Unifier integration templates now available from the Oracle middleware marketplace.

"It's more important than ever for our customers to quickly recognise the benefits from digitising processes and unifying data," says Mark Webster, senior vice president and general manager, Oracle Construction and Engineering. "With our increased strategic investment in Primavera Unifier and other components of the Oracle Smart Construction Platform, we are transforming the delivery of capital and construction projects by addressing key customer requirements, enhancing interoperability, and bringing the expanded capabilities to market faster."

View original article

August 7, 2024 8:39:03AM

CRN - India

The Data That Powers A.I. Is Disappearing Fast

Apple says it took a 'responsible' approach to training its Apple Intelligence models

July 30, 2024 2:31:39AM

TechCrunch

Apple has published a technical paper detailing the models that it developed to power Apple Intelligence, the range of generative AI features headed to iOS, macOS and iPadOS over the next few months.

In the paper, Apple pushes back against accusations that it took an ethically questionable approach to training some of its models, reiterating that it didn't use private user data and drew on a combination of publicly available and licensed data for Apple Intelligence.

"[The] pre-training data set consists of ... data we have licensed from publishers, curated publicly available or open-sourced datasets and publicly available information crawled by our web crawler, Applebot," Apple writes in the paper. "Given our focus on protecting user privacy, we note that no private Apple user data is included in the data mixture."

In July, Proof News reported that Apple used a data set called The Pile, which contains subtitles from hundreds of thousands of YouTube videos, to train a family of models designed for on-device processing. Many YouTube creators whose subtitles were swept up in The Pile weren't aware of and didn't consent to this; Apple later released a statement saying that it didn't intend to use those models to power any AI features in its products.

The technical paper, which peels back the curtains on models Apple first revealed at WWDC 2024 in June, called Apple Foundation Models (AFM), emphasizes that the training data for the AFM models was sourced in a "responsible" way -- or responsible by Apple's definition, at least.

The AFM models' training data includes publicly available web data as well as licensed data from undisclosed publishers. According to The New York Times, Apple reached out to several publishers toward the end of 2023, including NBC, Condé Nast and IAC, about multi-year deals worth at least $50 million to train models on publishers' news archives. Apple's AFM models were also trained on open source code hosted on GitHub, specifically Swift, Python, C, Objective-C, C++, JavaScript, Java and Go code.

Training models on code without permission, even open code, is a point of contention among developers. Some open source codebases aren't licensed or don't allow for AI training in their terms of use, some developers argue. But Apple says that it "license-filtered" for code to try to include only repositories with minimal usage restrictions, like those under an MIT, ISC or Apache license.

To boost the AFM models' mathematics skills, Apple specifically included in the training set math questions and answers from webpages, math forums, blogs, tutorials and seminars, according to the paper. The company also tapped "high-quality, publicly-available" data sets (which the paper doesn't name) with "licenses that permit use for training ... models," filtered to remove sensitive information.

All told, the training data set for the AFM models weighs in at about 6.3 trillion tokens. (Tokens are bite-sized pieces of data that are generally easier for generative AI models to ingest.) For comparison, that's less than half the number of tokens -- 15 trillion -- Meta used to train its flagship text-generating model, Llama 3.1 405B.

Apple sourced additional data, including data from human feedback and synthetic data, to fine-tune the AFM models and attempt to mitigate any undesirable behaviors, like spouting toxicity.

"Our models have been created with the purpose of helping users do everyday activities across their Apple products, grounded

in Apple's core values, and rooted in our responsible AI principles at every stage," the company says.

There's no smoking gun or shocking insight in the paper -- and that's by careful design. Rarely are papers like these very revealing, owing to competitive pressures but also because disclosing too much could land companies in legal trouble.

Some companies training models by scraping public web data assert that their practice is protected by fair use doctrine. But it's a matter that's very much up for debate and the subject of a growing number of lawsuits.

Apple notes in the paper that it allows webmasters to block its crawler from scraping their data. But that leaves individual creators in a lurch. What's an artist to do if, for example, their portfolio is hosted on a site that refuses to block Apple's data scraping?

Courtroom battles will decide the fate of generative AI models and the way they're trained. For now, though, Apple's trying to position itself as an ethical player while avoiding unwanted legal scrutiny.

View original article

July 30, 2024 2:31:39AM

TechCrunch

Similar articles from other sources

Show(7)

Teaching computers to forget

Policymakers have been grappling with the rising complexity of Machine Learning (ML) models that churn huge swathes of data through Large Language Models (LLMs) and deep neural networks. The complexity has made it difficult for data fiduciaries to effectively "correct, complete, update and erase" sensitive data from computer systems. Simultaneously, we are witnessing an increase in AI (Artificial Intelligence) bias, misinformation, and breach of privacy, which gets heightened during events such as elections.

OPINION | Life in the times of AI

The antithesis of ML

In order to deal with this problem, a possible solution that has ignited interest among researchers and companies alike is the idea of Machine Unlearning (MUL). First mooted by Cao and Yang in 'Towards Making Systems Forget with Machine Unlearning', MUL ponders upon the question of how we can make machines forget data from trained AI models. It is the antithesis of ML. An algorithm is added to the AI model for the purpose of identifying and deleting false, incorrect, discriminatory, outdated, and sensitive information.

The concept builds on the challenge of removing information due to the constant churning of data by these LLMs. So much so that it gets difficult to keep track of the data as it can be utilised for multiple objectives, creating a complex web of algorithms, also known as data lineage, that adversely affect its quality, leading to manipulation, adversarial outputs, and difficulty in locating and removing sensitive information. Moreover, as there is no sandbox approach for choosing and processing data in these models, there is also a proven possibility of hackers inserting manipulated data to produce biased results (data poisoning).

One might argue for simply deleting the entire data set, i.e. data pruning, and re-training the entire AI model. However, it will lead to inflated computational costs and undue delays for the data fiduciaries while simultaneously carrying the risk of losing substantial accuracy. Consequently, MUL is gaining traction as a viable option among data fiduciaries such as IBM where the models are being tested for enhanced unlearning accuracy, intelligibility, reduced unlearning time and cost efficiency.

Three approaches

The question, however, remains how a MUL model can be implemented to effectively fulfil the obligation. There could be three approaches based on their viability for on-ground implementation: private, public, and international. In the private approach, data fiduciaries will be primarily responsible for testing MUL algorithms, which can then be applied across their training models for efficient deletion based on specific requirements. This voluntary approach gives companies much headroom to enhance their AI models and preserve users' rights without undue government intervention. However, the problem occurs in expertise and affordability to execute these models, which might discourage smaller companies from testing the solutions. This is the model currently being followed, albeit at a preliminary stage.

In the public approach, the government has the responsibility to prepare the statutory blueprint, either through soft-law or hard-law approaches, to obligate data fiduciaries to fulfil their legal obligations. This approach has to be read with the context of rising mentions of AI in legislative proceedings (from 1,247 in 2022 to 2,175 in 2023) across 49 jurisdictions. The data reflect a high possibility of government intervention in the near future if a major breakthrough in a MUL model parallels the rising regulatory landscape. The government can issue guidelines under the respective Data or AI Protection Regime mandating that data fiduciaries implement a plausible MUL model. For instance, the European Union's AI Act has adopted a soft-law approach by adding a provision to tackle data poisoning. It considers data poisoning as a form of cyber attack and directs data fiduciaries to put security controls "to ensure a level of cybersecurity appropriate to the risks."

On the contrary, the government can itself prepare a MUL model as part of its Digital Public Infrastructure for the perusal of data fiduciaries to implement across platforms uniformly. This is especially useful in developing countries where the state has substantive stakes in the DPI for the country's overall development. Moreover, it addresses the problem of affordability and expertise for smaller companies.

The international approach emphasises the role of nation states in coming together and preparing a framework to be adopted uniformly at a domestic level. The rationale flows from the idea that any innovation in AI has trans-boundary implications, and it is preferable to follow uniform standards across jurisdictions as a step ahead towards global governance of AI. As the efficacy of this approach is not clear amid geopolitical frictions, the onus effectively shifts to the role of international standard-setting organisations such as the International Electrotechnical Commission to come up with MUL standards that can be applied across jurisdictions.

These approaches represent a formal blueprint for one of the solutions that can be utilised to subdue the menace of Generative AI and preserve the user's right to be forgotten. The MUL is still in the preliminary stages. Therefore, stakeholders must address technical and regulatory considerations to ensure its effective implementation in this evolving landscape of AI.

Read Comments

View original article

July 29, 2024 9:30:18PM

The Hindu

Can AI Truly Self-Improve? New Findings Show Human Data Crucial for AI Advancement

AI models trained on synthetic data may experience rapid degradation in output quality

Artificial Intelligence (AI) systems, such as ChatGPT, have made remarkable progress, yet they still need human assistance for enhancement and accuracy. According to a new study published by Nature, AI systems remain heavily reliant on large datasets curated and labelled by humans to learn and generate their responses.

Despite significant advancements in large language models (LLMs), one major limitation is their inability to train on their content to improve intelligence. Training an AI involves feeding it vast amounts of data to help it understand context, language patterns, and various nuances before fine-tuning based on performance.

In Nature's groundbreaking study, researchers experimented by providing LLMs with AI-generated text. This approach led to models needing to remember the less frequently mentioned information in their datasets, causing their outputs to become more homogeneous and eventually nonsensical. "The message is, we have to be very careful about what ends up in our training data," says co-author Zakhar Shumaylov, an AI researcher at the University of Cambridge, UK. Otherwise, "things will always, provably, go wrong," he added.

The team used mathematical analysis to show that the problem of model collapse is likely to be universal, affecting all sizes of language models that use uncurated data, simple image generators, and other types of AI. "This is a concern when trying to make AI models that represent all groups fairly because low-probability events often relate to marginalised groups," says study co-author Ilia Shumailov, who worked on the project at the University of Oxford, UK.

Language models build associations between tokens -- words or word parts -- in large volumes of text, often scraped from the Internet. They generate text by predicting the statistically most probable next word based on these learned patterns. In the study, the researchers started by using an LLM to create Wikipedia-like entries, then trained new iterations of the model on text produced by its predecessor.

As AI-generated information -- synthetic data -- polluted the training set, the model's outputs became gibberish. For instance, the ninth iteration of the model completed a Wikipedia-style article about English church towers with a treatise on the many colours of jackrabbit tails. The team expected to see errors but were surprised by how quickly "things went wrong."

"Collapse happens because each model is necessarily sampled only from the data on which it is trained. This means that infrequent words in the original data are less likely to be reproduced, and the probability of common ones being regurgitated is boosted," explains Shumaylov. "Complete collapse eventually occurs because each model learns not from reality but from the previous model's prediction of reality, with errors amplified in each iteration."

The study also highlights a more significant issue. As synthetic data builds up on the web, the scaling laws suggesting that models should improve with more data may break down because training data will lose the richness and variety of human-generated content. Hany Farid, a computer scientist at the University of California, Berkeley, compares this problem to "inbreeding within a species." Farid states, "If a species inbreeds with its own offspring and doesn't diversify its gene pool, it can lead to a collapse of the species."

Farid's work has demonstrated similar effects in image models, producing eerie distortions of reality. The solution? Developers might need to find ways, such as watermarking, to keep AI-generated data separate from accurate data, which would require unprecedented coordination by big-tech firms, suggests Shumailov.

Even when Shumailov and his team fine-tuned each model on 10 per cent real data alongside synthetic data, the collapse occurred more slowly. Society might need to find incentives for human creators to continue producing content, and filtering may become essential. For example, humans could curate AI-generated text before it goes back into the data pool.

View original article

July 29, 2024 7:41:35PM

International Business Times UK

This AI Paper from Stanford Provides New Insights on AI Model Collapse and Data Accumulation

Large-scale generative models like GPT-4, DALL-E, and Stable Diffusion have transformed artificial intelligence, demonstrating remarkable capabilities in generating text, images, and other media. However, as these models become more prevalent, a critical challenge emerges the consequences of training generative models on datasets containing their outputs. This issue, known as model collapse, poses a significant threat to the future development of AI. As generative models are trained on web-scale datasets that increasingly include AI-generated content, researchers are struggling with the potential degradation of model performance over successive iterations, potentially rendering newer models ineffective and compromising the quality of training data for future AI systems.

Existing researchers have investigated model collapse through various methods, including replacing real data with generated data, augmenting fixed datasets, and mixing real and synthetic data. Most studies maintained constant dataset sizes and mixing proportions. Theoretical work has focused on understanding model behavior with synthetic data integration, analyzing high-dimensional regression, self-distillation effects, and language model output tails. Some researchers identified phase transitions in error scaling laws and proposed mitigation strategies. However, these studies primarily considered fixed training data amounts per iteration. Few explored the effects of accumulating data over time, closely resembling evolving internet-based datasets. This research gap highlights the need for further investigation into the long-term consequences of training models on continuously expanding datasets that include both real and synthetic data, reflecting the dynamic nature of web-scale information.

Researchers from Stanford University propose a study that explores the impact of accumulating data on model collapse in generative AI models. Unlike previous research focusing on data replacement, this approach simulates the continuous accumulation of synthetic data in internet-based datasets. Experiments with transformers, diffusion models, and variational autoencoders across various data types reveal that accumulating synthetic data with real data prevents model collapse, in contrast to the performance degradation observed when replacing data. The researchers extend existing analysis of sequential linear models to prove that data accumulation results in a finite, well-controlled upper bound on test error, independent of model-fitting iterations. This finding contrasts with the linear error increase seen in data replacement scenarios.

Researchers experimentally investigated model collapse in generative AI using causal transformers, diffusion models, and variational autoencoders across text, molecular, and image datasets.

To test the model collapse in transformer-based language models researchers used GPT-2 and Llama2 architectures of various sizes, pre-trained on TinyStories. They compared data replacement and accumulation strategies over multiple iterations. Results consistently showed that replacing data increased test cross-entropy (worse performance) across all model configurations and sampling temperatures. In contrast, accumulating data maintained or improved performance over iterations. Lower sampling temperatures accelerated error increases when replacing data, but the overall trend remained consistent. These findings strongly support the hypothesis that data accumulation prevents model collapse in language modeling tasks, while data replacement leads to progressive performance degradation.

Researchers tested GeoDiff diffusion models on GEOM-Drugs molecular conformation data, comparing data replacement and accumulation strategies. Results showed increasing test loss when replacing data, but stable performance when accumulating data. Unlike language models, significant degradation occurred mainly in the first iteration with synthetic data. These findings further support data accumulation as a method to prevent model collapse across different AI domains.

Researchers used VAEs on CelebA face images, comparing data replacement and accumulation strategies. Replacing data led to rapid model collapse, with increasing test error and decreasing image quality and diversity. Accumulating data significantly slowed collapse, preserving major variations but losing minor details over iterations. Unlike language models, accumulation showed slight performance degradation. These findings support data accumulation's benefits in mitigating model collapse across AI domains while highlighting variations in effectiveness depending on model type and dataset.

This research investigates model collapse in AI, a concern as AI-generated content increasingly appears in training datasets. While previous studies showed that training on model outputs can degrade performance, this work demonstrates that model collapse can be prevented by training on a mixture of real and synthetic data. The findings, supported by experiments across various AI domains and theoretical analysis for linear regression, suggest that the "curse of recursion" may be less severe than previously thought, as long as synthetic data is accumulated alongside real data rather than replacing it entirely.

View original article

July 29, 2024 5:20:19PM

MarkTechPost

Beware of AI 'model collapse': How training on synthetic data pollutes the next generation

To feed the endless appetite of generative artificial intelligence (gen AI) for data, researchers have in recent years increasingly tried to create "synthetic" data, which is similar to the human-created works that have been used to train AI models but is itself created by AI.

The synthetic data movement is a vibrant one because of copyright infringement issues with human-based training data, and also because the requirements of training better and better models may eventually exceed the availability of human-generated data.

Also: 3 ways Meta's Llama 3.1 is an advance for Gen AI

For example, in Meta's flagship open-source model, Llama 3.1 405B, which the company introduced last week, the researchers made extensive use of synthetic data to "fine-tune" the model and to supplement the human feedback they gathered.

There's a catch, though. Oxford University scholars warn in the most recent issue of the prestigious science journal Nature that using such synthetic data to train gen AI can drastically degrade the accuracy of the models, to the point of making them useless.

In the paper, lead author Ilia Shumailov and his team describe what they call "model collapse," and how it becomes worse each time models feed the next model with fake data.

Also: Google's DeepMind AI takes home silver medal in complex math competition

"Model collapse is a degenerative process affecting generations of learned generative models, in which the data they generate end up polluting the training set of the next generation," Shumailov's team wrote. "Being trained on polluted data, they then mis-perceive reality."

Specifically, the models lose track of the less-common facts over generations, becoming more and more generic. As they do so, the answers they produce become totally irrelevant to the questions they ask, turning into effectively gibberish. "Models start forgetting improbable events over time, as the model becomes poisoned with its own projection of reality," they write.

The authors wrote that the findings "must be taken seriously," as gen AI risks a compounding process of deterioration the more that the internet is flooded with the output of AI models that then gets re-used. "The use of LLMs at scale to publish content on the internet will pollute the collection of data to train their successors: data about human interactions with LLMs will be increasingly valuable," they wrote.

Also: OpenAI offers GPT-4o mini to slash the cost of applications

To arrive at that conclusion, the authors conducted an experiment using Meta's open-source AI model, OPT, for "open pre-trained transformer," introduced in 2022. It is similar in structure to OpenAI's GPT-3, but much smaller, with only 125 million neural parameters, or "weights."

Shumailov's team used the Wikitext2 dataset of Wikipedia articles to "fine-tune" OPT, meaning, to re-train it with additional data, a very common practice in gen AI. The authors then used the fine-tuned OPT to in turn generate synthetic copies of the Wikitext data, and they fed that new, fake data to the next fine-tuning operation, a kind of cannibalistic use of the output of one model as the input of another.

The authors provided examples of what happens after five rounds of using each fine-tuned model as the source for teaching the next: by generation five, it's complete gibberish. At the same time, they wrote, specific errors of fact became more common with each generation: "We find that, over the generations, models [...] start introducing their own improbable sequences, that is, errors."

Reflecting on what can be done to avoid model collapse, the authors ended their paper on an ominous note. It's essential to preserve the original, human-created training data, and to also have continued access to new human-created data, but doing so becomes harder as synthetic data from gen AI fills up more and more of the internet, creating a kind of lost internet of the past.

They warned, "It may become increasingly difficult to train newer versions of LLMs without access to data that were crawled from the internet before the mass adoption of the technology or direct access to data generated by humans at scale."

The editors of the magazine summed up the problem perhaps most succinctly with the old data science adage they placed on the cover: "garbage in, garbage out."

View original article

July 29, 2024 4:12:34PM

ZDNet

Has the AI boom plateaued?

First, we are quickly running out of data required to train newer models. The voracious appetite of existing models has already vacuumed up most of the publicly available text on the open internet and it is estimated that the entire lot of high-quality textual data online will be used up by 2028, creating a so-called "data wall", that might be difficult to breach. While there are still abundant sources of video, audio and images that can be tapped into, these are far more difficult to use for training than text and are subject to greater intellectual property protections.

Second, the quality of the public data that remains available is a matter of serious concern with much of the high-quality data already accounted for, leaving only subpar sources which necessitates spending more time on "cleaning" available data to make it fit for consumption. Good quality data is an absolute necessity for training quality LLMs. Finding new, untapped sources of such data will be a difficult task, forcing developers and companies to look at improving the quality of existing data sets and drawing more utility from them.

A sourcing issue

Third, any further expansion of data for LLMs would need to come from two sources. Either proprietary data protected by intellectual property, or from "synthetic data" generated by AI systems themselves. Access to proprietary data can legally only be acquired on a case-by-case basis, subject to specific agreements with the data owners, which reduces the speed of said access. While there has already been some movement on this front with institutions like The New York Times signing data access agreements with OpenAI for their content, how much this model can be replicated globally is up for debate. Synthetic data, on the other hand, could provide a more ready alternative to publicly available online sources, by training AI models on data produced by other AI systems, thereby bypassing the "data wall" problem.

The growing interest in the use of synthetic data and its theoretical possibilities has led firms like Nvidia and IBM to launch their own synthetic data generators. There is, however, a great deal of scepticism about the efficacy of such data compared to human-generated sources. A recent study in Nature magazine suggests that models trained on synthetic data are more likely to "hallucinate" or produce nonsensical outputs because it amplifies mistakes made in previous generations of the trainings, leading to AI models "collapsing" on themselves. AI offerings trained on synthetic data are therefore likely to have less commercial uptake when compared to models that are trained on human-generated data.

Fourth, an honest assessment would show that despite the hype over the last year and a half, AI adoption has been relatively slow and narrow, creating doubts about the short-term commercial viability of prevalent models. The most recent quarterly financial report from Alphabet for example shows that the cost of training and deploying their respective models has far outweighed any immediate commercial returns, leading to a hammering of its share price after 18 months of giddy ascent. Odds are, other AI-focused Big Tech firms like Meta and Microsoft are likely to show similar results. In essence, the majority of the world still does not know what to do with existing AI models, forget about trying any new ones.

Still a niche

Beyond a small niche, AI as a product has not really taken off as expected. Big Tech's focus now is on creating more use-cases for existing AI, either by integrating it at the backend within existing products the way Apple is, or cannibalising existing rivals' businesses like OpenAI is attempting to do with SearchGPT.

The task, for now

The primary question the AI and tech community faces at this point is not technological but commercial -- does good technology translate into good business? After all, developing a technology and commercialising it are two very different problems. Will AI change the world? Undoubtedly, but this will take time and will be subject to the very real problems of capital, regulations, and market forces. There is also significant work that remains to be done on the physical aspects of the AI ecosystem, such as designing more efficient and environmentally positive data centres, creating more robust chip supply lines, and building new power plants, amongst a host of other issues.

Therefore, until significant AI-attributable revenues start coming in, AI and tech companies will be forced to focus on the incremental tinkering needed to make existing models better, sustainable, and importantly, more commercially viable. This phase of consolidation is not only good but is also required to set the necessary groundwork for the next generation of frontier models to be more easily accepted globally.

View original article

July 29, 2024 12:38:09PM

The Indian Express

The data that powers AI is disappearing fast

SAN FRANCISCO -- For years, the people building powerful artificial intelligence systems have used enormous troves of text, images and videos pulled from the internet to train their models.

Now that data is drying up.

Over the past year, many of the most important web sources used for training AI models have restricted the use of their data, according to a study published last week by the Data Provenance Initiative, a Massachusetts Institute of Technology-led research group.

The study, which looked at 14,000 web domains that are included in three commonly used AI training data sets, discovered an "emerging crisis in consent," as publishers and online platforms have taken steps to prevent their data from being harvested.

The researchers estimate that in the three data sets -- called C4, RefinedWeb and Dolma -- 5% of all data, and 25% of data from the highest-quality sources, has been restricted. Those restrictions are set up through the Robots Exclusion Protocol, a decades-old method for website owners to prevent automated bots from crawling their pages using a file called http://robots.txt.

The study also found that as much as 45% of the data in one set, C4, had been restricted by websites' terms of service.

"We're seeing a rapid decline in consent to use data across the web that will have ramifications not just for AI companies, but for researchers, academics and noncommercial entities," Shayne Longpre, the study's lead author, said in an interview.

Data is the main ingredient in today's generative AI systems, which are fed billions of examples of text, images and videos. Much of that data is scraped from public websites by researchers and compiled in large data sets, which can be downloaded and freely used, or supplemented with data from other sources.

Learning from that data is what allows generative AI tools like OpenAI's ChatGPT, Google's Gemini and Anthropic's Claude to write, code and generate images and videos. The more high-quality data is fed into these models, the better their outputs generally are.

For years, AI developers were able to gather data fairly easily. But the generative AI boom of the past few years has led to tensions with the owners of that data -- many of whom have misgivings about being used as AI training fodder or at least want to be paid for it.

As the backlash has grown, some publishers have set up paywalls or changed their terms of service to limit the use of their data for AI training. Others have blocked the automated web crawlers used by companies like OpenAI, Anthropic and Google.

Sites like Reddit and StackOverflow have begun charging AI companies for access to data, and a few publishers have taken legal action -- including The New York Times, which sued OpenAI and Microsoft for copyright infringement last year, alleging that the companies used news articles to train their models without permission.

Companies like OpenAI, Google and Meta have gone to extreme lengths in recent years to gather more data to improve their systems, including transcribing YouTube videos and bending their own data policies.

More recently, some AI companies have struck deals with publishers including The Associated Press and News Corp., the owner of The Wall Street Journal, giving them ongoing access to their content.

But widespread data restrictions may pose a threat to AI companies, which need a steady supply of high-quality data to keep their models fresh and up to date.

They could also spell trouble for smaller AI outfits and academic researchers who rely on public data sets and can't afford to license data directly from publishers. Common Crawl, one such data set that comprises billions of pages of web content and is maintained by a nonprofit, has been cited in more than 10,000 academic studies, Longpre said.

It's not clear which popular AI products have been trained on these sources, since few developers disclose the full list of data they use. But data sets derived from Common Crawl, including C4 (which stands for Colossal, Cleaned Crawled Corpus) have been used by companies including Google and OpenAI to train previous versions of their models. Spokespeople for Google and OpenAI declined to comment.

Yacine Jernite, a machine-learning researcher at Hugging Face, a company that provides tools and data to AI developers, characterized the consent crisis as a natural response to the AI industry's aggressive data-gathering practices.

"Unsurprisingly, we're seeing blowback from data creators after the text, images and videos they've shared online are used to develop commercial systems that sometimes directly threaten their livelihoods," he said.

But he cautioned that if all AI training data needed to be obtained through licensing deals, it would exclude "researchers and civil society from participating in the governance of the technology."

Stella Biderman, the executive director of EleutherAI, a nonprofit AI research organization, echoed those fears.

"Major tech companies already have all of the data," she said. "Changing the license on the data doesn't retroactively revoke that permission, and the primary impact is on later-arriving actors, who are typically either smaller startups or researchers."

AI companies have claimed that their use of public web data is legally protected under fair use. But gathering new data has gotten trickier. Some AI executives I've spoken to worry about hitting the "data wall" -- their term for the point at which all of the training data on the public internet has been exhausted, and the rest has been hidden behind paywalls, blocked by http://robots.txt or locked up in exclusive deals.

Some companies believe they can scale the data wall by using synthetic data -- that is, data that is itself generated by AI systems -- to train their models. But many researchers doubt that today's AI systems are capable of generating enough high-quality synthetic data to replace the human-created data they're losing.

Another challenge is that while publishers can try to stop AI companies from scraping their data by placing restrictions in their http://robots.txt files, those requests aren't legally binding, and compliance is voluntary. (Think of it like a "no trespassing" sign for data, but one without the force of law.)

Major search engines honor these opt-out requests, and several leading AI companies, including OpenAI and Anthropic, have said publicly that they do, too. But other companies, including the AI-powered search engine Perplexity, have been accused of ignoring them. Perplexity's CEO, Aravind Srinivas, said that the company respects publishers' data restrictions. He added that while the company once worked with third-party web crawlers that did not always follow the Robots Exclusion Protocol, it had "made adjustments with our providers to ensure that they follow http://robots.txt when crawling on Perplexity's behalf."

Longpre said that one of the big takeaways from the study is that we need new tools to give website owners more precise ways to control the use of their data. Some sites might object to AI giants using their data to train chatbots for a profit but might be willing to let a nonprofit or educational institution use the same data, he said. Right now, there's no good way for them to distinguish between those uses, or block one while allowing the other.

But there's also a lesson here for big AI companies, who have treated the internet as an all-you-can-eat data buffet for years, without giving the owners of that data much of value in return. Eventually, if you take advantage of the web, the web will start shutting its doors.

View original article

July 29, 2024 12:00:02PM

The Seattle Times

The Data That Powers A.I. Is Disappearing Fast

For years, the people building powerful artificial intelligence systems have used enormous troves of text, images and videos pulled from the internet to train their models.

Now, that data is drying up.

Over the past year, many of the most important web sources used for training A.I. models have restricted the use of their data, according to a study published this week by the Data Provenance Initiative, an M.I.T.-led research group.

The study, which looked at 14,000 web domains that are included in three commonly used A.I. training data sets, discovered an "emerging crisis in consent," as publishers and online platforms have taken steps to prevent their data from being harvested.

The researchers estimate that in the three data sets -- called C4, RefinedWeb and Dolma -- 5 percent of all data, and 25 percent of data from the highest-quality sources, has been restricted. Those restrictions are set up through the Robots Exclusion Protocol, a decades-old method for website owners to prevent automated bots from crawling their pages using a file called http://robots.txt.

The study also found that as much as 45 percent of the data in one set, C4, had been restricted by websites' terms of service.

"We're seeing a rapid decline in consent to use data across the web that will have ramifications not just for A.I. companies, but for researchers, academics and noncommercial entities," said Shayne Longpre, the study's lead author, in an interview.

View original article

July 26, 2024 8:44:03PM

The New York Times

2 months ago

Luka

TechnologyIntranetStandards and ProtocolsEnterprise Information Integration

AVA AI: Aboitiz Data Innovation's new GenAI platform for enterprises - Back End News

Aboitiz Data Innovation Launches AVA AI: Leading the Charge in Enterprise-Grade GenAI Innovation

July 19, 2024 9:05:52AM

mindanaoexaminernewspaper.blogspot.com

MANILA - Aboitiz Data Innovation (ADI), the Data Science and Artificial Intelligence (DSAI) arm of the Aboitiz Group, recently announced the launch of AVA AI, its Generative AI (GenAI) Platform designed to empower enterprises across the public and private sectors. Built with enhancing productivity in mind, AVA AI is able to seamlessly integrate into existing systems and handle complex workflows, allowing teams to gain access to constructive insights from their organisations' collective data securely and efficiently.

View original article

July 19, 2024 9:05:52AM

mindanaoexaminernewspaper.blogspot.com

Google Cloud Enhances Vertex AI Grounding Capabilities For Reliable AI Responses

Google Cloud enhances Vertex AI grounding capabilities for reliable AI responses - ExBulletin

June 29, 2024 8:13:08AM

ExBulletin

Google Cloud has expanded Vertex AI's grounding capabilities, significantly enhancing the platform's ability to generate more accurate and reliable AI responses. These advancements are intended to mitigate AI hallucinations and improve the overall quality of generative AI applications and agents.

One key addition is the introduction of dynamic search for grounding with Google Search, now generally available. This innovative feature enables Gemini, Google's advanced large-scale language model, to intelligently decide whether to ground a user query with Google Search or rely on its own knowledge. This approach helps balance response quality and cost-effectiveness, as grounding with Google Search incurs additional processing costs. Gemini makes this decision by understanding whether the requested information is static, slowly changing, or rapidly evolving.

For example, if you ask about the latest movies, Gemini will use Google search to get the latest information. Conversely, for general questions like "What is the capital of France?", it will provide answers from its existing knowledge base without any external evidence. This dynamic approach not only improves the accuracy of responses, but also optimizes resource usage.

Google Cloud is also introducing an "high fidelity" grounding mode that is targeted at industries such as healthcare and financial services where accuracy and reliability are paramount.

Additionally, Google will soon enable grounded models with third-party datasets, expected to be released in Q3. Working with specialist data providers such as Moodys, MSCI, Thomson Reuters and Zoominfo, Google will provide access to their datasets via Vertex AI. This capability will enable companies to integrate highly specific and reliable information into their AI models, further increasing the accuracy and relevance of the responses generated.

For enterprises looking to incorporate AI models on private data, Google Cloud offers a suite of APIs for Vertex AI Search and Retrieval Augmented Generation (RAG). These tools help enterprises create custom RAG workflows, build semantic search engines, and enhance existing search capabilities. The APIs, now generally available, provide capabilities for document parsing, embedding generation, semantic ranking, grounded answer generation, and a fact-checking service called Check Grounding.

These enhancements are part of Google Cloud's broader strategy to make generative AI more trustworthy and suitable for enterprise use. By connecting AI models to diverse, trusted sources of information like web data, enterprise documents, operational databases, and enterprise applications, Google aims to root AI in what it calls "enterprise truth."

The focus on grounding heightens industry concerns about AI hallucinations: as AI models become more complex, the risk of them producing erroneous or unreliable outputs increases. Grounding techniques such as RAG mitigate this risk by feeding models facts from external knowledge sources, improving the accuracy and reliability of their responses.

By enabling businesses to leverage both public and private data sources, Google is enabling the development of more robust and reliable AI applications across industries.

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

View original article

June 29, 2024 8:13:08AM

ExBulletin

Enterprise Information Integration (2024)

Web Design and DevelopmentStandards and ProtocolsEnterprise Information IntegrationTechnology

Oxylabs introduces the first AI copilot for public web data collection

Oxylabs Launches OxyCopilot to Revolutionize Web Data Collection

Similar articles from other sources

Oxylabs introduces the first AI copilot for public web data collection

Oxylabs introduces the first AI copilot for public web data collection

Oxylabs introduces the first AI copilot for public web data collection

Oxylabs introduces AI copilot for web data collection - Global Security Mag Online

DatabasesEnterprise Information IntegrationData AdministrationTechnology

The Tie Launches Polkadot Ecosystem Dashboard on The Tie Terminal - Decrypt

Bill Laboon, Director of Education at Web3 Foundation, on Polkadot's 1000 Referendums, Centralized Points in Decentralized Polkadot, and Sprinkling Blockchain Dust | Ep. 369

Similar articles from other sources

Weekly Digest: Polkadot sees growth in funding, analytics and engagement By Investing.com

Weekly Digest: Polkadot sees growth in funding, analytics and engagement By Investing.com

Kampela Secures Polkadot Network Investment, Becomes First Fully DAO-Funded Hardware Wallet

Kampela Secures Polkadot Network Investment, Becomes First Fully DAO-Funded Hardware Wallet - Decrypt

The Tie Launches Polkadot Ecosystem Dashboard on The Tie Terminal

The Tie Launches Polkadot Ecosystem Dashboard on The Tie Terminal

Kampela Secures Polkadot Network Investment, Becomes First Fully DAO-Funded Hardware Wallet - Blockonomi

The Tie Launches Polkadot Ecosystem Dashboard on The Tie Terminal - Decrypt

Dune Becomes the Most Comprehensive Onchain Data Hub for Polkadot's 50+ Parachains

Dune Analytics builds a full data hub on Polkadot's 50 parachains

Dune Becomes the Most Comprehensive Onchain Data Hub for Polkadot's 50+ Parachains

Dune expanding onchain analytics to cover 50+ Polkadot parachains

Dune expands onchain data coverage to all Polkadot parachains By Investing.com

Human ResourcesWorkflowEnterprise Information IntegrationTechnology

Customers Unify Critical Data and Processes with New Oracle Primavera Unifier Capabilities

Customers Unify Critical Data and Processes with New Oracle Primavera Unifier Capabilities - CRN - India

Similar articles from other sources

Customers unify critical data and processes with new Oracle Primavera Unifier capabilities - CRN - India

Customers unify critical data and processes with New Oracle Primavera unifier capabilities - Express Computer

Syniverse to Offer Personalized Messaging Services to Oracle Cloud Customers

Customers Unify Critical Data and Processes with New Oracle Primavera Unifier Capabilities

Customers Unify Critical Data and Processes with New Oracle Primavera Unifier Capabilities By Investing.com

Customers Unify Critical Data and Processes with New Oracle Primavera Unifier Capabilities

Odyssey House implements Oracle Health to help enhance patient care, improve revenue management - Utah Business

Master Data ManagementData CentersEnterprise Information IntegrationTechnology

The Data That Powers A.I. Is Disappearing Fast

Apple says it took a 'responsible' approach to training its Apple Intelligence models

Similar articles from other sources

Teaching computers to forget

Can AI Truly Self-Improve? New Findings Show Human Data Crucial for AI Advancement

This AI Paper from Stanford Provides New Insights on AI Model Collapse and Data Accumulation

Beware of AI 'model collapse': How training on synthetic data pollutes the next generation

Has the AI boom plateaued?

The data that powers AI is disappearing fast

The Data That Powers A.I. Is Disappearing Fast

TechnologyIntranetStandards and ProtocolsEnterprise Information Integration

AVA AI: Aboitiz Data Innovation's new GenAI platform for enterprises - Back End News

Aboitiz Data Innovation Launches AVA AI: Leading the Charge in Enterprise-Grade GenAI Innovation

Similar articles from other sources

AVA AI: Aboitiz Data Innovation's new GenAI platform for enterprises - Back End News

Aboitiz Data Innovation launches AVA AI, leading the charge in enterprise-grade GenAI innovation

Aboitiz Data Innovation launches AVA AI

Aboitiz Data Innovation launches AVA AI to lead in enterprise-grade GenAI innovation

SearchingInformation RetrievalEnterprise Information IntegrationTechnology

Google Cloud Enhances Vertex AI Grounding Capabilities For Reliable AI Responses

Google Cloud enhances Vertex AI grounding capabilities for reliable AI responses - ExBulletin

Similar articles from other sources

Google Cloud Enhances Vertex AI Grounding Capabilities For Reliable AI Responses

Google, Moody's and Thomson Reuters team up to provide real-world data for AI - ExBulletin

Google partners with Moody's to counter its AI's hallucination problems

Google Cloud's Vertex AI gets new grounding options

Google Cloud says enterprise AI chatbots will be more "grounded" in real-world facts

Google to partner with Moody's, Thomson Reuters to give AI more real world data

Google partners with Thomson Reuters, Moody's and more to give AI real-world data

Google partners with Thomson Reuters, Moody's and more to give AI real-world data - RocketNews