Significance of High-Quality Training Data

Here are a few simple truths about Data Quality: 1. Data without quality isn't trustworthy 2. Data that isn't trustworthy, isn't useful 3. Data that isn't useful, is low ROI Investing in AI while the underlying data is low ROI will never yield high-value outcomes. Businesses must put an equal amount of time and effort into the quality of data as the development of the models themselves. Many people see data debt as another form of technical debt - it's worth it to move fast and break things after all. This couldn't be more wrong. Data debt is orders of magnitude WORSE than tech debt. Tech debt results in scalability issues, though the core function of the application is preserved. Data debt results in trust issues, when the underlying data no longer means what its users believe it means. Tech debt is a wall, but data debt is an infection. Once distrust drips in your data lake, everything it touches will be poisoned. The poison will work slowly at first and data teams might be able to manually keep up with hotfixes and filters layered on top of hastily written SQL. But over time, the spread of the poison will be so great and deep that it will be nearly impossible to trust any dataset at all. A single low-quality data set is enough to corrupt thousands of data models and tables downstream. The impact is exponential. My advice? Don't treat Data Quality as a nice to have, or something that you can afford to 'get around to' later. By the time you start thinking about governance, ownership, and scale it will already be too late and there won't be much you can do besides burning the system down and starting over. What seems manageable now becomes a disaster later on. The earliest you can get a handle on data quality, you should. If you even have a guess that the business may want to use the data for AI (or some other operational purpose) then you should begin thinking about the following: 1. What will the data be used for? 2. What are all the sources for the dataset? 3. Which sources can we control versus which can we not? 4. What are the expectations of the data? 5. How sure are we that those expectations will remain the same? 6. Who should be the owner of the data? 7. What does the data mean semantically? 8. If something about the data changes, how is that handled? 9. How do we preserve the history of changes to the data? 10. How do we revert to a previous version of the data/metadata? If you can affirmatively answer all 10 of those questions, you have a solid foundation of data quality for any dataset and a playbook for managing scale as the use case or intermediary data changes over time. Good luck! #dataengineering

Sanjeev Mohan dives into why the success of AI in enterprise applications hinges on the quality of data and the robustness of data modeling. Accuracy Matters: Accurate, clean data ensures AI algorithms make correct predictions and decisions. Consistency is Key: Consistent data formats allow for smoother integration and processing, enhancing AI efficiency. Timeliness: Current, up-to-date data keeps AI-driven insights relevant, supporting timely business decisions. Just as a building needs a blueprint, AI systems require robust data models to guide their learning and output. Data modeling is crucial because: Structures Data for Understanding: It organizes data in a way that machines can interpret and learn from efficiently. Tailors AI to Business Needs: Customized data models align AI outputs with specific enterprise objectives. Enables Scalability: Well-designed models adapt to increasing data volumes and evolving business requirements. As businesses continue to invest in AI, integrating high standards for data quality and strategic data modeling is non-negotiable.

Here’s Why Data is the Lock and Key to AI's Future 🗝 The AI landscape is humming with innovation, yet one thing is abundantly clear: your AI is only as good as the data that feeds it. A Lamborghini without fuel is, after all, just an expensive piece of sculpture. 📊 Why Data Matters in AI Data and processing power are the twin engines driving AI. But as we face a shortage of specialized AI chips, companies are doubling down on sourcing quality data to win in AI. Epoch ai, a research firm, estimates that high-quality text for AI training could be exhausted by 2026. That's not far off. To put this in perspective, the latest AI models are trained on over 1 trillion words, dwarfing the 4 billion English words on Wikipedia! 🎯 Quality Over Quantity But it's not just about having the most data; it's about having the right data. Models perform significantly better when trained on high-quality, specialized datasets. So while AI models are gobbling up data like Pac-Man, there's a clear hierarchy on the menu. Long-form, factually accurate, and well-written content is the gourmet meal for these systems. Specialized data allows for fine-tuning, making AI models more effective for niche applications. 🚧 Challenges Ahead With demand for data scaling up, copyright battles are flaring up and companies that own vast data troves are becoming gatekeepers, dictating terms and raising the costs for access. For example, Adobe, which owns a treasure-trove of stock images, has an advantage in image creation AI. The lay of the land is changing, and fast. 🔄 The Data Flywheel Effect Companies are improving data quality through user interactions. Feedback mechanisms are increasingly built into AI tools, creating a “data flywheel” effect. As users give thumbs-up or thumbs-down, that information becomes a new layer of data, enriching the AI model's understanding and performance. 🔒 Unlocking Corporate Data Beyond public datasets, a goldmine lies within corporate walls. Think customer spending records, call-center transcripts, and more. However, this data is often unstructured and fragmented across systems. Businesses now have the opportunity, and frankly the imperative, to organize these data silos. Not only would this amplify their own AI capabilities but also add a crucial source to the broader data ecosystem. 🛠 The Road Ahead The narrative is clear: for AI to reach its fullest potential, data sourcing, quality, and management can't be afterthoughts; they are central to the plot. As AI continues to stretch its capabilities, the race for data isn't slowing down. It's not just about finding the data; it's about cultivating it, refining it, and recognizing its true value in the grand scheme of AI development. #AI #DataQuality #Innovation #DataManagement #AIandData

10 of the most-cited datasets contain a substantial number of errors. And yes, that includes datasets like ImageNet, MNIST, CIFAR-10, and QuickDraw which have become the definitive test sets for computer vision models. Some context: A few years ago, 3 MIT graduate students published a study that found that ImageNet had a 5.8% error rate in its labels. QuickDraw had an even higher error rate: 10.1%. Why should we care? 1. We have an inflated sense of the performance of AI models that are testing against these datasets. Even if models achieve high performance on those test sets, there’s a limit to how much those test sets reflect what really matters: performance in real-world situations. 2. AI models trained using these datasets are starting off on the wrong foot. Models are only as good as the data they learn from, and if they’re consistently trained on incorrectly labeled information, then systematic errors can be introduced. 3. Through a combination of 1 and 2, trust in these AI models is vulnerable to being eroded. Stakeholders expect AI systems to perform accurately and dependably. But when the underlying data is flawed and these expectations aren’t met, we start to see a growing mistrust in AI. So, what can we learn from this? If 10 of the most cited datasets contain so many errors, we should assume the same of our own data unless proven otherwise. We need to get serious about fixing — and building trust in — our data, starting with improving our data hygiene. That might mean implementing rigorous validation protocols, standardizing data collection procedures, continuously monitoring for data integrity, or a combination of tactics (depending on your organization’s needs). But if we get it right, we're not just improving our data; we're setting our future AI models to be dependable and accurate. #dataengineering #dataquality #datahygiene #generativeai #ai

This is a great article to guide companies at the early stages of implementing Gen AI solutions. With Gen AI on the horizon, the spotlight isn't just on innovation—it's on our data. An overwhelming 80% of data leaders recognize its transformative potential, yet a stark disconnect lies in the readiness of our data environments. Only a minuscule 6% have operational Gen AI applications. The call to action is evident: for Gen AI to redefine our future, the foundation starts with high-quality, meticulously curated data. Organizations must create a data environment that supports and enhances the capabilities of Gen AI, turning it into a critical asset for driving innovation and business growth. Laying a solid data foundation for unlocking the full potential of Gen AI involves a well-thought-out approach: 1—Assess Data Quality: Begin by thoroughly assessing current data quality. Identify gaps in accuracy, completeness, and timeliness. 2 - Data Integration and Management: Integrate disparate data sources to create a unified view. Employ robust data management practices to ensure data consistency and accessibility. 3 - Curate and Annotate Data: Ensure relevance and annotate it to enhance usability for Gen AI models. 4 - Implement Data Governance: Establish a robust data governance framework to maintain data integrity, security, and compliance to foster data sharing and collaboration. 5 - Invest in Scalable Infrastructure: Build or upgrade to a data infrastructure that can scale future Gen AI applications. This includes cloud storage, powerful computing resources, and advanced data processing capabilities. 6—Upskill Your Team: Ensure the technical team has the necessary skills to manage, analyze, and leverage data to build Gen AI solutions. 7 Pilot and Scale: To test and refine your approach, start with pilot projects. Use these learnings to scale successful initiatives across the organization. 8 - Continuous Improvement: Gen AI and data landscapes are evolving rapidly. Establish processes for ongoing data evaluation and model training to adapt to new developments and insights.

Google announces Scaling (Down) CLIP A Comprehensive Analysis of Data, Architecture, and Training Strategies This paper investigates the performance of the Contrastive Language-Image Pre-training (CLIP) when scaled down to limited computation budgets. We explore CLIP along three dimensions: data, architecture, and training strategies. With regards to data, we demonstrate the significance of high-quality training data and show that a smaller dataset of high-quality data can outperform a larger dataset with lower quality. We also examine how model performance varies with different dataset sizes, suggesting that smaller ViT models are better suited for smaller datasets, while larger models perform better on larger datasets with fixed compute. Additionally, we provide guidance on when to choose a CNN-based architecture or a ViT-based architecture for CLIP training. We compare four CLIP training strategies - SLIP, FLIP, CLIP, and CLIP+Data Augmentation - and show that the choice of training strategy depends on the available compute resource. Our analysis reveals that CLIP+Data Augmentation can achieve comparable performance to CLIP using only half of the training data. This work provides practical insights into how to effectively train and deploy CLIP models, making them more accessible and affordable for practical use in various applications.

The saying "more data beats clever algorithms" is not always so. In new research from Amazon, we show that using AI can turn this apparent truism on its head. Anomaly detection and localization is a crucial technology in identifying and pinpointing irregularities within datasets or images, serving as a cornerstone for ensuring quality and safety in various sectors, including manufacturing and healthcare. Finding them quickly, reliably, at scale matters, so automation is key. The challenge is that anomalies - by definition! - are usually rare and hard to detect - making it hard to gather enough data to train a model to find them automatically. Using AI, Amazon has developed a new method to significantly enhance anomaly detection and localization in images, which not only addresses the challenges of data scarcity and diversity but also sets a new benchmark in utilizing generative AI for augmenting datasets. Here's how it works... 1️⃣ Data Collection: The process starts by gathering existing images of products to serve as a base for learning. 2️⃣ Image Generation: Using diffusion models, the AI creates new images that include potential defects or variations not present in the original dataset. 3️⃣ Training: The AI is trained on both the original and generated images, learning to identify what constitutes a "normal" versus an anomalous one. 4️⃣ Anomaly Detection: Once trained, the AI can analyze new images, detecting and localizing anomalies with enhanced accuracy, thanks to the diverse examples it learned from. The results are encouraging, and show that 'big' quantities of data can be less important than high quality, diverse data when building autonomous systems. Nice work from the Amazon science team. The full paper is linked below. #genai #ai #amazon

“Garbage in, garbage out” is the reason that a lot of AI-generated text reads like boring, SEO-spam marketing copy. 😴😴😴 If you’re training your organization's self-hosted AI model, it’s probably because you want better, more reliable output for specific tasks. (Or it’s because you want more confidentiality than the general use models offer. 🥸 But you’ll take advantage of the additional training capabilities, right?)  So don’t let your in-house model fall into the same problem! Cull the garbage data, only feed it the good stuff. Consider these three practices to ensure only high-quality data ends up in your organization’s LLM. 1️⃣ Establish Data Quality Standards: Define what “good” data looks like. Clear standards are a good defense against junk info. 2️⃣ Review Data Thoroughly: Your standard is meaningless if nobody uses it. Check that data meets your standards before using it for training. 3️⃣ Set a Cut-off Date: Your sales contracts from 3 years ago might not look anything like the ones you use today. If you’re training an LLM to generate proposals, don’t give them examples that don’t match your current practices! With better data, your LLM will provide more reliable results with less revision needed. #AI #machinelearning #fciso

LLaMA-3 is a prime example of why training a good LLM is almost entirely about data quality… (1) Model architecture: Only 5 sentences are provided about the model architecture, which simply state that LLaMa-3 uses a standard decoder-only architecture with grouped query attention to improve inference efficiency (and a longer 8K context). It’s pretty clear that model architectures are becoming standardized, and most of the research focus is going into constructing datasets. In fact, the main architecture modification made by LLaMA-3 is a more efficient tokenizer! “Llama 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, which leads to substantially improved model performance.” - from LLaMA-3 blog (2) Better tokenizer: LLaMA-3 comes with a custom tokenizer with a vocabulary of 128K tokens (LLaMA-2 was 32K). This tokenizer is more token efficient (i.e., fewer tokens are necessary to encode the same piece of text relative to LLaMA-2), which makes inference more efficient. Authors also note that the new tokenizer improves performance! In other words, making sure that we are encoding the model’s input data correctly is super important. (3) Massive pretraining corpus: LLaMa-3 is pretrained over 15T tokens of text (5% non-English), which is a 7X improvement over LLaMA-2 and even larger than the 12T pretraining corpus of DBRX. The pretraining corpus also has 4X more code relative to LLaMA-2. With this in mind, it’s not a surprise that LLaMA-3 has strong reasoning/code capabilities (more code -> better reasoning). “We found that previous generations of Llama are surprisingly good at identifying high-quality data, hence we used Llama 2 to generate the training data for the text-quality classifiers that are powering Llama 3.” - from LLaMA-3 blog (4) FIltering pretraining data: Few concrete details are provided on the filtering process for the pretraining corpus of LLaMA-3, but it’s clear that a lot of filtering is done. These filters include heuristic filters, NSFW filters, semantic deduplication, and text classifiers to predict data quality. (5) Overtraining: Chinchilla proposed the compute optimal training regime for LLMs, but recent work indicates that pretty much everyone overtrains their LLMs relative to the compute-optimal ratio. LLaMA-3 is pretrained on two orders of magnitude more data (for the 8B model) beyond the compute-optimal ratio, and we still see log-linear improvements. “The quality of the prompts that are used in SFT and the preference rankings that are used in PPO and DPO has an outsized influence on the performance of aligned models.” - from LLaMA-3 blog (6) Post training data quality: Even beyond pretraining, data quality is pivotal for LLaMA-3! The model is aligned with a combination of SFT, rejection sampling, PPO, and DPO. The biggest quality improvements in LLaMA-3 came from curating this data and performing multiple rounds of quality assurance on humans annotations!

Happy new year!! I've been reflecting lately on "themes" instead of predictions or resolutions. With that in mind, I think for the #PeopleAnalytics space, the theme of 2024 will be around "data acquisition" and "data quality". We're a year into AI now. Everyone has played with ChatGPT and the phrase "training data" means something to most people in HR now (e.g. "that model didn't have access to the right training data"). I feel pretty confident saying there's never been a business technology released that had such a complete ability for everyone in business to interact with it from day 1 (and answer questions about itself!). That year of literacy was incredibly important for understanding AI, but I think even more important for understanding what makes it work (or break!). In 2023, an incredible amount of HR teams were tasked with "generative AI" projects. Let's get #GenAI to redo our job descriptions, figure out our skills, predict attrition, answer employee questions, and decipher / write policies. Coming into 2024, it's time to acknowledge that ask was misguided. Most teams quickly realized they couldn't get GenAI going. Like a car without gas, GenAI did not have the data it needed in most HR departments. So what's the fix? What is needed is an investment in data acquisition and data quality for machine learning operations (#MLOps). We need to capture the right data and then make sure that data is clean, clear, and architected in the right way to train AI models. I wish AI just worked without that step, but we're all a year in now and that has not been the case. You still need skill taxonomies, human reviewers, data governance teams, and data quality checks. It's stuff #PeopleAnalytics teams have been trying to get for years, but now we have leverage. Want GenAI? Let's get the data together. And if you're unsure where to start on data acquisition or data quality, reach out! Looking back on 2023 with One Model, I spoke with over 300 companies about #HRData, #PeopleAnalytics, and HR Data Infrastructures. I'm coming into 2024 excited to continue to drive this theme. 🛑 And as a final note, let's make this clear up front that the state of HR data is not HR's fault. I'm dedicating 2024 to screaming this from the rooftops: businesses have systematically underinvested and underserved HR teams for all of HR's history. HR knows what it needs, has been asking for what the function needs, and I'm going to do my part to make sure 2024 will be the year that #HR finally gets heard. What do you think? Are you going after data quality and data acquisition in 2024? What stands in your way?