Francesco Mucio’s Post

Just wrote this on r/dataengineering Introducing Open Transformation Specification (OTS) – a portable, executable standard for data transformations https://lnkd.in/dCdse2Gt Hi everyone, I’ve spent the last few weeks talking with a friend about the lack of a standard for data transformations. Our conversation started with the Fivetran + dbt merger (and the earlier acquisition of SQLMesh): what alternative tool is out there? And what would make me confident in such tool? Since dbt became popular, we can roughly define a transformation as: - a SELECT statement - a schema definition (optional, but nice to have) - some logic for materialization (table, view, incremental) - data quality tests - and other elements (semantics, unit tests, etc.) If we had a standard we could move a transformation from one tool to another, but also have mutliple tools work together (interoperability). Honestly, I initially wanted to start building a tool, but I forced myself to sit down and first write a standard for data transformations. Quickly, I realized the specification also needed to include tests and UDFs (this is my pet peeve with transformation tools, UDF are part of my transformations). It’s just an initial draft, and I’m sure it’s missing a lot. But it’s open, and I’d love to get your feedback to make it better. I am also bulding my open source tool, but that is another story.

NL→SQL PoC → RAG-ready MVP (ASA series) Post 1 — The “Happy Path” (one prompt → safe chart) A copilot isn’t “AI magic.” It’s a controlled path: prompt → plan → validated SQL → chart. What I demo (E2E path, 1 page): Prompt→ → Router outcome (qa | sql | handoff | unsafe | clarify) → Planner JSON (intent, slots, candidate_sql, viz_hint) → Glossary hits & disambiguation (or “clarify”) → SQL validator (allowlist, dry-run bytes, row/scan caps) → Execution (BigQuery job id + stats) → VizSpec JSON (mark, encodings, filters) → Rendered chart + explain block (tables, filters, p95) Why this de-risks scope - One concrete, repeatable flow = shared reality for PM/Eng/Stakeholders. - Every transition is observable (JSON artifacts, job stats). - Guardrails make failures explicit and safe. Two outcomes (both are “wins”) - Safe path: chart renders; artifacts logged; success bars updated. - Blocked path: policy/validator stops it; refusal message + next step (clarify or handoff). No silent wrong answers. Success bars we track - Faithfulness (groundedness to retrieved/queried data) - Latency (p95) end-to-end - Cost per query (planner+model+warehouse) Thresholds are explicit; promotions require bars ≥ target across the golden set. #AISolutionsArchitect #RAG #NL2SQL #MLOps #VertexAI #DataProducts

𝐃𝐚𝐲: 11 𝐖𝐡𝐚𝐭 𝐈𝐬 𝐀𝐩𝐚𝐜𝐡𝐞 𝐀𝐢𝐫𝐟𝐥𝐨𝐰? 𝐀𝐩𝐚𝐜𝐡𝐞 𝐀𝐢𝐫𝐟𝐥𝐨𝐰 is a tool used to 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐞, 𝐬𝐜𝐡𝐞𝐝𝐮𝐥𝐞, 𝐚𝐧𝐝 𝐦𝐚𝐧𝐚𝐠𝐞 𝐝𝐚𝐭𝐚 𝐰𝐨𝐫𝐤𝐟𝐥𝐨𝐰𝐬 — meaning it helps you make sure your data tasks happen in the right order and at the right time without you doing them manually every day. It’s like giving your computer a 𝐭𝐨-𝐝𝐨 𝐥𝐢𝐬𝐭 for data — and it follows it automatically. 𝐖𝐡𝐲 𝐈𝐭’𝐬 𝐔𝐬𝐞𝐝 When working with data, you might need to: Extract data from websites or databases Clean or transform it Load it into a data warehouse Airflow helps 𝐜𝐨𝐧𝐧𝐞𝐜𝐭 𝐚𝐥𝐥 𝐭𝐡𝐞𝐬𝐞 𝐬𝐭𝐞𝐩𝐬 𝐚𝐧𝐝 𝐫𝐮𝐧 𝐭𝐡𝐞𝐦 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐜𝐚𝐥𝐥𝐲 — like a chain reaction. 𝐑𝐞𝐚𝐥-𝐋𝐢𝐟𝐞 𝐄𝐱𝐚𝐦𝐩𝐥𝐞 Imagine you’re the 𝐜𝐥𝐚𝐬𝐬 𝐦𝐨𝐧𝐢𝐭𝐨𝐫 in charge of preparing the attendance report every morning: You 𝐜𝐨𝐥𝐥𝐞𝐜𝐭 attendance from all classes (Extract) You 𝐜𝐡𝐞𝐜𝐤 𝐟𝐨𝐫 𝐦𝐢𝐬𝐭𝐚𝐤𝐞𝐬 like duplicate names (Transform) You 𝐬𝐮𝐛𝐦𝐢𝐭 the final list to the principal (Load) Instead of doing this manually every day, you can schedule it — that’s exactly what Apache Airflow does for data engineers. It’s like setting an 𝐚𝐥𝐚𝐫𝐦 𝐜𝐥𝐨𝐜𝐤 𝐟𝐨𝐫 𝐝𝐚𝐭𝐚 𝐭𝐚𝐬𝐤𝐬 — once set, it wakes up and finishes the work for you in the right order. My Takeaway What I really liked about Apache Airflow is how it simplifies complex workflows. Instead of manually running multiple scripts every day, you just schedule it once — and Airflow takes care of the rest. It reminded me how 𝐜𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲 𝐚𝐧𝐝 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 save a lot of time, not just in data engineering but in life too! 

Let's be honest: spending hours reverse-engineering a data model just to understand its structure is really challenging. Without a shared visual, it's a struggle to identify inconsistencies or find opportunities to improve the data model. Then, reaching an agreement with team members on changes can take weeks of meetings, slowing down the entire development process. The new data modeling experience in MongoDB Compass streamlines data modeling in MongoDB, enabling developers to harness the flexibility of the document model. With it, you can: ✅ Understand complex schemas by visualizing them through a clear entity-relationship diagram ✅ Refactor data models using an interactive graphical interface. ✅ Collaborate with teams to gather feedback. Refactored data models can be shared as images, JSON files, or .mdm files that open directly in Compass. This tool is generally available today in the latest version of Compass. See it in action ⬇️ https://lnkd.in/gfxP-Gis

𝐘𝐨𝐮𝐫 𝐜𝐨𝐝𝐞 𝐰𝐢𝐥𝐥 𝐛𝐞 𝐫𝐞𝐰𝐫𝐢𝐭𝐭𝐞𝐧 𝐧𝐞𝐱𝐭 𝐪𝐮𝐚𝐫𝐭𝐞𝐫. 𝐁𝐮𝐭 𝐭𝐡𝐚𝐭 𝐝𝐚𝐭𝐚 𝐲𝐨𝐮'𝐫𝐞 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐭𝐨𝐝𝐚𝐲? 𝐈𝐭'𝐥𝐥 𝐬𝐭𝐢𝐥𝐥 𝐛𝐞 𝐫𝐮𝐧𝐧𝐢𝐧𝐠 𝐢𝐧 2030. Here's why that changes everything about how you design systems: Every piece of code that touches data is secretly playing telephone with a future version of itself. Your database reader is a "different person" than your database writer, which is also different from your API consumer and your API producer. Here's the thing: these "conversations" happen across different timescales. An API call expects a response in milliseconds. A database write might not be read for years. Understanding these patterns is crucial for building systems that don't break when things change. In this carousel, you'll learn: - Why data outliving code is the golden rule - The fundamental difference between APIs and databases (it's about time) - How app servers become clients (and why that matters) - The real difference between REST and RPC (hint: it's not what you think) - Why REST is a philosophy, not a protocol Shoutout to Martin Kleppmann's "Designing Data-Intensive Applications" for crystallizing these concepts! 👇 Swipe to decode the hidden conversations happening in your stack. Feel free to add resources on these topics below! This is a data community space 🗣️ Follow for more #ByteSized snackable data engineering insights 💡 #DataEngineering #SystemDesign #APIs #DatabaseDesign #SoftwareArchitecture

TLDR - 🐳 How to Use 1. Pull the image from Docker Hub docker pull dbhitman/sql2025samples:latest 2. Run the container in detached mode docker run -d -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Welcome123!" -p 1433:1433 dbhitman/sql2025samples:latest Welcome123! is the default SQL Server sa password. You can change it if needed. 3. Connect via SSMS - Server name: localhost,1433 - Authentication: SQL Server Authentication - Login: sa - Password: Welcome123! 🚀 Launch Alert! After months of ritualizing schema design, reverse engineering, and container-native onboarding… I’m thrilled to unveil SQL2025Samples — a fully-loaded SQL Server Docker image featuring 7 realistic, modular databases for demos, dashboards, and SSMS exploration. 🧪 From Indian aviation to restaurant logistics, event ticketing to agri-tech, fitness tracking to pet care and travel-tech — this image is built for educators, engineers, analysts, and anyone who loves clean schemas and seeded data. 🔧 Plug it into your CI/CD pipeline, reverse engineer with SSMS, or just explore the schemas to spark your next project. 📦 Docker Image 👉 https://lnkd.in/gspCmC39 📚 GitHub Project 👉 https://lnkd.in/gUctkjEX 💬 I’d love your feedback, forks, and feature ideas on my Github page. #SQL2025Samples #Docker #SQLServer #SSMS #DataEngineering #OpenSource #CI_CD #ContainerNative #TechCommunity #dbhitman #DatabaseSamples #SQLServer2025

I’ve been thinking a lot about how to go beyond just spotting issues and actually start fixing them automatically. My goal is to build a self-healing system that can detect a problem, try to fix it, and only involve a human if it gets stuck. In this post, I share Phase 1 of my pet project. I walk through how to: • Pull information from SQL Server • Set up a Jira project to track automation • Send SQL Server data to Jira automatically This is just the foundation. Eventually I plan to integrate AI so tickets can be interpreted and acted on before a human ever sees them. I’m considering adding screenshots. I don't like how quickly things in Jira change and are outdated but concepts are the same. You can read the full post and access the GitHub links here: https://lnkd.in/ga2BjAUh

Turn complex documents into RAG-ready data! Here are the highlights from our latest issue: 1️⃣ Agentic Document Extraction (ADE) - Parse Jobs API Convert visually complex documents into structured, layout-aware JSON with visual grounding.   - New Parse Jobs API for async large-scale processing (up to 1GB or 1,000 pages)   - Extracts text, layout, and hierarchy not just OCR text   - Handles multi-column layouts and complex tables with precision 2️⃣ Fine-Tune 100+ LLMs Without Writing Code LLaMA-Factory lets you train and fine-tune open-source LLMs & VLMs via CLI or Web UI no code required. Supports 100+ models like LLaMA, Gemma, Qwen, Mistral, and DeepSeek. - Zero-code CLI & Web UI for training, merging, and evaluation  - Supports full-tuning, LoRA, QLoRA, PPO/DPO, OFT, and reward modeling   - Optimized with FlashAttention-2, RoPE scaling, and vLLM backend 3️⃣ MCP Containers: Containerized versions of 400+ MCP servers Running MCP servers manually can be messy. MCP Containers makes it effortless with secure, auto-updated, containerized servers. - 450+ pre-containerized MCP servers with auto-updates   - Secure, isolated environments with Nixpacks support  - Covers GitHub, Stripe, Cloudflare, databases, and more Read the full issue here: https://lnkd.in/gXxFn5xH

Model your SQL as data, not strings In my spare time I started exploring an idea that I always felt was missing — a way to treat SQL as structured, immutable data rather than plain strings. That exploration became SQM – Structured Query Model, and I’ve just released the first prototype. SQM represents SQL as a composable object model — an Abstract Syntax Tree (AST) for SQL. It lets you build, transform, and render queries programmatically. Each node (Query, Expression, Predicate, etc.) is immutable, serializable (JSON), and can be rendered or transformed. It’s not an ORM or a query builder — it’s a domain model for SQL itself, useful for analysis, rewriting, and code generation tools. GitHub: https://lnkd.in/d2jtsusH What do you think about this concept? Would such a framework be useful in your projects or tooling?

𝗗𝗕𝗧 𝗰𝗵𝗲𝗮𝘁 𝘀𝗵𝗲𝗲𝘁 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 Writing SQL is easy. Making it reliable at scale? That’s where dbt comes It brings order to your SQL — with testing, versioning, and clean data flow ✅ Modular, tested SQL with ref() and CTEs ✅ YAML-driven lineage and metadata ✅ Macros and Jinja to remove repetition ✅ Incremental models for big data ✅ Snapshotting, documentation, exposures ✅ Best practices that keep your DAG clean Built to help you ship production-grade models faster. Follow Abhishek Agrawal for more Data Engineering insights, tips, and best practices! 🔗 𝑱𝒐𝒊𝒏 𝒐𝒖𝒓 𝑾𝒉𝒂𝒕𝒔𝑨𝒑𝒑 𝒄𝒉𝒂𝒏𝒏𝒆𝒍 𝒕𝒐 𝒔𝒕𝒂𝒚 𝒖𝒑𝒅𝒂𝒕𝒆𝒅 𝒐𝒏 𝑫𝒂𝒕𝒂 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓𝒊𝒏𝒈! https://lnkd.in/dUuscrch 📲