AI-க்கான Data pipelines
ஒரு AI system-ஓட data-வை fresh-ஆ வச்சிருக்குற pipeline-ஐ design பண்ண முடியும்: ETL அல்லது ELT, scheduled அல்லது change-driven updates, incremental indexing, parsers அல்லது embedding models மாறும்போது re-processing.
Pro 1 · Quality engineerabout 4 hours7 sections
சுருக்கமா சொன்னா
ஒரு AI data pipeline, sources-ல இருந்து data-வை extract பண்ணி, சுத்தம் பண்ணி, transform பண்ணி, ஒரு schedule-லயோ ஒவ்வொரு change-லயோ index-ஐ update பண்ணும்; மாறினதை மட்டும் re-process பண்ணும்.
இது ஏன் முக்கியம்
ஒரு முறை export பண்ண data மேல கட்டின RAG system, வாரங்கள்ல பழசாகிடும்; பழைய answers, தப்பான answers-ஐ விட வேகமா நம்பிக்கையை அரிக்கும். நேத்து வந்த circular பத்தி assistant-க்கு தெரியணும்னு clients எதிர்பார்க்குறாங்க. அதை உண்மையாக்குறது pipelines தான், production RAG bugs-ல பெரும்பாலானவை இங்கதான் இருக்கு.
Sign up free to read the whole lesson
The rest of this lesson: 7 parts, video lessons, the best course, practice and a quick check. It is free, takes a few seconds and needs no card.
Accounts are for learners aged 18 and over. Under 18? The quick answer and the first part of every lesson stay open: browse lessons by topic.
Lessons on Generative AI & LLM Engineer0 of 65 done
- 1What AI, machine learning and a language model are
- 2Use a chat assistant for a real task and check the answer
- 3Use AI safely at work
- 4Set up Python and run your first file
- 5Variables, loops and functions in Python
- 6Write your own functions
- 7Lists and dictionaries
- 8Files and errors
- 9The terminal and Git
- 10Packages and virtual environments
- 11Call a web API and read JSON
- 12Tables with pandas
- 13Call an LLM API from code
- 14System prompts and prompt design
- 15Tokens and the context window
- 16How a model picks its next word: temperature, top-p and top-k
- 17Get structured JSON output
- 18Errors, retries and cost
- 19How a chat model is made: pre-training, SFT and preference tuning
- 20AI in Indian languages: tokens, quality gaps and what to use
- 21Embeddings
- 22Chunk documents for retrieval
- 23Vector search
- 24Build a RAG app
- 25Check whether retrieval works
- 26Parsing documents and OCR
- 27Metadata that helps retrieval
- 28Hybrid search, reranking and filters
- 29Query rewriting, HyDE and parent-child chunks
- 30Grounding, citations and refusing to guess
- 31Long context or RAG
- 32Tool calling
- 33The agent loop
- 34Logs and traces for agents
- 35How agents fail
- 36Long documents without vectors
- 37Test sets and evals
- 38LLM judges and their limits
- 39Choosing a model: benchmarks, leaderboards and your own tests
- 40Scoring open-ended answers: exact checks, similarity and side-by-side comparison
- 41Catching hallucinations: why models make things up and how to detect it
- 42Regression tests for prompts
- 43Defend against prompt injection
- 44Reasoning models and test-time compute: when to let the model think longer
- 45Data pipelines for AI
- 46Permission-aware RAG
- 47Graph RAG
- 48Multimodal RAG
- 49How fine-tuning works: data, training and what can go wrong
- 50LoRA, QLoRA and the GPU memory maths
- 51Fine-tuning in practice: tools, settings and reading the loss curve
- 52Synthetic data and distillation: training with model-made data
- 53Building a training dataset: quality, coverage and quantity
- 54Deploy and monitor an LLM app
- 55Caching and speed
- 56Inside LLM inference: prefill, decode and the KV cache
- 57Grow an AI app in layers: context, guardrails, routing, caching and agents
- 58Advanced harness engineering
- 59Harnesses for long-running agents
- 60Cost control
- 61Security and privacy
- 62Prompting, RAG or fine-tuning
- 63Design an AI system
- 64Quality, cost and speed trade-offs
- 65The design interview
Related
On these roadmaps: Generative AI & LLM Engineer, Forward Deployed Engineer (FDE)
DataPro 1: Quality engineerVideo lessonsETL and ELTChange data captureChunkingEmbedding