Narcis MiclausI'm a data engineer in Italy, working toward becoming a financial advisor. What I love about both: building solutions, solving problems, turning big complex goals into small clear steps. I write about it every day. Stay a while; maybe you'll learn something. Maybe I will too.https://narcismiclaus.com/Capstone: what you know now, where to go nexthttps://narcismiclaus.com/programming/python/60-capstone/https://narcismiclaus.com/programming/python/60-capstone/A look back at the 60 lessons, a look forward at where Python is heading, and the resources that take you from intermediate to expert.Fri, 19 Jun 2026 00:00:00 GMTpythoncourse-summarynext-stepsA 30-minute health check on a Spark cluster you've never seenhttps://narcismiclaus.com/programming/pyspark/60-spark-cluster-health-check/https://narcismiclaus.com/programming/pyspark/60-spark-cluster-health-check/The capstone checklist: hand over your laptop, you have until 5pm to figure out what's broken.Thu, 18 Jun 2026 00:00:00 GMTpysparksparkdbahealth-checkcourse-summaryAI vs ML in 2026: when to call an LLM, when to trainhttps://narcismiclaus.com/programming/python/59-ai-vs-ml/https://narcismiclaus.com/programming/python/59-ai-vs-ml/The decision that didn't exist five years ago: use a hosted model, fine-tune an open one, or train your own?Tue, 16 Jun 2026 00:00:00 GMTpythonaillmmachine-learningdeploymentAdaptive Query Execution: Spark 3.x's killer featurehttps://narcismiclaus.com/programming/pyspark/59-adaptive-query-execution/https://narcismiclaus.com/programming/pyspark/59-adaptive-query-execution/Dynamic partition coalescing, runtime skew handling, and join strategy switching — the configs to know and the cases AQE still can't help.Mon, 15 Jun 2026 00:00:00 GMTpysparksparkaqeoptimizationperformancePre-trained models + transfer learning + Hugging Facehttps://narcismiclaus.com/programming/python/58-transfer-learning/https://narcismiclaus.com/programming/python/58-transfer-learning/The realistic path from zero to a working deep learning model in 2026 — start with a pre-trained one, fine-tune on your data.Fri, 12 Jun 2026 00:00:00 GMTpythondeep-learningtransfer-learninghuggingfacefine-tuningDebugging slow Spark jobs: the 30-minute checklisthttps://narcismiclaus.com/programming/pyspark/58-debugging-slow-jobs/https://narcismiclaus.com/programming/pyspark/58-debugging-slow-jobs/The systematic loop for figuring out what's wrong with a slow job — read the UI, find the slow stage, look at task skew, GC, shuffle volume, in that order.Thu, 11 Jun 2026 00:00:00 GMTpysparksparkdebuggingperformanceproductionThe training loop, in codehttps://narcismiclaus.com/programming/python/57-training-loop/https://narcismiclaus.com/programming/python/57-training-loop/The five lines that turn a randomly-initialized network into a trained model — and the bookkeeping that makes them production-grade.Tue, 09 Jun 2026 00:00:00 GMTpythonpytorchtraining-loopdeep-learningMemory tuning: executor memory, overhead, OOM diagnosticshttps://narcismiclaus.com/programming/pyspark/57-memory-tuning/https://narcismiclaus.com/programming/pyspark/57-memory-tuning/The four configs that actually matter, what spill means, how to read an OOM stack trace, and the rule for sizing executors.Mon, 08 Jun 2026 00:00:00 GMTpysparksparkmemorytuningproductionPyTorch: the modern defaulthttps://narcismiclaus.com/programming/python/56-pytorch/https://narcismiclaus.com/programming/python/56-pytorch/Tensors, autograd, the nn module, and the Python-feel that made PyTorch win.Fri, 05 Jun 2026 00:00:00 GMTpythonpytorchdeep-learningtensorsReading execution plans: .explain(True), parsed to physicalhttps://narcismiclaus.com/programming/pyspark/56-execution-plans/https://narcismiclaus.com/programming/pyspark/56-execution-plans/How to read every line of .explain() output, the operators that matter, and the optimizer steps that produce them.Thu, 04 Jun 2026 00:00:00 GMTpysparksparkexplainexecution-plancatalystNeural networks in plain Englishhttps://narcismiclaus.com/programming/python/55-neural-networks-plain/https://narcismiclaus.com/programming/python/55-neural-networks-plain/What a neural network actually is, why backpropagation works, and where deep learning genuinely beats classical ML.Tue, 02 Jun 2026 00:00:00 GMTpythondeep-learningneural-networksfundamentalsThe Spark UI: the most important tool you'll learnhttps://narcismiclaus.com/programming/pyspark/55-spark-ui/https://narcismiclaus.com/programming/pyspark/55-spark-ui/A guided tour of every tab — Jobs, Stages, Tasks, SQL, Storage, Executors — and what each one tells you when something is wrong.Mon, 01 Jun 2026 00:00:00 GMTpysparksparkuidebuggingproductionML project: a classification problem, end to endhttps://narcismiclaus.com/programming/python/54-ml-project/https://narcismiclaus.com/programming/python/54-ml-project/From raw CSV to deployed model: the lessons of Module 9 made tactile.Fri, 29 May 2026 00:00:00 GMTpythonmachine-learningprojectend-to-endclassificationOutput modes and idempotent sinks: foreachBatch and the upsert patternhttps://narcismiclaus.com/programming/pyspark/54-output-modes-and-sinks/https://narcismiclaus.com/programming/pyspark/54-output-modes-and-sinks/Append vs update vs complete, the sinks Spark ships, and the foreachBatch escape hatch for everything else.Thu, 28 May 2026 00:00:00 GMTpysparksparkstreamingsinksidempotentforeach-batchHyperparameter tuning: grid, random, bayesian, optunahttps://narcismiclaus.com/programming/python/53-hyperparameter-tuning/https://narcismiclaus.com/programming/python/53-hyperparameter-tuning/The four search strategies, when each makes sense, and why optuna is the 2026 default.Tue, 26 May 2026 00:00:00 GMTpythonmachine-learninghyperparametersoptunagridsearchStateful operations: aggregations, sessions, and the state storehttps://narcismiclaus.com/programming/pyspark/53-stateful-operations/https://narcismiclaus.com/programming/pyspark/53-stateful-operations/Where Spark Streaming keeps the state between micro-batches, the standard stateful patterns, and when to drop down to mapGroupsWithState.Mon, 25 May 2026 00:00:00 GMTpysparksparkstreamingstatesessionizationLinear and regularized models: when simple winshttps://narcismiclaus.com/programming/python/52-linear-models/https://narcismiclaus.com/programming/python/52-linear-models/Why linear models are still the right answer surprisingly often — and the regularization tricks that make them production-grade.Fri, 22 May 2026 00:00:00 GMTpythonmachine-learninglinear-regressionlogistic-regressionregularizationWatermarks and event time: the part most beginners get wronghttps://narcismiclaus.com/programming/pyspark/52-watermarks-event-time/https://narcismiclaus.com/programming/pyspark/52-watermarks-event-time/Why event time matters more than processing time, what a watermark actually does, and the worked example with concrete timestamps.Thu, 21 May 2026 00:00:00 GMTpysparksparkstreamingwatermarksevent-timeTree-based models: random forest, XGBoost, LightGBMhttps://narcismiclaus.com/programming/python/51-tree-models/https://narcismiclaus.com/programming/python/51-tree-models/Why trees dominate tabular ML, the differences between the three big libraries, and the hyperparameters that matter.Tue, 19 May 2026 00:00:00 GMTpythonmachine-learningrandom-forestxgboostlightgbmgradient-boostingKafka source: the most common production ingesthttps://narcismiclaus.com/programming/pyspark/51-kafka-source/https://narcismiclaus.com/programming/pyspark/51-kafka-source/How Spark reads from Kafka, the offset semantics, and the at-least-once vs exactly-once question.Mon, 18 May 2026 00:00:00 GMTpysparksparkkafkastreamingstructured-streamingFeature engineering: the part that matters mosthttps://narcismiclaus.com/programming/python/50-feature-engineering/https://narcismiclaus.com/programming/python/50-feature-engineering/The transforms that turn raw data into model fuel — and the ones that quietly leak information from the future.Fri, 15 May 2026 00:00:00 GMTpythonmachine-learningfeaturespreprocessingStructured Streaming basics: readStream, writeStream, triggershttps://narcismiclaus.com/programming/pyspark/50-structured-streaming-basics/https://narcismiclaus.com/programming/pyspark/50-structured-streaming-basics/The streaming entry points, the trigger semantics, and the checkpoint that everything depends on.Thu, 14 May 2026 00:00:00 GMTpysparksparkstreamingstructured-streamingdataframescikit-learn: the standard ML library tourhttps://narcismiclaus.com/programming/python/49-scikit-learn/https://narcismiclaus.com/programming/python/49-scikit-learn/The fit/predict pattern that underpins everything, the model categories, and the pipelines that make code reproducible.Tue, 12 May 2026 00:00:00 GMTpythonscikit-learnmachine-learningmlWhy streaming, and what 'streaming' even means in Sparkhttps://narcismiclaus.com/programming/pyspark/49-why-streaming/https://narcismiclaus.com/programming/pyspark/49-why-streaming/Bounded vs unbounded data, batch-vs-streaming as a continuum, and why DStreams are deprecated in favor of Structured Streaming.Mon, 11 May 2026 00:00:00 GMTpysparksparkstreamingstructured-streamingfundamentalsThe 80% architecturehttps://narcismiclaus.com/programming/architecture/the-80-percent-architecture/https://narcismiclaus.com/programming/architecture/the-80-percent-architecture/The default modern SaaS stack that fits most companies. What it is, why it's standard, and when to deviate.Fri, 08 May 2026 00:00:00 GMTarchitecturereferencesaasdefault-stackWhy your best work happens when you're not at the keyboardhttps://narcismiclaus.com/programming/career/best-work-not-at-keyboard/https://narcismiclaus.com/programming/career/best-work-not-at-keyboard/The walks, the showers, the runs. Hard problems unblock when your conscious brain stops chewing on them. Why protecting non-coding time is part of the job.Fri, 08 May 2026 00:00:00 GMTcareerproductivitylearningdeep-workNumerical project: a real analysishttps://narcismiclaus.com/programming/python/48-numerical-project/https://narcismiclaus.com/programming/python/48-numerical-project/Take a public dataset and run a complete numerical analysis: descriptive stats, fits, hypothesis tests, plots.Fri, 08 May 2026 00:00:00 GMTpythonnumpyscipyprojectanalysisSchema evolution: when columns change underneath youhttps://narcismiclaus.com/programming/pyspark/48-schema-evolution/https://narcismiclaus.com/programming/pyspark/48-schema-evolution/Why schema-on-read formats handle change badly, why Avro+registry handles it well, and the Delta/Iceberg way of getting both.Thu, 07 May 2026 00:00:00 GMTpysparksparkschemaparquetavroevolutionJupyter, notebooks, and when to leave themhttps://narcismiclaus.com/programming/python/47-jupyter-and-notebooks/https://narcismiclaus.com/programming/python/47-jupyter-and-notebooks/Why notebooks are addictive, where they shine, and the moment you should stop and write a real script.Tue, 05 May 2026 00:00:00 GMTpythonjupyternotebooksvscodekernelCloud storage: S3, GCS, Azure Blob — what changeshttps://narcismiclaus.com/programming/pyspark/47-cloud-storage/https://narcismiclaus.com/programming/pyspark/47-cloud-storage/The consistency caveats, the rename problem, and why direct-write committers exist.Mon, 04 May 2026 00:00:00 GMTpysparksparks3cloudstoragehadoopCapstone: design a complete architecture for a fictional company at three scaleshttps://narcismiclaus.com/programming/architecture/80-capstone/https://narcismiclaus.com/programming/architecture/80-capstone/Eighty lessons of system architecture, condensed into one design exercise. The same fictional SaaS company, three scales, three architectures, and a guided tour of what changes and why. The closing lesson of the course.Fri, 01 May 2026 00:00:00 GMTarchitecturecapstonecourse-summaryThe Python features I learned too latehttps://narcismiclaus.com/programming/python/features-learned-too-late/https://narcismiclaus.com/programming/python/features-learned-too-late/Match statements, the walrus operator, f-string debugging, dataclasses, and other Python features that would have saved me hours if I'd known about them sooner.Fri, 01 May 2026 00:00:00 GMTpythontipsmodern-pythonWriting to JDBC: parallelism, batches, idempotencyhttps://narcismiclaus.com/programming/pyspark/46-writing-jdbc/https://narcismiclaus.com/programming/pyspark/46-writing-jdbc/How to write Spark output back to a relational database without crushing it, breaking transactions, or losing data on retry.Thu, 30 Apr 2026 00:00:00 GMTpysparksparkjdbcwritetransactionsWhen to leave a job (and when to stay)https://narcismiclaus.com/programming/career/when-to-leave/https://narcismiclaus.com/programming/career/when-to-leave/The signals that mean it's time to go and the ones that just mean you're having a bad month. A framework for the decision people most often get wrong.Wed, 29 Apr 2026 00:00:00 GMTcareerdecision-makingitalygrowthSciPy: the toolbox most people forget abouthttps://narcismiclaus.com/programming/python/45-scipy/https://narcismiclaus.com/programming/python/45-scipy/Statistics, optimization, signal processing, sparse matrices — the standard library of scientific Python.Tue, 28 Apr 2026 00:00:00 GMTpythonscipystatisticsoptimizationML platform architecture: feature store, model registry, servinghttps://narcismiclaus.com/programming/architecture/79-ml-platform/https://narcismiclaus.com/programming/architecture/79-ml-platform/The five layers a modern ML platform standardised on, the train-serve skew problem the feature store was invented to solve, and the build-versus-buy calculus for each layer in 2026.Mon, 27 Apr 2026 00:00:00 GMTarchitectureml-platformfeature-storemlflowservingReading from JDBC: pulling from Postgres, MySQL, SQL Serverhttps://narcismiclaus.com/programming/pyspark/45-reading-jdbc/https://narcismiclaus.com/programming/pyspark/45-reading-jdbc/The JDBC source connector, the partitionColumn trick, and why a naive read kills your source database.Mon, 27 Apr 2026 00:00:00 GMTpysparksparkjdbcpostgresmysqlparallel-readPrivacy and compliance: GDPR, CCPA, data residencyhttps://narcismiclaus.com/programming/architecture/78-privacy-compliance/https://narcismiclaus.com/programming/architecture/78-privacy-compliance/Privacy regulations as architectural drivers. Right to erasure, data residency, customer-managed keys, and the consent and audit infrastructure compliance frameworks require.Fri, 24 Apr 2026 00:00:00 GMTarchitecturegdprccpaprivacycomplianceresidencyPlotting: matplotlib, seaborn, plotly — picking yourshttps://narcismiclaus.com/programming/python/44-plotting/https://narcismiclaus.com/programming/python/44-plotting/Three plotting libraries, three philosophies, and which one to reach for given the audience.Fri, 24 Apr 2026 00:00:00 GMTpythonmatplotlibseabornplotlyvisualizationORC, Avro, Delta: the alternatives and when each winshttps://narcismiclaus.com/programming/pyspark/44-orc-avro-delta/https://narcismiclaus.com/programming/pyspark/44-orc-avro-delta/Three format families that aren't Parquet, when each is the right choice, and why Delta has been quietly taking over.Thu, 23 Apr 2026 00:00:00 GMTpysparksparkorcavrodeltafile-formatSecurity architecture: least privilege, defense in depthhttps://narcismiclaus.com/programming/architecture/77-security-architecture/https://narcismiclaus.com/programming/architecture/77-security-architecture/The security principles every system needs as load-bearing architecture. Least privilege, defense in depth, zero trust, and the IAM and network controls that turn principles into reality.Wed, 22 Apr 2026 00:00:00 GMTarchitecturesecurityiamleast-privilegeNumPy: arrays, broadcasting, the foundation of scientific Pythonhttps://narcismiclaus.com/programming/python/43-numpy/https://narcismiclaus.com/programming/python/43-numpy/What an ndarray is, why broadcasting changes how you write loops, and the small set of functions that cover most use cases.Tue, 21 Apr 2026 00:00:00 GMTpythonnumpyarraysbroadcastingvectorizationDisaster recovery: RTO, RPO, the drillhttps://narcismiclaus.com/programming/architecture/76-disaster-recovery/https://narcismiclaus.com/programming/architecture/76-disaster-recovery/What disaster recovery actually means in practice. The four DR tiers, RTO and RPO as design dials, and the discipline of the drill that proves the plan works.Mon, 20 Apr 2026 00:00:00 GMTarchitecturedisaster-recoveryrtorpobackupsParquet: why it's the default for a reasonhttps://narcismiclaus.com/programming/pyspark/43-parquet/https://narcismiclaus.com/programming/pyspark/43-parquet/Columnar storage explained, compression codecs, predicate pushdown, and the row-group structure that makes selective reads fast.Mon, 20 Apr 2026 00:00:00 GMTpysparksparkparquetfile-formatcolumnarMulti-region deployments: active-active, active-passive, follow-the-sunhttps://narcismiclaus.com/programming/architecture/75-multi-region/https://narcismiclaus.com/programming/architecture/75-multi-region/Why teams go multi-region (latency, DR, compliance, capacity), the three deployment shapes, the hard problems (replication, conflicts, cost), and when not to bother.Fri, 17 Apr 2026 00:00:00 GMTarchitecturemulti-regiongeographylatencyData engineering project: build a real pipelinehttps://narcismiclaus.com/programming/python/42-data-engineering-project/https://narcismiclaus.com/programming/python/42-data-engineering-project/From source to destination, with monitoring, idempotency, and a schedule. The lessons of Module 7 made tactile.Fri, 17 Apr 2026 00:00:00 GMTpythondata-engineeringpipelineprojectend-to-endTungsten: code generation and the columnar memory layouthttps://narcismiclaus.com/programming/pyspark/42-tungsten/https://narcismiclaus.com/programming/pyspark/42-tungsten/How Spark fuses operations into compiled code, the off-heap columnar format, and why DataFrame Spark is fast.Thu, 16 Apr 2026 00:00:00 GMTpysparksparktungstenperformanceinternalsA 30-minute SQL Server health check: the script you actually runhttps://narcismiclaus.com/programming/sql-server/40-health-check/https://narcismiclaus.com/programming/sql-server/40-health-check/The ordered checklist I run on any SQL Server I've never seen before. Printable, pasteable, and the closing lesson of the course.Thu, 16 Apr 2026 00:00:00 GMTsql-serverdbahealth-checkcourse-summaryLuca at 18: what I wish someone had told mehttps://narcismiclaus.com/finance/why-im-writing-about-money/https://narcismiclaus.com/finance/why-im-writing-about-money/A letter to a first-year university student. Three concrete things to do in year one. The capstone of the course — everything in 2,500 words.Thu, 16 Apr 2026 00:00:00 GMTintropersonal-financecapstonelucaEvent-driven architecture: saga, choreography, orchestrationhttps://narcismiclaus.com/programming/architecture/74-event-driven-architecture/https://narcismiclaus.com/programming/architecture/74-event-driven-architecture/Services that talk by emitting events, the choreography vs orchestration choice, the saga pattern, and the 2026 toolset (Temporal, Step Functions, Camunda, Argo).Wed, 15 Apr 2026 00:00:00 GMTarchitectureevent-drivensagachoreographyorchestration