--- title: Ian Rudd, PhD: AI systems you can defend url: https://drrudd.com/ language: en-CA description: I architect the AI platforms large organisations run on, and I lead the teams that build them. Close to twenty years taking artificial intelligence out of... modified: 2026-08-09T15:31:02-04:00 --- # Ian Rudd, PhD Chief Enterprise AI Architect, PhD in Machine Learning & AI, 20+ years of experience I architect the AI platforms large organisations run on, and I lead the teams that build them. Close to twenty years taking artificial intelligence out of the research lab and into production, across federal government, banking, insurance, national retail and transport. I set the architecture, lead the data science, data architecture and engineering teams that build to it, and prove the result holds up in front of an auditor. Figure 1 is the failure I get hired to prevent: a system that is sure of itself and wrong. Chief AI Architect, Government · Banking · Insurance · Retail, Teams, labs and platforms led, English and French Contact Me A hundred decisions, made by an AI system on its own - 67 right - 19 wrong, and it said it was unsure - 14 wrong, and it said it was confident It tells you it will be right 81 in 100 It is actually right 67 in 100 Figure 1. Every square is one decision. Blue, it got right. Grey, it got wrong while saying it was unsure, so a person could have caught it. Red, it got wrong while reporting it was confident, so nothing was flagged and nobody looked. A demo will never show you the red ones, because a demo shows answers and never how sure the system was. Finding them is the first thing I do. A simulated system, not a client’s: 2,000 decisions generated in your browser, shown as 100 squares (seed 7). measured ## The short version - 20 Years in artificial intelligence, from a corporate research lab to a national government platform - 6 Industries delivered in: government, banking, insurance, retail, transport and AI/ML research (IBM & Microsoft) - 1M+ Payments and claims a day checked for fraud by systems I designed - PhD Machine Learning and AI, alongside certifications in architecture, security and project management - 3 Patents granted, along with a number of published research papers and seminars on Deep Learning and AI Safety - 2 Official languages, English and French, delivered to the same standard ## What overconfidence costs you Most teams put a confidence threshold in front of their AI: anything the model is sure enough about goes through without a human looking at it. That rule is only ever as good as the confidence score behind it, and confidence scores are usually inflated. Put your own volume and error cost in below, and watch what the rule is actually letting through. Approve without review above 90% Decisions a month Cost of one wrong decision ### The model as shipped Goes through unreviewed 47.3% Wrong, among those 17.4% Bad decisions a year 9,870 Certainty not earned +14.2% Annual exposure $2.47M ### After calibration Goes through unreviewed 8.1% Wrong, among those 3.1% Bad decisions a year 300 Certainty not earned -3.6% Annual exposure $75.0K What this shows. At this threshold the model as shipped waves through 47.3% of your cases and gets 17.4% of them wrong, while reporting at least 90% confidence on every single one. Calibration does not make the model smarter and does not reorder a single answer. It makes the number mean what it says. The money is your two numbers multiplied by the measured rate: 10,000 a month × 12 months × 47.3% through the gate × 17.4% wrong × $250 each = $2.47M a year. derived Notice the trade. The corrected column automates less: 8.1% of cases instead of 47.3%. That is the decision worth making on purpose rather than by accident. Less automation, in exchange for decisions you can defend afterwards. And where it doesn’t apply This is a synthetic model, not yours. The two inputs are your own and everything after them is counting, but the error rates come from a generated classifier, so read this as the shape of the problem rather than a quote for your system. Measuring it properly on your data is a two-week job. ## What I do Ten things organisations bring me in for. Some are strategy and architecture, some are hands-on modelling, and some are leading the teams that has to keep the result running after I leave. Each one lists when you would call me, what I actually do, what lands on your desk at the end, and the condition that makes the whole area not worth hiring anyone for. ### AI strategy and enterprise architecture When you call me in You have pilots that impress in a demo and nothing that survives contact with security, procurement, or a second team trying to build on it. I set the direction and the shape: reference architectures, standards, security patterns and landing zones, the review gates a solution has to clear before production, and the serving layer that lets business applications reach a model through a stable, secured contract. I do the arithmetic that decides hardware and hosting before anyone signs, so the cost of running the thing is a number rather than a surprise. Then I take the options, the cost and the risk into the room where the funding decision actually gets made, in the language that room uses. What you get A reference architecture with the standards and review gates that go with it, a multi-year roadmap tied to business priorities, and a cost and capacity model your vendor can be held to. Not worth doing if Nobody owns the platform after I leave. Standards without a named owner become documentation, and documentation does not stop the next isolated pilot. ### Assistants that answer from your own documents When you call me in The assistant answers from your policies, contracts or records, and it is confidently wrong often enough that nobody trusts it any more. I treat search as the main system rather than a preprocessing step: how documents get split, what metadata rides along with each piece, combining keyword matching with meaning-based matching, re-ranking what comes back, and instrumenting how often the right source was found at all. Where the rules change over time, the answer has to be the rule that was in force on the date asked about, with the clause cited so a person can verify it in seconds rather than trusting a paraphrase. What you get A retrieval scorecard that separates “never found it” from “found it and answered badly”, the chunking and metadata design behind it, and the fixes ranked by what each one buys. Not worth doing if Nobody will label which documents were relevant. Without that, every quality number is inferred from the final answer and the search step stays invisible. ### Agents that know when to stop When you call me in You want the system to take actions rather than just answer questions, and nobody can tell you what it will do on the day it is wrong. A planner that decomposes a request into typed tasks, specialist agents underneath it, a typed contract on every tool the agent can reach, step and token budgets, cycle detection, and a fallback ladder that narrows the answer and then hands to a person rather than inventing something. Anything the agent retrieves is treated as untrusted input, with injection containment, tool allowlists, output filtering and exfiltration checks kept in the regression suite rather than run once at launch. What you get An agent design with its tool contracts, budgets and escalation path written down, plus the red-team suite that runs on every change instead of on launch day. Not worth doing if The task has no clean handoff to a human. An agent with nowhere safe to fail is a liability whatever its accuracy. ### Models beyond the chatbot When you call me in The decision that actually costs you money is a forecast, a risk score or a triage call, and a language model is not the answer to it. Most of the value in a large organisation is still ordinary machine learning done properly. Demand forecasting reconciled across item, site and network so the plans add up. Fraud and risk scoring under extreme class imbalance, measured at a fixed alert budget because analyst capacity is the binding constraint. Duration and return to work as a survival problem with competing risks kept separate. Document understanding over mixed correspondence, with human review on anything that drives a payment. Recommenders, ranking, and the sequence models underneath extraction. What you get A model that fits the decision it feeds, a metric chosen to match what somebody actually does with the output, and backtesting that does not flatter whoever picked the split. Not worth doing if The decision it feeds is not really made on the number. A model attached to a process nobody is willing to change is an expensive report. ### Confidence you can act on When you call me in Nobody can tell you which of the system’s answers are safe to use without a person checking them first. I measure whether the stated confidence matches reality, correct it where it does not, and set the threshold where sending a case to a person is cheaper than getting it wrong. Where a hard guarantee is needed, conformal prediction puts a provable error rate on the system without assuming the model is any good. What you get A measured calibration report, a threshold policy with the cost arithmetic behind it, and a routing rule your operations team can actually run. Not worth doing if The calibration set stops resembling production. Everything here assumes the future looks like the sample you measured, and a quiet shift in the data invalidates the whole apparatus without raising an alarm. ### Evaluation that can fail When you call me in The demo works, the metrics look good, and the system keeps embarrassing you in front of real users. Building evaluation sets from failures you have actually seen rather than from convenient examples. Error taxonomies, results broken out by segment, hard cases held separately from representative ones, model-graded scoring checked against human raters instead of trusted on faith, significance testing on every claimed improvement, and the whole suite wired into the build so a prompt change cannot quietly cost you recall. What you get An evaluation set built from your real failures, with a named owner, and a reporting format that makes a regression impossible to miss. Not worth doing if Nobody is willing to own the labelling. Evaluation is a data-collection problem wearing a metrics costume, and it stalls the moment annotation has no owner. ### Data that can leave the building When you call me in You need to use sensitive data, or publish something derived from it, and nobody is willing to sign off. Deciding what crosses a boundary, and proving it. Detecting personal information and measuring how often the detector misses, irreversible redaction, differential privacy for aggregate releases, budget accounting across a whole query workload, and re-identification risk assessment. Alongside it, the governance a reviewer asks for: subgroup performance testing, explanations an assessor can act on, data residency, lineage, and model risk controls built into the first diagram rather than the week before launch. What you get A written threat model, a measured detection error profile, a privacy budget with a ledger against it, and the fairness and explainability evidence a reviewer will ask for. Not worth doing if The threat model is unstated. Detection rates, noise levels, and budgets are all meaningless until someone writes down who the adversary is and what they already know. ### From pilot to production, and staying there When you call me in The model works on somebody’s laptop, and every release since has depended on that person being available. Model and prompt registries, versioned data, automated evaluation gates a release has to clear before promotion, pipelines across development, test and production, and a rollback path that does not depend on anyone being awake. Then the monitoring that tells you it is still working: answer quality, retrieval quality, drift on the inputs, latency, failure rates and cost per interaction, reported as numbers rather than opinions. What you get A repeatable, auditable deployment path, the monitoring that sits on top of it, and an incident runbook for the day a model starts drifting. Not worth doing if The organisation cannot yet deploy ordinary software reliably. Machine learning operations built on broken software operations inherits every one of those problems and adds several of its own. ### Data platforms your models can stand on When you call me in Every model project starts with six weeks of finding, cleaning and arguing about the data, and the argument never gets settled the same way twice. Before there is a model there has to be a platform. Lakehouse and warehouse design, streaming and batch pipelines, feature stores, vector stores, and the multi-cloud and event-driven integration underneath all of it. Then the part most programmes skip: who owns which dataset, what a consumer is entitled to assume about it, where it came from, and what happens the day it changes. I have led data architecture teams through exactly this, and cut the path from raw event to usable training set from days to hours, which is what makes frequent model refreshes possible at all. What you get A target data architecture with data contracts, lineage and ownership named against each domain, and pipelines a second team can build on without asking you first. Not worth doing if The organisation will not agree on who owns a dataset. Every technical fix downstream of that question is temporary, and you will be back inside a year. ### Teams, practices and labs When you call me in You have people doing artificial intelligence work and no practice around them, or you need somebody to lead the group while it is still being built. I have built machine learning, data and architecture practices from nothing inside large organisations, led data science teams, data architecture teams and engineering teams, and run a corporate research lab where results had to be publishable and also had to ship. In practice that means the operating model, the intake and prioritisation path, the decision forums, hiring and mentoring, and the enablement sessions that leave a client able to run the thing without me. It also means saying no to the proposals that were never going to work, early, in front of the people who proposed them. What you get A working operating model with named owners, a prioritised pipeline of work that is genuinely deliverable, and a team that does not need me in the room. Not worth doing if Leadership wants the capability without the authority to change how decisions get made. A practice that cannot decline a bad request is a queue, not a capability. ## Where I’ve done it Described by role, sector and constraint rather than by employer. What matters for judging whether I can help you is the shape of the problem, the seat I held while solving it, and the rules it all had to run under. ### Federal Government of Canada 2025 to now Role Chief AI architect Department wide, bilingual What I built and led Set the enterprise AI direction for a federal department: reference designs, security patterns, landing zones, and the review gates a solution clears before it goes near production. Built the platform that delivery teams work on, the assistants that answer from the department’s own records, and the multi-agent orchestration behind the harder journeys. Built the machine learning and architecture practice itself, from ways of working to who owns what, and advised at director general and assistant deputy minister level on cost, risk and trade-offs. What made it hard Protected information, equal service in both official languages, procurement-grade documentation, and a full audit trail behind every answer given. ### Public Sector Insurance 2022 to 2025 Role Lead consultant, AI and data strategy Claims, medical and financial records What I built and led Stood up the AI architecture and governance foundation, then led the modelling behind claim risk: duration and return to work treated as a survival problem, calibrated so an adjudicator could act on the number, with competing risks handled explicitly rather than collapsed into one label. Added document understanding over medical and financial correspondence, knowledge assistants grounded in policy, and the fairness and explainability layer the regulator was always going to ask about. What made it hard Health and financial record sensitivity, automated decisions that affect whether a person is paid, and every one of those decisions open to appeal. ### Transport and logistics 2019 to 2022 Role Consultant, machine learning and data architecture National network What I built and led Established the enterprise architecture and machine learning practice, built the platform underneath it, and set the forecasting approach: volumes reconciled across item, facility and network level so the plans actually added up, sparse demand handled on its own terms, and delivery estimates framed as a window a customer experiences rather than an average nobody does. What made it hard Sparse and seasonal demand, a network where forecasts that are individually plausible can still be collectively wrong, and an operation planning against them every day. ### Banking and financial services 2015 to 2019 Role Advisory director, machine learning delivery ~1M events a day What I built and led Set the modelling posture for fraud and risk over transaction streams at roughly a million events a day: cost-sensitive learning under extreme class imbalance, precision measured at a fixed alert budget because analyst capacity is the real constraint, and graph features to catch the rings that look innocent one account at a time. Built the operations pattern the teams worked to, and took architecture and model risk positions into chief information officer and chief risk officer forums. What made it hard Prudential model governance: every change to a model is an auditable event with a named owner, and adversaries who adapt the moment you stop watching. ### Retail and consumer goods 2010 to 2015 Role Senior manager, enterprise machine learning and data platforms Item, store and distribution centre What I built and led Led the architecture modernisation that widened the organisation’s capacity for machine learning: multi-cloud strategy, event-driven integration, and feature pipelines that cut the path from raw event to usable training set from days to hours. Set the demand forecasting architecture, reconciled across item, store and distribution centre, with promotional lift and cannibalisation modelled rather than assumed, and built the recommender foundations underneath merchandising. What made it hard Seasonality, holiday regimes that behave like their own separate problem, and a forecast horizon shorter than the supply chain it was steering. ### AI/ML research (IBM & Microsoft) 2007 to 2010 Role Architect and team lead, AI research and engineering Research into product What I built and led Led the research and engineering team behind a question answering system: question decomposition, passage retrieval, evidence scoring and answer ranking, fanned out across parallel components, merged, with a defined path for when confidence came back too low to answer at all. That is the same shape as the agent orchestration problem the field is rediscovering now, roughly a decade before it had a name. Set the research agenda, mentored engineers and graduate interns, and shipped it to clients. What made it hard Research to product transfer: results that had to be publishable and also had to ship, with every claimed improvement held to a significance test that killed about half of them. ## What I’ll tell you that a vendor won’t Eight things I will defend in a room, from the boardroom question down to the engineering one. Each comes with the condition that voids it, stated plainly, in red. A vendor will not give you those, and that is the whole difference. ### Most AI programmes fail on operations, not on the model. What it costs you The budget goes to the part that was never the bottleneck, and the programme stalls a year in with nothing in production. The modelling is usually the shortest part of the work. What kills programmes is everything around it: nobody owns the data, there is no repeatable path from a laptop to production, there is no monitoring, no rollback, no answer for the auditor, and no named person accountable when quality drifts. Fund those first and the modelling looks easy. Fund the modelling first and you get a demo. And where it doesn’t apply Not true at the research frontier. If you are genuinely attempting something nobody has done, the scientific risk is real and dominates everything else. Very few enterprise programmes are in that position, and most of the ones that believe they are have never checked. ### Buying a platform does not buy you a capability. What it costs you You pay for the licence and still need the team, so the business case that justified the purchase was wrong by the cost of the team. Tooling removes the parts that were already easy. It does not decide what to build, it will not write your evaluation set, it will not label your documents, and it will not tell your executive which trade-off to accept. Every platform I have seen deliver real value had people around it who could have built a worse version themselves. The ones that failed bought the tool instead of the people. And where it doesn’t apply For a narrow, standard, low-risk workload with no bespoke data, buying genuinely is the right answer and building is vanity. The test is whether your data and your rules are ordinary. Usually they are not, which is why you are reading this. ### Your AI’s confidence score is a number it made up. What it costs you Every rule of the form “approve it automatically above 90%” is, right now, a rule applied to noise. Nothing in ordinary training forces a model’s stated confidence to match how often it turns out to be right. Training rewards putting the correct answer first, not being honest about how sure it is. So the number looks like a probability, behaves like a ranking, and gets used like a guarantee. Measure it before you route real decisions on it. And where it doesn’t apply Calibration is an average over a population. A model that looks well behaved overall can still be badly wrong on the one subgroup you care about, and a single headline number will hide exactly that. ### When AI answers wrongly from your documents, the search usually failed, not the AI. What it costs you Teams spend months tuning the model when the fix was one step upstream, and cheap. If the right passage was never fetched, no amount of prompt engineering recovers it. Measure how often the correct source is retrieved at all, separately from whether the final answer reads well. They are different bugs, they have different fixes, and one impersonates the other for months. And where it doesn’t apply You can only separate them if somebody labels which documents were genuinely relevant. Without those labels you are inferring search quality from the final answer, which is the exact confusion you were trying to escape. ### An evaluation you cannot fail is marketing, not evidence. What it costs you A rising score on a frozen test set is the most expensive false comfort in this business. A test set assembled from cases the system already handles measures agreement with yesterday. Build it from failures you have actually seen, keep the ones that hurt, and report the hard cases separately from the representative ones. Report the slow tail too: average response time is nobody’s experience, and the worst one in a hundred is the one your users remember. And where it doesn’t apply Adversarial test sets drift away from reality. You need a representative set and a hard one, reported separately, because averaging them produces a number that describes neither. And a worst-in-a-hundred figure measured over two hundred requests is an anecdote with a decimal point. ### The scarce person on an AI team is not the modeller. What it costs you You hire three data scientists, and eighteen months later the work is still stuck between a notebook and a production system. Building a model is now the commodity part of this work. The scarce people are the ones who carry a system across the boundary: the engineer who makes serving reliable and observable, the person who will own the labelled evaluation set and keep owning it after the launch party, and the architect who can hold a position in a security review. Teams staffed only with modellers produce excellent models that nobody deploys. And where it doesn’t apply If you have no modelling capability at all, hire that first. This is a claim about the second, third and tenth hire, and about what a working team looks like, not about where to start from zero. ### Running the model is a memory problem, not a computing-power problem. What it costs you It is why your hardware quote is wrong, and why it can be wrong in either direction. Every word the model generates requires reading the whole model back out of memory. That is why compressing the model buys more speed than buying faster chips, and why serving more users at once raises total throughput while making each individual user wait longer. The hardware decision follows from arithmetic you can do before anyone signs anything. And where it doesn’t apply True only while memory bandwidth is the bottleneck. Large batches, short inputs, or document-heavy workloads move you to the other side of the line, and every one of these intuitions inverts. ### A privacy budget is spent, not held. What it costs you Most teams size it for one release, then answer a thousand questions against the same data. Each query against a protected dataset uses up part of your privacy guarantee and does not give it back. After enough questions the guarantee is gone, even though every individual query looked perfectly safe at the time. Somebody has to keep the ledger, and it is almost never on the architecture diagram. And where it doesn’t apply The standard accounting is worst-case. A tighter method buys back real headroom, but only when the mechanism qualifies for it, which is a modelling decision rather than a configuration flag. ## What I can show you Four working instruments, not screenshots. Each computes in your browser as you move the controls, so you can check the claim instead of taking it. Read the plain-language line on each one; the machinery underneath is there for whoever you forward this to. Every figure here is verified against an independent calculation before it ships: a closed form, a hand-worked example, or a known analytic limit. A figure that cannot be reproduced does not go on the page. Figure 2 Making a confidence score mean something A model can be made honest about its own certainty by adjusting a single number, without changing any of its answers. The right-hand panel goes further: it puts a guaranteed error rate on a system even when you have no reason to trust the model at all. The mechanism, for the specialist Overconfidence ×1.90 Temperature T = 1.00 Miscoverage α = 0.10 Seed ECE 0.1465 Max bin gap 0.225 Accuracy 66.6% Mean confidence 81.3% Fitted T = 2.45 Left: observed accuracy per confidence bin against the diagonal a calibrated model would trace. Bars below the line are bins where the model claimed more than it delivered. Temperature scaling divides the logits by one constant, which moves confidence without moving a single decision. At T = 1 this is the uncorrected model. Press Fit T to see what one constant recovers. Right: split conformal prediction takes a threshold from a calibration half and applies it to a held-out half, with no assumption that the model is any good. Measured coverage 91.7% against a nominal 90%, at q̂ = 0.978. The guarantee is marginal and assumes exchangeability. Neither survives a distribution shift, and nothing in the pipeline announces when one happens. n = 2000 · seed = 7 · bins = 10 · split 50/50 calibration/test measured Figure 3 What it will cost to run, before you buy anything Before you approve hardware or sign a serving contract, this arithmetic tells you whether the model you want will even fit, and roughly how fast it will run. It is the cheapest hour of due diligence available, and it is the one most often skipped. The mechanism, for the specialist Model Device Weights Context 8,192 tok Batch 8 Weights 130.4 GB KV cache 20.0 GB 13% Resident 150.4 GB / 80 GB Bound by memory Aggregate 166 tok/s Per sequence 20.7 tok/s Prefill 9.28 s KV = weights at 53,406 tok Left: weights are flat in context; the cache is linear in it, and in batch. The shaded band is the cache, and where it crosses the capacity line the configuration stops being deployable. Right: the roofline. Decode sits far to the left of machine balance, which is why throughput tracks bytes moved rather than arithmetic. This configuration needs 150.4 GB against 80 GB of device memory, so it does not run. A dense-transformer approximation. It ignores activation memory, fragmentation, paged attention, and speculative decoding, all of which move the real number. Treat it as the floor, not the forecast. 2·P·b = 1.1T FLOP/step · 150.4 GB/step · 6.94 FLOP/byte vs balance 295 derived Figure 4 Two ways of searching your documents, side by side One method misses things it should have found. The other confidently returns documents that cannot answer the question at all, shown here in red. Neither is safe on its own, and “we use AI search” does not tell you which one you bought. The mechanism, for the specialist Your query Probes Lexical (BM25) off topic | # | id | topic | score | | --- | --- | --- | --- | | 1 | p7 | privacy | 3.016 | | 2 | s4 | systems | 2.374 | | 3 | c1 | uncertainty | 2.272 | | 4 | p6 | privacy | 2.198 | | 5 | c4 | uncertainty | 2.065 | Top result · p7 An audit trail has to record what was retrieved and why, not merely what was answered. Latent (LSA) on topic | # | id | topic | score | | --- | --- | --- | --- | | 1 | c4 | uncertainty | 0.735 | | 2 | c5 | uncertainty | 0.712 | | 3 | p7 | privacy | 0.651 | | 4 | p1 | privacy | 0.512 | | 5 | p6 | privacy | 0.467 | Top result · c4 Conformal prediction returns a set rather than a label, with a coverage guarantee that holds without assuming the model is any good. Both rankings, same corpus. The query shares no content word with its answer. Surface matching has nothing to grip and lands on the wrong topic; the latent space bridges it. This is the case dense retrieval is for. 32 documents · 290 terms · rank-8 truncated SVD retaining 31.5% of squared Frobenius norm · BM25 (k₁=1.2, b=0.75) matched 5 documents LSA is not a sentence embedding model, and a light suffix stripper is not a real stemmer. Both choices make the lexical method look worse than a tuned production index would. The disagreement pattern is the point, not the absolute scores. measured Figure 5 What a privacy guarantee actually buys Privacy protection works like a budget, and every question asked of the data spends part of it. This shows what you can still learn at a given level of protection, and the point at which the answers stop supporting any conclusion at all. The mechanism, for the specialist Privacy loss ε = 0.100 Queries k = 100 Mechanism Release Noise scale 10.00 Measured MAE 10.079 Closed form 10.000 95th pct error 30.76 Relative error 6.51% Budget spent 5.85 Left: one release against the true histogram. Press Draw again: the true counts are fixed and the released ones are not, which is the entire mechanism. Measured mean absolute error 10.079 against the closed form 10.000, over 1000 releases. At this budget the 95th-percentile error is ±30.8, against a smallest true bin of 24, so the release no longer supports conclusions about that bin. Right: answering 100 queries costs 10.0 under basic composition, or 5.9 under the advanced bound. Sensitivity 1 assumes one record moves one bin by one. Group privacy, unbounded contributions, or a join upstream all break that assumption, and the noise calibrated here would then be too small. measured ## Questions I get asked The things people actually want to know before they make contact, answered directly. Everything below is stated elsewhere on this page as well. ### Who is Ian Rudd? Ian Rudd is a chief enterprise AI architect based in Canada, with a PhD in machine learning and AI and close to twenty years spent taking artificial intelligence out of the research lab and into production. He sets the architecture for enterprise AI platforms and leads the data science, data architecture and engineering teams that build to it. He has delivered in six industries: the federal government of Canada, public sector insurance, transport and logistics, banking and financial services, retail and consumer goods, and corporate AI research at IBM and Microsoft. He holds three granted patents, has published research and given seminars on deep learning and AI safety, and works in both of Canada’s official languages. ### What does a chief AI architect actually do? Three jobs at once. The first is architecture: deciding what gets built, setting the reference designs, security patterns and landing zones, and writing down the review gates a solution clears before it goes anywhere near production. The second is leadership: building and running the data science, data architecture and engineering teams, and the practice around them, including who owns what when something breaks. The third is the one that gets left out of job descriptions, which is being accountable for the result in front of an executive, a regulator or an auditor months after launch. Most AI programmes fail on the second and third of those, not on the model. ### What do organisations bring Ian Rudd in to do? Usually one of four things. Setting enterprise AI and data strategy, and the architecture underneath it. Leading a data science, data architecture or research team, or building that practice from nothing. Model governance, uncertainty and evaluation design, so that automated decisions can be defended after the fact rather than only demonstrated in advance. Or a second opinion on a system that has to be right, typically where a pilot impressed in a demo and nobody can say whether it survives contact with security, procurement or a second team. If you are not certain which of these you need, describing the symptom is usually enough to tell. ### Which industries has Ian Rudd worked in? Six: the federal government of Canada, public sector insurance, transport and logistics, banking and financial services, retail and consumer goods, and AI and machine learning research at IBM and Microsoft. Systems he designed check more than a million payments and claims a day for fraud. The constraint regimes differ more than the technology does. Government work means protected information, service in both official languages, and an audit trail behind every answer. Insurance means automated decisions that affect whether a person gets paid, every one of them open to appeal. Transport means forecasts a national network plans against every day. ### Does Ian Rudd work in French as well as English? Yes, in both official languages, to the same standard rather than one translated after the fact. That matters for Government of Canada work, where equal service in English and French is a legal requirement rather than a courtesy, and for any organisation operating in Quebec as well as the rest of Canada. This site is published in full in both languages, and the French version is a real document at its own address rather than a machine translation of this one. ### What makes an AI system defensible? That someone can reconstruct, months later, why the system did what it did, and show the reasoning was sound at the time. In practice that takes four things: a stated confidence that matches how often the system actually turns out to be right, an evaluation the system is genuinely capable of failing, a full audit trail behind every answer given, and a named person accountable when quality drifts. A system missing all four can still work. It just cannot be defended when somebody asks, and in a regulated setting somebody eventually asks. ### Why do AI confidence scores need calibration? Because nothing in ordinary training forces a model’s stated confidence to match how often it turns out to be right. Training rewards putting the correct answer first, not being honest about how sure it is. So the number looks like a probability, behaves like a ranking, and then gets used like a guarantee. Every rule of the form “approve it automatically above 90%” is, until that score has been measured against real outcomes, a rule applied to noise. Calibration does not make the model smarter and does not reorder a single answer. It makes the number mean what it says, which is what lets you set the threshold on purpose instead of by accident. ### How do I get in touch with Ian Rudd? By email at ianrudd.phd@gmail.com, or through LinkedIn. He is based in Canada and works with teams across time zones, holds Government of Canada Enhanced Reliability and Secret clearance, and holds a PhD in machine learning and AI alongside Azure Solutions Architect Expert, PMP, CISM, COBIT and TOGAF certifications. ## Let's talk! Enterprise AI and data strategy and architecture, leading a data science, data architecture or research team, model governance, uncertainty and evaluation design, or a second opinion on a system that has to be right. If you are not certain what you need, describe the symptom. That is usually enough to tell. Connect on LinkedIn ianrudd.phd@gmail.com Security clearance Government of Canada Enhanced Reliability, Secret Languages English and French Based in Canada, working with teams across time zones Credentials PhD in Machine Learning & AI. Azure Solutions Architect Expert, PMP, CISM, COBIT, TOGAF No number on this site appears without a statement of where it came from. That is what the small coloured tags mean. Each one says how the number beside it was obtained: computed here in front of you, derived by arithmetic printed on the figure, or taken from a paper listed below. - derived derived: follows from stated inputs by arithmetic given on the figure - measured measured: computed in your browser from a seed you can change - cited cited: attributed to a source listed under the papers - estimated estimated: an approximation whose error is stated - illustrative illustrative: chosen to demonstrate a shape, not to report a result The papers behind the figures Each note says what the paper settles and what it leaves open, because the second part is where the engineering happens. All are real, citable, and worth the afternoon. Guo, Pleiss, Sun & Weinberger (2017). On Calibration of Modern Neural Networks. ICML. Establishes that accuracy and calibration came apart as networks got deeper, and that a single temperature fixes most of it. Leaves open everything about calibration under shift, which is the case that matters in deployment. Angelopoulos & Bates (2021). A Gentle Introduction to Conformal Prediction. arXiv:2107.07511. The clearest route into distribution-free coverage. Read it for what the guarantee is not: marginal rather than conditional, and void the moment exchangeability fails. Ovadia et al. (2019). Can You Trust Your Model’s Uncertainty? NeurIPS. The empirical answer is mostly no, and it degrades smoothly with shift. The most useful negative result in the area; it is why I measure rather than assume. Robertson & Zaragoza (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Where the ranking function in Figure 4 comes from, with the derivation rather than the recipe. Still the correct first baseline, and still beating things that cost a thousand times more. Deerwester et al. (1990). Indexing by Latent Semantic Analysis. JASIS. The original argument that co-occurrence structure carries meaning. Its failure mode, confident neighbours that are not neighbours, is exactly what modern dense retrieval reproduces at scale. Dwork & Roth (2014). The Algorithmic Foundations of Differential Privacy. The standard reference for the mechanisms in Figure 5. The composition chapters are the ones teams skip and then need. Narayanan & Shmatikov (2008). Robust De-anonymization of Large Sparse Datasets. IEEE S&P. Why removing identifiers is not anonymisation. Twenty years on it is still the fastest way to end an argument about whether a dataset is safe to release. Williams, Waterman & Patterson (2009). Roofline: An Insightful Visual Performance Model. CACM. The model underneath Figure 3. Predates the current hardware by fifteen years and still explains why decode is slow. Pope et al. (2023). Efficiently Scaling Transformer Inference. MLSys. Makes the memory-bound argument concretely, with the partitioning arithmetic worked through. The paper to hand someone who thinks throughput is a FLOPs question. Sculley et al. (2015). Hidden Technical Debt in Machine Learning Systems. NIPS. Ten years old and still the most accurate description of why these systems decay. Nothing in it is about modelling, which is the point. This website is built by Ian Rudd, to contribute to the public understanding of AI and to demonstrate the architecture and governance practices that make it safe. The data is open and the papers are cited. If you find a number that does not match its provenance, please let me know.