Analyze Data by Checking How It Was Collected and Ruling Out Other Explanations

## Start from the Question and Learn How the Data Was Collected Begin with a concrete question or hypothesis, not a preferred tool or technique. State the decision the analysis should inform and what evidence could change that decision. Before calculating metrics, document: - the population, event, or process represented by the data; - how, where, and when observations were collected; - experiment assignment, instrumentation, and eligibility rules; - known gaps, delayed records, sampling, and retention limits; - unusual events during the measurement window; - the units of analysis and any dependencies between observations. Inspect the product, process, or experiment directly when possible. If direct use is unavailable, review trustworthy behavior descriptions and collection specifications. Do not assume a table represents the intended phenomenon merely because its fields have plausible names. Record the initial question, assumptions, and expected invariants. Allow the question to evolve during exploration, but distinguish revised questions from the original one. ## Check the Data First, Then Keep Describing Results Separate from Judging Them Keep three analytical stages distinct even when work moves iteratively among them. 1. Validation asks whether the data is internally consistent, correctly collected, and representative of the intended process. 2. Description states what the measurements show without assigning value or benefit. 3. Evaluation judges whether the described change is desirable for the relevant people, system, or objective. Label claims by stage in notes and reports, and resolve validation failures before relying on descriptive results. Use neutral descriptive language for direction, magnitude, uncertainty, and population. Make evaluation criteria explicit instead of embedding them in metric names. Before answering the main question, check variables expected to remain stable: - population size and composition; - assignment or exposure counts; - event and query volumes; - logging and error rates; - missingness and latency; - unaffected reference metrics. Do not interpret a change in the target metric until plausible measurement or population changes have been ruled out. ## Look Beyond Averages at the Spread and Raw Examples Do not rely only on averages or other summary statistics. Examine representations that reveal shape, spread, tails, multiple modes, and discontinuities. Choose suitable histograms, quantiles, cumulative distributions, or other distribution views. Attach uncertainty to estimates using methods appropriate to the sampling and dependence structure. Large datasets do not eliminate randomness, bias, or measurement error. Distinguish a precise estimate of the wrong quantity from a valid estimate of the intended quantity. Inspect raw examples whenever new analysis logic transforms or classifies data. Sample from: - every important class; - common and rare categories; - central and extreme portions of numeric distributions; - successful, failed, missing, and malformed cases; - records near decision thresholds. Investigate outliers before excluding or pooling them; they may reveal instrumentation defects, uncounted behavior, invalid assumptions, or genuinely unusual cases. If outliers are filtered or grouped, record the rule, count, rationale, and effect on conclusions. Limit investigation only after the remaining uncertainty is immaterial to the decision. ## Split Results into Groups and Time Periods to See If They Hold Break aggregate results into subgroups when the underlying behavior may differ across populations or environments. Select slices from domain knowledge, collection boundaries, and plausible failure modes rather than searching indefinitely for striking results. For each important slice, examine: - metric value and uncertainty; - sample size and missingness; - relative share of the comparison groups; - data quality and interaction differences; - consistency with the aggregate result. When comparing groups or periods, check for composition shifts; a changed mixture of subgroups can reverse or distort aggregate conclusions even when within-slice behavior is stable. Do not condition on variables changed by the treatment or system feedback unless the resulting bias is explicitly addressed. Slice by time to detect launches, outages, seasonality, logging changes, and gradual drift. Do not discard an unusual period solely because it is unusual; seek a causal explanation first. Test stability across multiple time windows, populations, and reasonable random subsamples. Separate statistical significance from practical significance by evaluating effect size, uncertainty, affected population, decision cost, and the consequences of acting or not acting. Absence of statistical significance is not evidence of no meaningful effect. ## Write Down Metric Definitions and What Each Filter Removed Define every metric so another analyst can reproduce its population and calculation. For a ratio, name the numerator, denominator, unit, time window, and eligibility criteria. If a metric exists only after an event occurs, state that conditioning explicitly. Track filtering as part of the analysis rather than as invisible preprocessing. At every filtering step, record: - the rule and its purpose; - rows, entities, or events before and after; - the share removed; - overlap with earlier filters; - whether removal differs across comparison groups; - sensitivity of the result to the rule. Where feasible, compute diagnostic metrics for excluded and included populations. Do not hide difficult-to-parse, unknown, or inconvenient cases without quantifying them. Start with established metrics before introducing custom ones. If standard and custom metrics disagree, investigate definitions, logging, and filtering before accepting the novel measure. Apply a new metric first to historical data or a well-understood case where its expected behavior can be checked. Do not evaluate a system change with a metric that the same change directly feeds back into or manipulates. Use an independent outcome or a design that accounts for the feedback loop. ## Measure Important Results a Second Way and Make Sure They Repeat Measure important phenomena in more than one defensible way. Prefer cross-checks that differ in implementation, data source, sampling method, or operational definition. Compare results with: - established reference metrics; - prior measurements on related populations; - independent data sources; - alternative implementations; - repeated runs on fixed inputs; - nearby time ranges and reasonable subsamples. Exact agreement is not required when populations or methods differ, but material discrepancies require explanation. Treat surprising results as unverified until measurement error and known confounders have been investigated. For models and derived estimates, perturb inputs within realistic bounds and check whether conclusions remain stable. Verify that repeated execution produces the same result when determinism is expected. Preserve analysis parameters, data versions, filtering counts, and code revisions needed for reproduction. Run standard validation first, then add custom checks motivated by the specific question. Do not use a single successful computation as proof that the result is reproducible. ## Test Other Explanations Before Claiming One Thing Caused Another Convert observed anomalies and trends into explicit hypotheses. For each hypothesis, write down evidence that would support it and evidence that would weaken or falsify it. Search for alternative explanations involving: - selection and assignment; - logging or instrumentation changes; - seasonality and external events; - population composition; - multiple comparisons; - reverse causality; - hidden common causes; - system feedback. Use comparisons, natural variation, controlled experiments, or external evidence appropriate to the claim. Check whether the effect appears where the proposed mechanism predicts and is absent where it should not apply. Compare the observed magnitude with the plausible reach and timing of the mechanism. Correlation alone does not establish causation. Do not present a proxy relationship as stable without a theory and evidence explaining why it should persist. State clearly what the design can establish, what it cannot establish, and which assumptions a causal interpretation requires. Actively seek disconfirming evidence, especially when stakeholders prefer a particular answer. Revise or abandon the explanation when contradictory evidence is stronger. ## Get a Peer Review and Report the Limits Along with the Findings For exploratory work, complete an end-to-end analysis early enough to learn from the whole pipeline. Record rough edges, exclusions, and unresolved cases, then improve the stages that materially affect the conclusion. Do not perfect the first transformation while leaving the final decision logic untested. Seek peer review before presenting conclusions to decision-makers. Ask reviewers to challenge: - collection and population assumptions; - metric and filter definitions; - uncertainty calculations; - slice selection and composition changes; - causal language; - reproducibility; - practical relevance. In the final report, distinguish observed facts, interpretations, evaluations, and recommendations. Include the question, scope, data window, metric definitions, effect sizes, uncertainty, exclusions, sensitivity checks, and unresolved limitations. Report null, contradictory, and unsupported findings alongside favorable ones. Use tools suited to the analytical problem rather than forcing every question through a familiar method. State ignorance and mistakes promptly; unsupported certainty weakens future decisions. End with the decision-relevant conclusion, the evidence behind it, the main caveats, and the next check that would most reduce uncertainty.