Research Analysis Methods
Analytical principles for examining research evidence, comparing observations, assessing uncertainty and communicating defensible findings in SEO, GEO and AI search research.
Executive Summary
Research analysis is the disciplined process of transforming observations and source material into findings that answer a defined question. The process requires more than calculation: researchers must choose suitable methods, examine assumptions, consider alternative explanations and distinguish what evidence demonstrates from what remains uncertain.
These responsibilities are particularly important in SEO, generative engine optimisation and AI search research. Search systems change frequently, generated answers may vary between observations, and the internal mechanisms of proprietary platforms are often inaccessible. An apparent relationship between visibility, citations and source characteristics may therefore support a useful hypothesis without proving a causal mechanism.
This document describes quantitative, qualitative, comparative and mixed-method approaches; the treatment of uncertainty, bias and confounding; and principles for synthesis and responsible interpretation. It is methodological guidance rather than a claim that every historical CGO Media publication has used identical analytical procedures.
It is the third document in the CGO Media Research Methodology series, following Research Scope & Questions and Research Data & Evidence Standards.
Core Principles of Research Analysis
Analytical rigour requires alignment between the research question, the evidence and the strength of the eventual claim. The following principles provide a common reference across different publication types.
Question-led analysis
Select methods according to the research question and the evidence available.
Clear definitions
Identify units, outcomes, denominators and categories before interpreting results.
Appropriate inference
Separate description, association, prediction and causation.
Bias awareness
Examine sampling, confounding, measurement error and missing information.
Transparent decisions
Record material analytical choices, changes and sensitivity checks.
Proportionate claims
State conclusions at the level justified by the design and evidence.
1. Analytical Design and Research Questions
1.1 Aligning analysis with the question
Analysis begins with the claim a study is equipped to examine. A descriptive question about citation prevalence requires a different method from a comparative question about sectors or an explanatory question about why sources are selected. The unit of analysis, population, observation period and proposed outcome should be established before calculations are chosen. This relationship is developed in Research Scope & Questions.
A study asking how often organisations appear in generated answers might count answers containing a defined type of mention. A study asking whether mentions differ by sector requires comparable query sets, denominators and conditions. Neither design alone establishes that an optimisation activity caused the difference.
1.2 Analysis plans and decision rules
An analysis plan can record primary outcomes, comparisons, exclusions, coding definitions and methods for addressing missing information. It should distinguish planned confirmatory analyses from exploratory work identified after examining the data. Material deviations should be documented rather than disguised as decisions made in advance.
An analysis plan is not a guarantee that results will be significant or favourable. Its purpose is to make the relationship between questions, methods and claims inspectable. Researchers should identify when a question cannot be answered with available evidence.
1.3 Levels of inference
Descriptive inference summarises observations; statistical inference considers what a sample may suggest about a population; causal inference evaluates whether an intervention produces an outcome. Each requires different assumptions. Researchers should avoid moving from one level to another without evidence.
An association between source characteristics and citations may justify further investigation but does not demonstrate the internal weighting of a proprietary AI model. Similarly, a rise in referral traffic after a website change does not isolate the change from seasonality or other events.
1.4 Analytical transparency
Analytical transparency involves reporting what was measured, how calculations were performed, which decisions affected inclusion and what limitations remain. A complex method does not compensate for poor source quality or an unsuitable research question.
The Research Data & Evidence Standards companion page addresses the provenance and reliability of material entering the analysis. Conclusions should be traceable to those inputs and the decisions applied to them.
2. Quantitative Research Analysis
2.1 Descriptive statistics and denominators
Counts, proportions, rates, medians and distributions can summarise research observations. Every percentage requires a defensible denominator: the number of answers, queries, sessions, citations or domains may yield very different results. Denominators should be defined explicitly, especially when one answer can contain several citations.
For example, the share of answers with at least one citation is not the same as the average number of citations per answer. Reporting both may be useful, but they answer distinct questions. Researchers should avoid presenting either as a direct measure of trust or commercial conversion.
2.2 Distributions and outliers
Averages can conceal skew, concentration and unusual cases. A small group of highly cited domains may account for a large share of observed references, making a simple mean unrepresentative of typical sources. Medians, ranges and distributional descriptions can provide a more complete picture.
Outliers should be investigated rather than automatically discarded. An unusual observation may reflect a collection error, an important subgroup or genuine system behaviour. Exclusion rules should follow documented reasons, not the desired direction of a result.
2.3 Statistical uncertainty
Samples are subject to variability. Confidence intervals and other uncertainty estimates can be useful when their assumptions are justified by the sampling design and measurement process. They should not be attached mechanically to convenience samples as if they established population representativeness.
A narrow numerical interval can still be misleading if the underlying selection process is biased. Statistical precision and external validity are different matters. Researchers should describe uncertainty arising from measurement, platform changes and incomplete coverage as well as sampling.
2.4 Effect sizes and practical importance
A statistically detectable difference may be too small to matter in practice, while a potentially important effect may remain uncertain because evidence is limited. Effect sizes, baseline levels and operational context help readers evaluate relevance.
A change in citation frequency should be interpreted alongside the original rate, sample construction, variation across queries and possible confounding factors. The significance of a finding should not be judged solely by whether a threshold has been crossed.
2.5 Multiple comparisons and exploratory findings
Testing many relationships increases the chance of apparently notable results occurring by chance. Researchers should distinguish planned comparisons from exploratory searches and consider appropriate multiplicity controls where statistical testing is used.
Exploratory patterns can generate valuable hypotheses. They should be labelled as exploratory and, where feasible, examined with fresh data before being presented as robust general findings.
3. Qualitative Research Analysis
3.1 Purpose of qualitative analysis
Qualitative methods examine meaning, context, structure and process. In AI search research, they can help interpret answer construction, citation placement, wording of recommendations and the way an organisation is represented. They complement numerical summaries by exploring distinctions that counts alone may miss.
A qualitative account should describe the material selected and the basis for interpretation. An illustrative answer is not evidence that every answer on a platform follows the same pattern.
3.2 Coding and category development
Coding assigns defined labels to observations. Categories may be developed deductively from a research question or inductively through systematic examination of material. Researchers should explain category definitions and how ambiguous cases are handled.
For example, a generated answer might contain a direct recommendation, a neutral mention, an attributed quotation or a linked citation. These are analytically distinct events and should not be collapsed into one visibility category without a reason.
3.3 Consistency and researcher judgement
Qualitative analysis inevitably involves judgement. Consistent definitions, example cases and documented decisions can reduce arbitrary variation. When multiple coders are involved, disagreement can reveal unclear categories that require refinement.
Agreement statistics may be appropriate for some coding tasks, but high agreement alone does not prove that categories measure the intended concept. Interpretive quality also depends on contextual understanding and clear reporting.
3.4 Negative cases and competing interpretations
Researchers should look for examples that challenge an emerging theme. Negative cases can reveal boundary conditions, overlooked categories or an explanation that is too broad. They should not be omitted merely because they complicate a narrative.
An organisation that appears prominently despite lacking an expected attribute may suggest that the proposed relationship is incomplete. The responsible conclusion may be to refine the model rather than dismiss the observation.
3.5 Thematic synthesis
Thematic synthesis groups recurring observations into broader patterns while preserving distinctions between source material and researcher interpretation. Themes should be supported by sufficient examples and should not imply numerical prevalence unless that prevalence has been measured.
A theme such as preference for institutional sources may be plausible in a selected corpus, but its generality depends on sampling, platform, query intent and the definitions used.
Research principle: A persuasive explanation is not a substitute for a suitable research design and evidence capable of supporting the claim.
4. Comparative and Longitudinal Analysis
4.1 Constructing meaningful comparisons
Comparisons require groups that are sufficiently alike on relevant dimensions. Differences in language, query intent, sector, brand familiarity or platform configuration can distort results if attributed to one focal variable.
For instance, comparing branded navigational prompts with generic informational prompts could make source citation behaviour appear to differ by industry when query type is the more plausible explanation.
4.2 Baselines and controls
A baseline provides a reference condition. Controls can help isolate factors under investigation, although true experimental control is difficult in open search environments. Researchers should describe the rationale for selecting reference periods or comparison groups.
A website performance before and after a technical change may be affected by unrelated platform updates, competitor activity and demand shifts. A before-and-after difference is not, by itself, proof of causation.
4.3 Longitudinal measurement
Repeated observations can reveal change, persistence or volatility. Longitudinal studies should preserve stable measurement definitions and document platform or interface changes that threaten comparability.
If a generative search service alters how citations are displayed, a change in observed link counts may reflect a presentation update rather than a change in source selection. Methodological discontinuities should be recorded.
4.4 Cohorts and segmentation
Segmentation can reveal variation hidden in overall averages. Useful groups may be defined by query intent, industry, language, source type or organisational scale. Groups should be meaningful and defined transparently.
Very small subgroups can produce unstable estimates, while post hoc selection of favourable segments can exaggerate apparent patterns. Researchers should communicate subgroup sample sizes and exploratory status.
4.5 Sensitivity analysis
Sensitivity analysis examines whether conclusions change under reasonable alternative assumptions, coding decisions or inclusion rules. It can expose findings that depend heavily on one disputed definition.
For example, researchers might compare results when unlinked mentions are included or excluded from a visibility measure. If the conclusion reverses, that dependence is an important finding in its own right.
5. Bias, Confounding and Uncertainty
5.1 Selection and survivorship bias
Selection bias occurs when included observations differ systematically from the target population. Survivorship bias arises when only visible, accessible or successful cases are examined. Both can distort conclusions about search and AI discovery.
A study of highly cited organisations cannot establish what causes visibility across all organisations if it omits those never cited. The evidence standards should guide evaluation of the sample and its provenance.
5.2 Measurement and classification error
Measurement error may arise from inconsistent collection settings, missing responses, interface changes or coding ambiguity. Errors may be random or systematic. Researchers should consider how these affect observed rates and relationships.
A parser that misses citations displayed in a new interface may generate a false decline. Quality checks should therefore consider both the collection mechanism and the meaning of the recorded variable.
5.3 Confounding variables
A confounder is a factor associated with both the explanatory variable and the outcome. Brand recognition, source size, topic, language and query intent may all complicate interpretation of AI citation patterns.
Adjustment can reduce some confounding when relevant factors are measured accurately, but no statistical model can guarantee removal of unobserved confounding. Causal language should remain proportionate to the design.
5.4 Model assumptions and limitations
Analytical models rely on assumptions about data structure, independence, distributions or relationships. Repeated answers from the same query or domain may be correlated, making naive independence assumptions inappropriate.
Researchers should explain material assumptions and, where feasible, assess how sensitive results are to them. Complex modelling should not create an illusion of certainty unsupported by the underlying evidence.
5.5 Communicating uncertainty
Uncertainty includes statistical variability, methodological limitations and uncertainty about how closed systems operate. Readers should be told which conclusions are supported, which remain tentative and which cannot be inferred.
The companion Research Limitations & Transparency document provides a fuller account of disclosure, corrections and responsible qualification.
Research principle: Analytical uncertainty should be communicated explicitly rather than concealed behind apparently precise figures or confident language.
6. Triangulation and Evidence Synthesis
6.1 Combining evidence types
Triangulation examines a question using different methods, sources or perspectives. Quantitative citation counts, qualitative answer analysis and technical source checks may each contribute distinct information. Their convergence can strengthen an interpretation but does not automatically prove it.
Evidence types should be assessed independently before synthesis. Multiple weak or dependent sources do not become reliable simply because they appear to agree.
6.2 Literature synthesis
Research synthesis requires attention to source selection, study design, populations, publication dates and methodological differences. A narrative review may provide valuable interpretation but should not be described as systematic unless its search and screening methods justify that designation.
When quantitative results from different studies are compared, researchers should check whether the same concepts, denominators and populations were measured. Combining incompatible percentages can create false precision.
6.3 Contradictory evidence
Conflicting results should be treated as an analytical problem, not an inconvenience. Differences may arise from study design, sampling, platform changes, terminology or genuine contextual variation.
A useful synthesis explains where findings agree, where they diverge and what further evidence would help resolve the disagreement. Selective citation of only supportive studies weakens research credibility.
6.4 Evidence hierarchies and fit for purpose
No single evidence hierarchy applies equally to every research question. Controlled experiments may be especially useful for some causal questions, while technical specifications, qualitative observations or administrative records may be more appropriate for others.
The central question is whether a source and method are suitable for the proposition being made. Repository publication, citation volume or institutional status alone does not establish that suitability.
6.5 From synthesis to conceptual models
A synthesis may inform a framework that organises constructs and relationships. The framework should identify which elements rest on empirical findings, which are reasoned interpretations and which remain hypotheses.
The Research Frameworks & Models methodology page addresses this distinction, while the Framework Library presents related CGO Media models.
7. Applications to SEO, GEO and AI Search
7.1 Technical SEO analysis
Technical SEO research may combine crawling, indexing, rendering, structured-data inspection and search performance evidence. These signals describe different aspects of discoverability and should not be treated as interchangeable.
Successful crawling does not guarantee indexing; indexing does not guarantee ranking; ranking does not guarantee conversion. Analysis should preserve these distinctions and account for measurement limitations.
7.2 AI citation and source selection
Citation research should define the unit of analysis and distinguish linked citations, unlinked references and recommendations. Repeated queries may be needed to characterise variability, while results should be reported within the relevant platform and collection period.
A citation does not establish that a source is accurate, trusted by users or causally responsible for a generated answer. Related themes are explored in the Research Library and Research Observations Library.
7.3 Entity and brand analysis
Entity research may examine naming consistency, source relationships and representation in generated responses. Brand analysis may examine mentions, recommendations and the contexts in which organisations appear.
These indicators can support descriptive or comparative analysis, but claims about proprietary model weights or internal authority calculations require evidence not normally available from public outputs alone. See the Entity Authority Framework and Brand Signal Framework.
7.4 Sector-specific interpretation
Sector conditions shape both data collection and interpretation. Healthcare, legal, financial services, travel and ecommerce queries differ in informational stakes, intent and source ecosystems. A single pooled estimate may conceal important variation.
The Sector Research programme provides a context for sector-specific questions. Comparisons should account for sector boundaries rather than assuming that findings transfer unchanged.
7.5 Connecting visibility to outcomes
Visibility indicators, referral traffic, engagement and commercial outcomes are different measurements. A change in AI mentions cannot automatically be converted into a revenue estimate without a defensible attribution model and relevant evidence.
The CGO Visibility Framework provides a conceptual structure for discussing visibility; empirical business impact requires separate measurement and validation.
8. Interpretation, Reporting and Governance
8.1 Evidence-proportionate conclusions
A research conclusion should answer the original question at the level justified by the data and design. Descriptive findings should not be rewritten as causal mechanisms, and exploratory relationships should not be presented as validated laws.
Where evidence is mixed or incomplete, the conclusion should explain the uncertainty. A null or inconclusive result may still improve understanding of measurement limits or suggest better future research.
8.2 Clear reporting of methods
Readers should be able to identify the population, observation period, data sources, coding rules, analyses, exclusions and important limitations. Where sharing underlying data is restricted, the reason and any available alternatives should be stated.
Methods descriptions should reflect what was actually done. Proposed quality checks should not be represented as completed audits, and repository publication should not be described as independent peer review without evidence.
8.3 Review, correction and revision
Errors may be discovered after publication, and changing platforms can alter the relevance of earlier conclusions. Material corrections should identify what changed and whether the original interpretation is affected.
A revised result should not silently replace an earlier claim when the difference matters to readers. The Ethics & Integrity page provides the wider principles for accountability.
8.4 Responsible use of AI-assisted analysis
AI tools may assist with organising records, suggesting codes, drafting summaries or writing analytical code. Their outputs require appropriate verification because generated explanations, references and calculations may be incorrect.
Human responsibility for methodological choices and published conclusions remains essential. AI assistance should not be treated as independent corroboration of the same material supplied to the tool.
8.5 Integration with the research programme
CGO Media research papers, observations, statistics and frameworks address different questions and evidence types. Analytical methods should be selected to suit each publication rather than forcing all outputs into a single format.
The Research Architecture and Research Methodology provide the organisational context. The aim is a coherent research programme whose individual claims remain open to scrutiny.
Research principle: Published findings should remain traceable to their evidence and should distinguish established observations from interpretations and hypotheses.
Practical Research Analysis Workflow
This workflow is a recommended methodological sequence, not a representation that every previously published study completed the same steps.
01. Define the question
Specify the target population, unit of analysis and intended inference.
02. Inspect evidence
Assess provenance, completeness, coding and collection limitations.
03. Choose methods
Select descriptive, comparative, qualitative or mixed approaches that fit the question.
04. Check assumptions
Examine independence, confounding, measurement reliability and missingness.
05. Test robustness
Consider negative cases and reasonable alternative definitions or exclusions.
06. Report proportionately
Separate observations, interpretations, limitations and proposed future research.
Explore the Research Methodology Collection
The five companion documents explain how research questions, evidence, analysis, conceptual models and transparency connect within the Research Architecture.
Conclusion
Research analysis is credible when its methods correspond to the question, its measurements represent clearly defined concepts and its conclusions remain proportionate to the evidence. Counts, comparisons, themes and statistical models can all contribute knowledge, but none independently establishes causal mechanisms or universal behaviour.
In AI search research, particular care is required because platforms are dynamic, observation conditions may be difficult to reproduce and internal retrieval or recommendation mechanisms are often inaccessible. Analytical transparency therefore includes reporting what was measured, how it was interpreted and which explanations remain unresolved.
CGO Media Research Team presents these principles as a methodological reference for evaluating search evidence, developing analytical models and communicating research findings responsibly.
References and Methodological Foundations
The following academic publications provide methodological context. Their inclusion does not establish that every procedure they describe has been implemented in each CGO Media study.
- National Academies of Sciences, Engineering, and Medicine (2019). Reproducibility and Replicability in Science. National Academies Press. https://doi.org/10.17226/25303
- Wasserstein, R. L., Schirm, A. L., and Lazar, N. A. (2019). Moving to a World Beyond p < 0.05. The American Statistician, 73(sup1), 1–19. https://doi.org/10.1080/00031305.2019.1583913
- Munafò, M. R. et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
- Page, M. J. et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
- von Elm, E. et al. (2007). The STROBE statement: guidelines for reporting observational studies. PLoS Medicine, 4(10), e296. https://doi.org/10.1371/journal.pmed.0040296
- Wilkinson, M. D. et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18
Research Analysis Methods: 24 Frequently Asked Questions
The first ten questions address the most important methodological distinctions. Additional questions cover technical analysis, bias, synthesis and governance.
1. What are research analysis methods?
Research analysis methods are procedures for organising, examining and interpreting evidence in relation to a defined question. They may be quantitative, qualitative, comparative or mixed. The Research Methodology explains their place in the wider programme.
2. How does CGO Media select an analytical method?
A method should fit the research question, available evidence, unit of analysis and level of inference. A descriptive citation count does not require the same design as an investigation of causation. See Research Scope & Questions.
3. What is the difference between quantitative and qualitative analysis?
Quantitative analysis examines numerical patterns, rates or relationships. Qualitative analysis examines meaning, context, categories and processes. The methods can complement one another but answer different aspects of a question.
4. Can citation frequency establish AI recommendation authority?
Not by itself. Citation frequency is one observable measure and may differ from recommendations, user trust and business outcomes. Claims about authority require clear definitions and supporting evidence.
5. What is a unit of analysis?
The unit of analysis is what is counted or examined, such as an answer, query, citation, source domain or organisation. Confusing units can create misleading percentages and comparisons.
6. Why do denominators matter?
A percentage depends on what forms its denominator. The proportion of answers containing a citation differs from the share of all citations attributable to a domain. Researchers should state the denominator explicitly.
7. What is statistical uncertainty?
Statistical uncertainty reflects variability in estimates and depends on the sampling and analytical design. Confidence intervals do not correct biased sampling or guarantee population representativeness.
8. Does correlation prove causation in SEO research?
No. Correlation may arise from confounding factors, selection effects or coincidence. Causal claims require stronger designs and examination of alternative explanations.
9. How should contradictory findings be handled?
Researchers should examine differences in methods, populations, timing and definitions rather than selecting only favourable evidence. Unresolved conflicts should be disclosed in the interpretation.
10. How are inconclusive findings reported?
They should be described as inconclusive, with the reasons and evidence limitations explained. Such findings may still identify measurement weaknesses or worthwhile future questions.
11. What is descriptive analysis?
Descriptive analysis summarises observed data through counts, proportions, distributions and other appropriate measures. It describes what was recorded without necessarily explaining why it occurred.
12. What is qualitative coding?
Qualitative coding assigns defined categories to text or other observations. Clear definitions and treatment of ambiguous cases improve consistency and interpretability.
13. What is thematic analysis?
Thematic analysis identifies and examines recurring patterns in qualitative material. A recurring theme should not be treated as a population-wide frequency unless prevalence was separately measured.
14. What is a baseline in research?
A baseline is a reference condition or earlier observation used to interpret change. Its comparability to the study observations is essential.
15. Why are repeated AI search observations useful?
Generative outputs can vary between sessions and over time. Repeated observations can help characterise variability, though exact reproducibility may be limited by proprietary systems.
16. What is sensitivity analysis?
Sensitivity analysis checks whether findings change under reasonable alternative assumptions, exclusions or coding rules. It helps identify conclusions that depend heavily on one methodological choice.
17. What is selection bias?
Selection bias occurs when included observations differ systematically from the intended population. The Research Data & Evidence Standards page discusses source selection and provenance.
18. What is confounding?
Confounding occurs when another factor is related to both the variable of interest and the outcome, complicating interpretation. Statistical adjustment may help but cannot eliminate all unmeasured confounding.
19. What is triangulation?
Triangulation examines a question through different sources or methods. Agreement can strengthen an interpretation, but dependent or weak sources should not be counted as independent confirmation.
20. How are research observations different from statistics?
Observations may document emerging patterns within limited conditions, while statistical publications may estimate numerical characteristics of defined populations. See the Research Observations Library.
21. How do analytical methods support research frameworks?
They help assess relationships, identify limits and distinguish empirical support from conceptual assumptions. See Research Frameworks & Models.
22. Can AI tools perform research analysis independently?
AI tools can assist with analysis, coding or summarisation, but generated results and references require verification. Responsibility for methods and published claims remains with the research authors.
23. How should research limitations be disclosed?
Material limitations involving sampling, measurement, unavailable evidence and changing platforms should accompany the interpretation. See Research Limitations & Transparency.
24. Who authored Research Analysis Methods?
The document is attributed to the CGO Media Research Team. Attribution does not imply external peer review or empirical validation of every example unless separately documented.
Research Attribution and Publication Information
Document: Research Analysis Methods
Research author: CGO Media Research Team
Publisher: CGO Media
Research programme: CGO Media Research Methodology
Subject areas: Quantitative research, qualitative analysis, comparative methods, research uncertainty, SEO, GEO and AI search.
Canonical publication: https://cgomedia.com/research-analysis-methods/
Research governance: CGO Media Ethics & Integrity
Publication date: 22 October 2026