---
title: "Getting started with statcanR"
author: "Thierry Warin"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with statcanR}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = FALSE
)
```

## Overview

`statcanR` connects R to [Statistics Canada's Web Data Service (WDS)](https://www.statcan.gc.ca/en/developers/wds). This vignette walks through a complete workflow: describing the data you need, choosing a table, understanding its identifier, downloading it, inspecting the result, and optionally saving a CSV copy.

The package supports four common tasks:

1. Find likely tables from an ordinary-language description.
2. Search for exact keywords in the official table catalogue.
3. Download a complete table in English or French into R.
4. Save the downloaded table as a CSV file.

The four public functions have distinct purposes:

| Function | Use it when... | Result |
|---|---|---|
| `statcan_find()` | You can describe the subject, geography, or period you need | A ranked data frame of likely tables and reasons for each match |
| `statcan_search()` | You know words that occur in the official table title | An interactive table of exact keyword matches |
| `statcan_data()` | You want the complete table in R | A data frame |
| `statcan_download_data()` | You want the data frame and a CSV copy | A data frame with the saved file path attached |

The download functions retrieve a **complete table**, not a filtered selection of observations. A Statistics Canada table can be large. It is therefore useful to identify the right table before starting the download.

An optional fifth function, `statcan_chat()`, wraps `statcan_find()` with a user-configured language model to explain candidates and ask clarifying questions; see [Optional: ask a language model for help](#optional-ask-a-language-model-for-help).

## Two concepts to know first

### Table number and Product ID

Statistics Canada displays table numbers such as `10-10-0001-01`. The first eight digits form the WDS Product ID (PID), `10100001`; the final `01` identifies the displayed view. `statcanR` accepts either the displayed table number or the eight-digit PID:

```{r id-equivalence}
table_data <- statcan_data("10-10-0001-01", "eng")
table_data <- statcan_data("10100001", "eng")
```

These two calls request the same table.

### Language

Use `lang = "eng"` for an English table and `lang = "fra"` for a French table. The language controls the table contents and labels returned by Statistics Canada; it is not a translation performed by `statcanR`.

## Install or upgrade

The command used for a first installation also upgrades an older CRAN installation:

```{r install}
install.packages("statcanR")
```

Then load the package:

```{r setup}
library(statcanR)
```

If the package was loaded while you upgraded it, restart the R session before calling `library(statcanR)` again. Check which version R will use with:

```{r version}
packageVersion("statcanR")
```

Version 0.3.0 preserves the familiar calls from earlier releases. In particular, code that supplies a table number and a language to `statcan_data()` or `statcan_download_data()` continues to work.

## Step 1: describe the table you need

Use `statcan_find()` when you know what data you want but do not know the
official title or table number. Write a short request containing as much of the
subject, Canadian geography, and period as you know:

```{r find-eng}
matches <- statcan_find(
  "R&D expenditures in Quebec since 2020",
  lang = "eng",
  n = 5
)

matches[, c("title", "id", "score", "match_reason")]
```

This request is interpreted as:

- a **subject**: research and development expenditures;
- a **geography**: Quebec; and
- a **date requirement**: the table must include 2020.

The result is an ordinary data frame. `rank` orders the candidates, `score`
summarizes the strength of each match, and `match_reason` states which clues
were confirmed. The score helps with discovery; it is not a measure of data
quality. Read the candidate titles because two tables can represent different
valid interpretations of the same short request.

When a province or territory appears in the query, `statcan_find()` checks WDS
metadata to confirm that the table includes it. `start_date` and `end_date`
describe the coverage of the **whole table**. These checks help choose a table;
they do not select rows. The download functions still retrieve the complete
table, after which you filter its geography and reference-period columns.

French requests and French catalogue titles are supported too:

```{r find-fra}
matches_fr <- statcan_find(
  "Dépenses de R-D au Québec depuis 2020",
  lang = "fra"
)
```

The parser recognizes common wording and abbreviations, but it is deliberately
simple and predictable. A concise request generally works better than a long
question. If the results are too broad, add a more specific subject term. If
there are no results, remove a detail or try an official keyword.

### Search exact title keywords

Use `statcan_search()` when you know words that occur in the official title. It searches titles without regard to letter case:

```{r search-eng}
statcan_search(
  c("federal", "expenditures", "objectives"),
  lang = "eng"
)
```

The result is an interactive table displayed in the RStudio Viewer or a browser. The most important columns are:

- `title`: the official table title;
- `id`: the table number to pass to a download function;
- `release_date`: the catalogue release date; and
- `lang`: the language searched.

When several keywords are supplied, **all** of them must occur in the title. This makes searches precise, but it can also produce no matches. If that happens, remove one keyword or use a broader term:

```{r search-broad}
statcan_search("expenditures", lang = "eng")
```

Search French titles by using `lang = "fra"`:

```{r search-fra}
statcan_search(c("dépenses", "fédérales"), lang = "fra")
```

The catalogue is cached for 24 hours in R's platform-appropriate user cache directory. `statcan_find()` caches the table metadata used for geography checks for seven days. These caches make repeated searches faster and avoid unnecessary requests to Statistics Canada. Set `refresh = TRUE` only when you specifically need fresh information:

```{r search-refresh}
statcan_find(
  "population in Alberta since 2021",
  lang = "eng",
  refresh = TRUE
)
```

If WDS is temporarily unavailable, catalogue searches use the most recent valid cache and issue a warning. If geography metadata cannot be refreshed, `statcan_find()` uses valid cached metadata where possible. Candidates whose geography could not be checked have `geography_match = NA`; the match explanation makes that uncertainty explicit.

### Optional: ask a language model for help

`statcan_chat()` is an entirely optional layer on top of `statcan_find()`. It sends your query and `statcan_find()`'s ranked candidates to a language model you configure, which explains which candidate best matches and asks a clarifying question when the query is ambiguous. The language model never sees or chooses a table number of its own; it only explains the candidates `statcan_find()` already returned.

This feature adds no new package dependencies, and it never makes a network request unless you call `statcan_chat()` yourself. Two providers ship built in, selected with the `provider` argument: `"openai"` (the default) and `"anthropic"` (Claude). The endpoint defaults to the chosen provider and must use `https://` so the key is never sent in cleartext (plain `http://` is accepted only for a loopback host such as `http://localhost`, for a local model). Supply the API key through the provider's environment variable (`OPENAI_API_KEY` or `ANTHROPIC_API_KEY`, or the generic `STATCANR_LLM_API_KEY`, or the `api_key` argument), since a secret should not be kept in `options()`:

```{r chat-configure}
Sys.setenv(OPENAI_API_KEY = "sk-...")        # OpenAI
Sys.setenv(ANTHROPIC_API_KEY = "sk-ant-...") # Anthropic (Claude)
```

Loading the package with `library(statcanR)` never requests these settings, and no key is needed for `statcan_find()` or `statcan_data()`; the key is read only when you call `statcan_chat()`. To set it once and reuse it across sessions without retyping it, add it to your `~/.Renviron` file (open it with `usethis::edit_r_environ()`, then restart R) instead of calling `Sys.setenv()` each time:

```
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
```

R reads `.Renviron` automatically at startup, so the key stays out of your scripts and `.Rhistory`.

```{r chat-use}
# OpenAI (the default provider)
statcan_chat("R&D expenditures in Quebec since 2020", model = "gpt-4o-mini")

# Anthropic (Claude)
statcan_chat(
  "R&D expenditures in Quebec since 2020",
  provider = "anthropic", model = "claude-opus-4-8"
)
```

The `"openai"` provider also covers any OpenAI-compatible server — Groq, Together, OpenRouter, Mistral, vLLM, or a local open-source model served by Ollama or LM Studio — by pointing `endpoint` at it.

A single `statcan_chat()` call is one turn. To answer its `clarifying_question` and keep the discussion going, use `statcan_chat_continue()`. Each follow-up stays scoped to the same candidate tables the first call found (it never re-runs `statcan_find()`, and the model still never invents a table number), and results are chainable. Reassign the same variable each turn — there is no need for `r1`/`r2`/`r3`-style names:

```{r chat-continue}
chat <- statcan_chat("R&D spending in Quebec", provider = "anthropic", model = "claude-opus-4-8")
chat$clarifying_question
chat <- statcan_chat_continue(chat, "annual data, since 2015")
chat <- statcan_chat_continue(chat, "just the total, not by industry")
```

To search the catalogue again, start a fresh `statcan_chat()` call rather than continuing.

## Step 2: download a complete table

After choosing an identifier, pass it and the desired language to `statcan_data()`. This example uses a relatively small table that is convenient for learning:

```{r data-eng}
table_data <- statcan_data("10-10-0001-01", lang = "eng")
```

The function downloads and unpacks the current full-table CSV archive, then returns a data frame. Start by examining its dimensions, names, and first observations:

```{r inspect}
dim(table_data)
names(table_data)
head(table_data)
```

The exact columns depend on the table selected. `statcanR` applies a few consistent rules:

- the first column is named `REF_DATE`;
- annual, monthly, daily, and fiscal reference periods are converted to `Date` values when they can be interpreted safely;
- coordinate columns remain character values so leading zeros and compound coordinates are not lost; and
- `INDICATOR` contains the official table title read from its metadata.

For example, you can select observations from 2020 onward with ordinary R subsetting:

```{r subset}
recent_data <- table_data[
  !is.na(table_data$REF_DATE) &
    table_data$REF_DATE >= as.Date("2020-01-01"),
]
```

To download the French version of the table, change the language:

```{r data-fra}
table_fr <- statcan_data("10-10-0001-01", lang = "fra")
```

Most source column names remain in the selected language, so do not assume that every English column name has an identical French equivalent.

## Step 3: save a CSV copy when needed

If you only need to analyse the data in the current R session, `statcan_data()` is sufficient. Use `statcan_download_data()` when you also need a CSV file.

Existing two-argument calls save the file in the current working directory:

```{r save-default}
table_data <- statcan_download_data("10-10-0001-01", "eng")
getwd()
```

This creates `statcan_10100001_eng.csv`. To keep project files organized, create a dedicated directory and pass it through `path`:

```{r save-path}
output_dir <- file.path(tempdir(), "statcanR-data")
dir.create(output_dir, recursive = TRUE, showWarnings = FALSE)

table_data <- statcan_download_data(
  "10-10-0001-01",
  "eng",
  path = output_dir
)

attr(table_data, "statcan_file")
```

The output directory must already exist. The function returns the same data frame as `statcan_data()` and stores the exact CSV path in its `statcan_file` attribute. The CSV uses UTF-8 encoding, excludes R row names, and writes missing values as empty fields.

## Compatibility with earlier statcanR scripts

The update does not require you to rewrite established calls:

```{r compatibility}
# This familiar two-argument form remains valid.
table_data <- statcan_data("10-10-0001-01", "eng")

# This also remains valid and saves into the working directory.
table_data <- statcan_download_data("10-10-0001-01", "eng")
```

The optional `path` argument extends `statcan_download_data()` without changing the meaning of the original arguments. Both hyphenated table numbers and eight-digit PIDs are accepted.

## Troubleshooting

The package validates inputs before downloading and reports network or service problems explicitly. Common issues include:

| Message or symptom | What to check |
|---|---|
| No natural-language results | Keep a clear subject, but remove a geography or date constraint; then inspect broader candidates |
| No exact keyword results | Try fewer keywords, check the selected language, or use a broader official term |
| Invalid `tableNumber` | Use a displayed number such as `10-10-0001-01` or an eight-digit PID such as `10100001` |
| Invalid `lang` | Use exactly `"eng"` or `"fra"` |
| Output directory does not exist | Create the directory before supplying it through `path` |
| WDS is unavailable | Check the internet connection and try again later; catalogue search may use a valid cache |
| Download takes a long time | The function retrieves the complete table, which may be large |

Network failures, invalid tables, unexpected API responses, and malformed archives stop with messages that identify the affected Product ID. Temporary files created by a call are removed when it finishes; other files in the R session's temporary directory are left untouched.

## Reproducible use

WDS provides the current version of a Statistics Canada table, and published observations may be revised. For work that must be reproduced later:

1. record the table identifier, language, package version, and retrieval date;
2. save a local CSV copy of the data used in the analysis; and
3. cite the table and Statistics Canada according to the applicable data licence.

## Data licence and citation

Review the [Statistics Canada Open Licence](https://www.statcan.gc.ca/en/terms-conditions/open-licence) before redistributing downloaded data. To obtain the package's current citation, run:

```{r citation}
citation("statcanR")
```