# statster-scraper

Multi-source scraper that collects football match statistics for the **top-5
European first divisions** and stores them in PostgreSQL. It is the data-
collection half of the **statster** project (match-result prediction).

| League | Country | internal code |
| --- | --- | --- |
| Premier League | England | `EN_PL` |
| Primera División | Spain | `ES_PD` |
| Bundesliga | Germany | `DE_BL1` |
| Serie A | Italy | `IT_SA` |
| Ligue 1 | France | `FR_L1` |

Default seasons: the **3 most recent completed seasons** (`config/seasons.php`).

## Requirements

- PHP **8.3+** with `pdo_pgsql`, `curl`, `mbstring`, `dom` (`intl` recommended)
- PostgreSQL **11+**
- Composer
- A free **football-data.org** API token — <https://www.football-data.org/client/register>

## Setup

```bash
composer install
cp .env.example .env          # then edit DB_* and FOOTBALL_DATA_API_TOKEN
createdb statster             # or: CREATE DATABASE statster;
php bin/migrate.php
```

## Usage

```bash
php bin/scrape.php --list                       # show configured sources/leagues/seasons
php bin/scrape.php                              # all sources × all leagues × all seasons
php bin/scrape.php --source=football-data
php bin/scrape.php --source=transfermarkt --league=EN_PL --season=2024/2025
php bin/scrape.php --league=EN_PL,ES_PD --season=2025/2026
```

The command is idempotent — re-running upserts and fills gaps (e.g. one source's
half-time score merged with another's kickoff time) without creating duplicates.

## Database schema (`migrations/001_init.sql`)

```
countries ─┬─< leagues ─┐
           └─< teams    │
                        ├─< season_teams >─ seasons
seasons ────────────────┘
                        └─< matches (matchday, kickoff_at, status,
                                     ht_home/ht_away, ft_home/ft_away)

source_references (source, entity_type, entity_id, source_id, source_url, raw_payload jsonb)
```

- **`source_references`** is polymorphic: every `country / league / season /
  team / season_team / match` row keeps one row here **per source**, holding
  that source's own id, URL and raw JSON payload. Look up an entity's sources
  with `WHERE entity_type = 'match' AND entity_id = ?`.
- **`season_teams`** answers "which teams played league L in season S".
- **`matches`** is keyed by `(season, league, home_team, away_team)`. `status`
  is normalized: `scheduled | finished | in_play | postponed | cancelled |
  suspended`. `status` is never downgraded from `finished` on re-scrape.
  **Not-yet-played fixtures are stored too** — `status = 'scheduled'`,
  `kickoff_at` set, `ht_*`/`ft_*` `NULL` — and a later run fills the scores and
  flips the status in place. These future rows are the input for prediction.
- **`team_aliases`** maps name variants to a canonical team. It is seeded from
  `config/team_aliases.php` and grows automatically as sources report new names.

### Views (`migrations/002_match_views.sql`)

- **`match_details`** — every match with league / season / team *names* and a
  derived `result` (`H` / `D` / `A`, `NULL` until full-time is known).
- **`upcoming_fixtures`** — `match_details` filtered to fixtures still to be
  played, ordered by kickoff. This is the set to generate predictions for:

  ```sql
  SELECT * FROM upcoming_fixtures WHERE league_code = 'EN_PL';
  ```

## Architecture

```
ScraperInterface
└── AbstractScraper            HTTP throttle, league-code mapping, status mapping
    ├── FootballDataOrgScraper  football-data.org REST API v4
    └── TransfermarktScraper    transfermarkt.com HTML scrape

  Scraper  ──►  LeagueSeasonPayload (normalized DTOs)  ──►  Repository  ──►  PostgreSQL
```

Every scraper returns the **same** `LeagueSeasonPayload` (league + season +
`TeamDto[]` + `MatchDto[]`, each carrying a `SourceRef`). `Repository::persist()`
takes any payload, resolves cross-source identity, and writes one transaction.

### Adding a source

1. Create `src/Scraper/YourScraper.php extends AbstractScraper`.
2. Implement `name()`, `supportedLeagues()` (internal code → your id),
   `fetchLeagueSeason()` returning a `LeagueSeasonPayload`.
3. Register it in `Container::scrapers()`.

No change to `Repository` or the schema is needed.

### Cross-source team matching

`Repository::resolveOrCreateTeam()` tries, in order: existing
`source_references` row for this source+id → `team_aliases` → `teams.normalized_name`
→ curated alias map → create. `NameNormalizer` lower-cases, folds accents and
strips noise tokens (`FC`, `AC`, `Calcio`, …). Divergent localised spellings
("München" vs "Munich") are covered by `config/team_aliases.php`.

## Known limitations (v1)

- **Transfermarkt half-time scores** are only captured when the schedule page
  prints them in the `2:1 (1:0)` form; otherwise HT lives on each match-report
  page, which v1 does not fetch. For fixtures also seen by football-data, HT is
  filled from there.
- **Transfermarkt anti-bot**: a 403 or consent/interstitial page yields 0 rows;
  the run logs a warning and continues. Lower the frequency
  (`TRANSFERMARKT_DELAY_SECONDS`) or route through a proxy if this happens.
- **football-data.org free tier**: 10 requests/minute (handled via
  `FOOTBALL_DATA_DELAY_SECONDS`) and limited historical depth — older seasons
  may return 403; those league/season combos are logged and skipped.