Technology and roadmap

Built on the stack we already run

R first, Parquet and PostgreSQL storage, one job per source, and a Shiny dashboard published like the other NXW apps.

Technology

The stack

Connectors and checks in R (httr2, data.table, arrow, DBI, validate); Python only where needed.

Parquet by table and date, queried with DuckDB; PostgreSQL for codes, sources and the quality log.

One job per source on a Linux server, then the daily build once every source is in.

IFRC SFTP folder, a plumber API, and the nxwdata R package used by every app.

A Shiny dashboard published like the other NXW apps, plus a daily email.

One Git repository, a folder per connector, tests on every check.

Roadmap

Phase 1

Vietnam prices, corporate actions, shares and FX; NXW codes; the three zones; core checks.

Done when IMS reads all its inputs from the DMS

Phase 2

Global data, external indexes, macro series; the nxwdata package; the dashboard.

Done when FUNDMS, INDOBS and VTP drop their own downloads

Phase 3

Fundamentals, ESG and alternative data, point-in-time panels, data dictionary.

Done when CCPR and CCPS back tests run on DMS panels

Phase 4

Intraday prices, paid feeds where needed, access roles, external data products.

Done when selected datasets are offered to clients

First steps

1List

List every download each app does today, with its source and schedule.

2Codes

Agree the NXW instrument code and build INSTRUMENT and ID_MAP for Vietnam.

3Connectors

Move the VTP Yahoo script and the IMS price load into DMS connectors.

4Checks

Write the core checks and the DQ_LOG.

5Switch

Switch IMS to the Curated zone, then the other apps one by one.

Clean data in. Confident decisions out.