Case Study · Cross-domain · Financial Data System
TRADESIGHT
A Taiwan equity chip-flow analysis tool — institutional and broker-branch flows, screening, health checks and backtesting. Desktop version shipped, and it cannot place orders.
Two decisions in this project were about deliberately giving up capability: I built a scraper and refused to use it, and I integrated a brokerage API but never created its order-placing component. Neither is a missing feature — both are structural choices.
The problem
Broker-level flow data for Taiwan equities is what retail investors most want and can least reliably obtain. Most tools treat "we fetched it" as the finish line. But if the data will drive screening and backtesting, the real question is not whether you can fetch it — it is: when and from where was this number fetched? Was it later revised at source? When two sources disagree, how do you know which is right?
Without those fields, no backtest result can be trusted, because it cannot be reproduced.
Decision one — I built the scraper, then refused to use it
A first-phase spike proved the data was obtainable: branch-level daily reports, institutional flows, margin data, custody dispersion and monthly revenue could all be fetched and parsed. Technically it worked.
Then I archived the entire spike as "reference only, not for production", for three reasons:
- ① The acquisition method does not hold up
Getting that data required breaking an image CAPTCHA with OCR on one exchange and facing reCAPTCHA on the other — both fragile (one site change breaks it) and questionable under the terms of service. A system meant to run for years cannot rest on that foundation.
- ② No provenance fields means no financial-grade reconciliation
Scraped rows carried no source, fetch timestamp, checksum or revision. Without those I cannot answer "was this number later corrected?", nor adjudicate when sources conflict — meaning I cannot reconcile, and backtests are not reproducible.
- ③ Storing prices as floating point is simply wrong here
The spike stored prices as
REAL. Accumulated floating-point error is unacceptable in financial computation. This later became one of the project's hard rules: never store monetary amounts or share counts as floats.
The production version moved to government open data and the user's own brokerage account, with all 38 sources behind one provider interface and every row carrying provenance. The cost is fewer fields and slower development; what I bought is a pipeline that is sustainable, auditable and reproducible.
Trade-off
- Narrower coverage: fields obtainable only by scraping simply do not exist in the production version — the price paid for legitimacy and reconcilability.
- Slower development: integrating licensed sources and adding provenance columns is far slower than writing a scraper.
- The public web version is therefore still unshipped: the licensing gate is ready, but guaranteeing that a public deployment touches only open data is itself real work.
Decision two — read-only, and structurally incapable of trading
After integrating the brokerage API, the natural next step was order placement. I did not build it, and deliberately made it structurally impossible:
- Only three read-only components are ever initialised
The brokerage API is initialised for login, quotes and reports only. The order component is never created — not hidden behind a disabled button, simply never loaded into the process.
- A CI acceptance test pins it down
An automated acceptance test guards this property: if anyone — including future me — adds the order component back, the test fails. Turning "I promise not to" into "doing it breaks the build" is a large difference in trustworthiness.
- Why this is worth doing
A tool that can trade carries a completely different risk profile: a bug goes from "shows a wrong number" to "submits a wrong order". Since this tool's value is data curation and deterministic computation, the attack surface and the liability are closed off together.
Architecture — three interfaces, one backend
A desktop executable, a tablet PWA and a (not-yet-deployed) public web build share one Python backend, with data sources and deterministic computation in strictly downward-depending layers.
- 1 · Desktop — Tauri v2 shell + Python sidecar
A Tauri v2 shell wraps a PyInstaller-packaged Python backend. The backend binds
127.0.0.1only and opens no external port. The front end is framework-free vanilla JS embedded in the shell. - 2 · Tablet — LAN HTTPS with a pairing code
A separate TLS-terminating proxy bound to a chosen interface forwards to the local backend, turning a tablet into a second monitor. It requires an 8-character pairing code (~2³⁹·⁶ entropy, rate-limited, constant-time comparison), closes on idle, and defends against DNS rebinding and CSRF; PC-side operations such as certificate handling always return 403 to the tablet.
- 3 · API boundary — fail-closed contract validation
FastAPI exposes 97 endpoints handling routing, contract validation, caching and LAN gating. Responses are validated against JSON Schema in
contracts/api_v1— a failing response is blocked, not passed through. - 4 · Deterministic computation layer
src/metrics(indicators, flows, dividend recovery, backtesting),src/screening,src/health,src/quality.src/api/assemble.pyonly assembles output and performs no financial computation — the boundary between computing and shaping data is drawn deliberately. - 5 · Provider layer — 38 sources, all with provenance
Government open data, custody data, third-party services and the user's own real-time quotes all implement one provider interface, and every row carries source and fetch information. Field-level contracts are documented in
docs/DATA_CONTRACTS.md.
The licensing gate — if data isn't licensed, say so honestly
The 20 data sources come with different licences: some are freely usable government open data, others are restricted by terms of service or require the user's own token and brokerage certificate. Mixing them together means a public deployment crosses a line.
So sources are tiered by licence and gated by a DATA_MODE environment variable — decided at startup, with no endpoint able to change it at runtime; any invalid value fails closed to the most conservative public mode.
| Source type | Licence condition | Public build |
|---|---|---|
| Government open data (exchange openapi) | Government open data licence | ✅ |
| ToS-restricted / token-required sources | Restricted by terms | ❌ personal mode only |
| Real-time quotes (tick / order book) | User's own account + certificate | ❌ desktop personal build only |
The last principle matters most to me: unlicensed data is always shown honestly as "unavailable", never faked with a zero or an estimate. In a financial tool, a blank that looks like a number is far more dangerous than an explicit "we don't have this".
Security design
- Credentials never touch HTTP
Brokerage credentials are entered in a local window: never over the network, never in logs, not stored by default. An optional third-party token is encrypted at rest with Windows DPAPI (user scope) — undecryptable on a different machine or account.
- The update channel must verify signatures
Auto-update runs over HTTPS with a pinned Ed25519 public key and mandatory signature verification (minisign) before install. Auto-update is the best supply-chain attack vector, which is exactly why it needs the strictest verification.
- Protective headers on every response
X-Frame-Options: DENY,nosniff,frame-ancestors 'none'andno-referrerare applied to all responses, not just pages.
Engineering discipline — three non-negotiable rules
This repo is developed by multiple AI models collaborating. To make that safe in a financial context, the rules have to be written down rather than left to discipline:
- The author may not approve their own merge; the final merge is performed by the repo owner — the most important rule in human code review, carried over to collaborating with AI.
- Financial critical paths must be independently recomputed by a non-author party — not "read the code again", but compute it separately and check the numbers match.
- Never use an LLM as a calculation engine; never store money or share counts as floats; never use
verify=False— the three places most often compromised for convenience, listed as outright prohibitions.
The first half of rule three is the same lesson I learned building medical software: a language model does not belong on a path that requires a correct answer. In ClinCalc, clinical interpretation is entirely deterministic and the LLM only paraphrases; here it goes further, making "no LLM as a calculation engine" a non-negotiable project rule.
Testing and verifiability
- 1,787 automated tests, all green — and not only functional tests: they include security and contract guard tests (such as the "the order component must not exist" check above).
- Fail-closed response boundary: every API response must validate against JSON Schema in
contracts/api_v1; a failing response is blocked rather than passed to the front end malformed. - Architecture Decision Records: recording why a choice was made, not just what was chosen. The reason the scraper and no-trading decisions can be explained clearly is that the reasoning was written down at the time.
- Release pitfalls are documented: e.g. "touching the front end requires re-running from the packaging step, or the tablet gets a stale front end while the desktop build looks fine" — the kind of thing you only learn by getting burned, written down so it does not recur.
Tech stack
Reflection / future work
- How much evidence justifies refusing something that works? Archiving the spike was a judgement based on three reasons, but I never quantified how many reconciliation failures the scraper would actually have caused. Quantifying it would turn intuition into evidence.
- Can the licensing gate be proven correct? Today it relies on start-up locking, fail-closed defaults and guard tests. But "the public mode can never reach a restricted source" is currently covered by tests rather than formally guaranteed — the same class of problem as the 37 database authorization rules I could not prove complete in my clinical system.
- How far can cross-source reconciliation be automated? Disagreements are flagged, but arbitration still needs a human. Could historical source reliability drive weighted automatic arbitration?
- Commonality with the medical systems: deterministic computation first, no LLM on the critical path, always keep traceable provenance, and prefer showing "unavailable" over estimating. These are the same principles I applied in ClinCalc and ExClinCalc. Two high-stakes domains converging independently on the same trade-offs is itself worth studying.
Further reading
- Public release history — verifiable version history and security updates (the main repo is private)
- ClinCalc case study — the same "rules over LLMs" stance applied to clinical interpretation
- Kaizei case study — another financial-domain work (zero-knowledge encryption)