Self-hosting LLM inference in a regulated bank

Jul 14, 2026 min read
Self-hosting LLM inference in a regulated bank

Why we self-hosted in the first place

At Colombia’s first fully digital bank, the pressure to use LLMs showed up the same way it did everywhere else. Everyone wanted summarization, log triage, and natural-language querying over operational data. The catch was that our data (transaction context, application logs, customer-adjacent operational records) is regulated. Sending it to a third-party inference API would have meant shipping regulated banking data outside our controlled boundary, and no amount of a vendor’s SOC 2 report was going to make that conversation with our compliance and risk teams a short one.

So the framing was never “which model is smartest.” It was “which architecture lets us use LLMs without the data leaving the perimeter we already defend.” We ran the bank against a 99.999% availability SLO with SOC 2, PCI DSS, and ISO 27001 obligations, and we’d already secured the operational banking license on the back of a specific set of tech controls. Any new capability had to inherit those controls rather than route around them. That single constraint drove almost every decision below.

The architecture: open weights first, hosted APIs as a graded option

We landed on a spectrum rather than a single answer. For anything touching regulated data, we ran open-weight models ourselves, using vLLM and Ollama on GPU-backed EC2, so the weights, the prompts, and the retrieved context all stayed inside our AWS accounts. We also used Amazon Bedrock, which let us stay inside the AWS trust boundary and our existing IAM and networking posture while offloading the model hosting.

Hosted third-party APIs (OpenAI, Hugging Face, DeepSeek) were still in the picture, but as a graded option. They were fine for non-sensitive, non-regulated work, and off-limits for anything that would carry regulated data across the boundary. Treating the destination as a data-governance decision, made per workload, mattered more than any benchmark.

The piece that made this tractable was a compatibility layer. We fronted everything with FastAPI endpoints that spoke the OpenAI wire format, and put OpenWebUI on top for human-facing use, all on EKS. A generic version of the idea:

@app.post("/v1/chat/completions")
async def chat_completions(req: ChatRequest):
    backend = route_for(req.model)  # local vLLM, Bedrock, or a hosted API
    return await backend.complete(req)

Because every backend looked like the OpenAI API to callers, switching a workload from a hosted API to a self-hosted model was a routing change, not a rewrite. Application teams, LangChain, and OpenWebUI all coded against one interface. The governance decision, that is, where a request’s data is allowed to go, lived in route_for, in one place, instead of being smeared across dozens of clients.

Retrieval over logs and operational data

The most useful workloads weren’t chatbots. They were retrieval over things we already had. We used LangChain to orchestrate retrieval over application logs and operational data, then used the models for log analysis, PostgreSQL and MySQL query analysis, and general text generation. “Why did this batch fail last night” turned into a retrieval step over the relevant logs plus a generation step, instead of an engineer grepping across systems by hand.

Keeping this in-house paid off precisely because logs are some of the most sensitive data we hold. They’re full of internal identifiers and behavior that you absolutely do not want leaving the building. Running retrieval and inference on infrastructure we already governed meant the sensitive part of the pipeline never had to cross a trust boundary to be useful.

Treating it as platform, not a science project

The thing I’m most convinced was right: we refused to let this become a shadow stack. The whole platform lived under the same Terraform IaC, the same CI/CD, the same secrets management, and the same security controls (SAST via SonarQube, cloud posture via Dome9) as the rest of the bank. GPUs are expensive, so it also went through the same FinOps discipline we applied elsewhere, with right-sizing and reservations rather than leaving accelerators idle.

That’s what let us have a credible answer when risk asked “where does the data go, who can reach it, and how do you know.” The honest lesson is that the model-serving part was the easy 20%. The hard, valuable 80% was making an LLM platform indistinguishable, operationally and from a compliance standpoint, from every other regulated service we ran.

Takeaways

  • In a regulated shop, the deployment destination is a data-governance decision, not a performance one. Decide it per workload.
  • An OpenAI-compatible layer (FastAPI in front of vLLM/Ollama/Bedrock/hosted APIs) turns “self-host vs. hosted” into a routing change and centralizes the governance decision in one place.
  • Bedrock is a pragmatic middle ground: managed hosting without leaving your cloud trust boundary.
  • Your logs are among your most sensitive data; keeping retrieval and inference in-house is what makes them safely useful.
  • Put the AI platform under the same IaC, CI/CD, secrets, security, and FinOps controls as everything else. The serving is the easy part; inheriting the controls is the point.