Universal Observability Stack: How We Used Dynatrace to Monitor 40+ APIs on a National Pension Platform
- Arpan Desai
- 2 hours ago
- 8 min read
How we rebuilt a mission-critical mobile application serving millions of pension subscribers — with 40+ API integrations, middleware orchestration, and a Dynatrace + Customer.io + Firebase analytics stack.
Key Results
83+ defects and VAPT items resolved in a single sprint cycle
40+ APIs integrated via middleware orchestration
60% faster MTTR through Dynatrace distributed tracing
<0.5% crash rate achieved within 60 days via Firebase Crashlytics
18% improvement in task completion from Firebase A/B-tested UX variants
25% reduction in development cycle time with CI/CD and continuous monitoring
Clean regulator VAPT clearance on the new mobile application
Introduction
When a leading eGovernance technology provider needed to modernise its national pension mobile application, the challenge went far beyond a UI refresh. The existing app had accumulated security vulnerabilities flagged by the national pension regulator, a fragmented API landscape with 40+ endpoints, and an architecture that made feature releases painfully slow.
We were brought in to rebuild the mobile platform end-to-end: new native apps for Android and iOS, a middleware orchestration layer to mediate between the mobile client and the core pension APIs, a complete encryption overhaul across every web service, and — critically — a full observability stack to ensure that a system processing millions of pension transactions daily would never go dark.
This article is a technical deep-dive into the architecture, API integration patterns, and observability infrastructure we built. It covers the middleware design, the registration and transaction flows, and how Dynatrace, Customer.io, and Firebase Analytics work together to give us end-to-end visibility from the mobile tap to the database write.
The Problem Space
The platform in question is a national-scale pension system. Subscribers — spanning government employees, private sector workers, and individuals enrolled in pension guarantee schemes — use the mobile app for account registration, contributions, scheme management, withdrawals, bank updates, PAN verification, grievance filing, and status tracking.
What Was Broken
Security: The regulator’s VAPT audit flagged 10+ observations on the existing mobile app. Encryption logic across all web services consumed by the app needed to be replaced with a new standard, including PRAN encryption in every API request.
API Complexity: The mobile app directly consumed 40+ APIs from the core pension platform — registration, eKYC, contribution, scheme change, withdrawals, grievance, bank update, PAN update, contact update, and more. Each had its own authentication, payload format, and error handling patterns.
UX Debt: 83 defects and feature gaps had accumulated — alignment issues, missing validations, incorrect error messages, hidden fields, broken flows for Tier II accounts, and registration journeys that failed silently.
No Observability: There was no APM, no crash reporting, no funnel analytics, and no lifecycle messaging. When something broke in production, the team found out from user complaints, not dashboards.
Architecture: The Three-Tier Design
We designed a clean three-tier architecture that decouples the mobile client from the core platform APIs, with a middleware layer handling orchestration, encryption, session management, and data transformation.
Layer 1: Mobile Application
Native Android and iOS apps built with a modular screen architecture. Each major pension operation — registration, contribution, scheme change, withdrawal, bank update, grievance — is a self-contained module with its own navigation flow, validation logic, and error handling.
The registration journey alone spans 8 stages: initial details, personal details, contact details, FATCA declaration, bank/scheme/nomination for Tier I, bank/scheme/nomination for Tier II, employment details, and photo/signature upload. Each stage persists to the middleware database with a flag-based progress system (Flag 0 through Flag 7), allowing users to resume incomplete registrations.
Layer 2: Middleware Orchestration
The middleware is the backbone of the system. It sits between the mobile client and the core pension platform APIs, and handles several critical functions:
Encryption & Decryption: All requests from the mobile app are encrypted at the middleware. PRAN (Permanent Retirement Account Number) is encrypted in every API payload before forwarding to the core platform. The new encryption logic replaced the previous implementation across all web services.
Session & State Management: Registration data is stored in the middleware database across the 8-stage journey. Only after the final stage does the middleware call the core platform’s Submit API (postPensionDetails) with the complete payload. A separate getPranId API polls for registration status using a reference ID.
API Mediation: The middleware normalises request/response formats across 40+ endpoints. Services include one-way switch, conditional withdrawal, subscriber shifting, dual OTP, bank details update, user profile retrieval, penny drop verification, and contribution processing.
Data Lifecycle: Once a subscriber is successfully registered and their first investment is processed, the middleware deletes the temporary registration data from its database — the core platform becomes the single source of truth.
Layer 3: Core Platform APIs
The core pension platform exposes APIs for every pension operation. Our middleware integrates with these APIs but does not modify them — it wraps, encrypts, validates, and routes. Key integrations include the CRA Online API for registration and status tracking, deduplication APIs for PAN and contact verification, eKYC/Aadhaar verification, payment gateway APIs for contributions, and OTP services for transaction authorisation.
Key Technical Flows
Subscriber Registration (8-Stage Pipeline)
The registration flow is the most complex journey in the application. It processes personal details, FATCA declarations, bank account information, scheme selection, nominee details, employment data, and biometric documents — across both Tier I and Tier II pension accounts.
Technical considerations we addressed include: eKYC-based registration with Aadhaar consent, CKYC integration for auto-populating Aadhaar details, a deduplication API to check if PAN/mobile/email already exist in the system, signature compression handling (resolving intermittent upload failures), default Tier I selection with “Same as Tier I” options for Tier II bank, nominee, and scheme details, and a goal planner module that conditionally hides when Tier II is unavailable.
Contribution & Payments
Contribution processing integrates with external payment gateways. We resolved a critical issue where the backend returned a success response but the UI displayed “payment terminated” — a state synchronisation bug between the middleware callback handler and the mobile app’s payment status polling. We also implemented a regulator directive capping first contributions at ₹50,000.
VAPT Remediation
The regulator’s VAPT audit produced observations for both Android and iOS. We closed all 83 items in a single 31-day sprint, covering encryption updates, input validation hardening (special character handling in registration, percentage fields in withdrawal), MPIN brute-force protection (limiting incorrect attempts), password policy enforcement (no spaces, no reuse of last 4 passwords), and secure session handling across biometric authentication flows.
The Observability Stack
This is where the modernisation goes beyond “rebuild and ship.” We instrumented every layer of the platform with purpose-built observability tools, each covering a distinct part of the monitoring spectrum.
Dynatrace: Full-Stack APM
Dynatrace is our primary application performance monitoring layer. It gives us end-to-end visibility from the mobile client through the middleware to the core platform APIs.
What Dynatrace Monitors |
API Latency Tracking: P95 response times monitored per API endpoint across all 40+ middleware services. Automatic baseline detection flags degradation before it impacts users.
Distributed Tracing: End-to-end request traces from the mobile tap through middleware orchestration to the core API and database. This is how we identify bottlenecks — whether the delay is in our middleware, the payment gateway, or the core platform’s response time.
Error & Exception Tracking: Automated root-cause analysis for failed transactions. Contribution failures, payment gateway timeouts, KYC verification errors, and signature compression failures are surfaced with full stack traces and correlated request data.
Real User Monitoring (RUM): Session-level analysis of how subscribers navigate the mobile app. We use RUM data to identify drop-off points in the registration funnel and correlate UX issues with backend performance.
Infrastructure Health: Server CPU, memory, and container health dashboards with auto-scaling triggers configured based on Dynatrace load signals — particularly critical during contribution surge periods.
SLA Dashboards: Custom dashboards tracking uptime, transaction success rates, and API health against the regulator’s SLA requirements. Proactive alerting goes to both the engineering team and stakeholders.
The impact has been measurable: distributed tracing alone reduced our mean-time-to-resolution by approximately 60%. Instead of sifting through logs across three layers, engineers can trace a failed transaction from the user’s device to the exact API call that errored.
Customer.io: Lifecycle Messaging & Engagement
Customer.io handles the communication layer — the automated messaging that keeps subscribers engaged and informed throughout their pension lifecycle.
Onboarding Drip Campaigns: When a subscriber starts registration but doesn’t complete it (their flag-based progress stalls at, say, Flag 3), Customer.io triggers a sequence of email and push notifications guiding them through the remaining steps — KYC completion, bank account linking, scheme selection, and first contribution.
Contribution Reminders: Event-triggered nudges when subscribers miss contribution windows. Messages are personalised with the subscriber’s contribution amount and due date, pulled from the middleware’s contribution records.
Transactional Notifications: Real-time confirmations for scheme changes, Tier II withdrawals, bank detail updates, and grievance status changes. These are triggered by webhook events from the middleware when transaction states change.
Re-Engagement Workflows: Dormant subscriber identification based on inactivity signals. If a subscriber hasn’t logged in or made a contribution within a defined window, Customer.io initiates a reactivation campaign with personalised content about their pension balance and upcoming deadlines.
Firebase Analytics: In-App Behavioural Intelligence
Firebase Analytics operates at the mobile app layer, giving us granular data on how subscribers interact with the application — data that Dynatrace’s RUM complements but doesn’t replace.
Registration Funnel Analysis: Step-by-step drop-off tracking across the 8-stage registration flow. Firebase event tracking at each flag transition revealed that eKYC was the primary abandonment point — leading us to optimise the Aadhaar consent flow and CKYC auto-population logic.
Feature Adoption Tracking: Screen-level engagement data showing which pension features subscribers actually use versus which they ignore. This data directly informed our roadmap prioritisation — for example, understanding that the scheme change module had low adoption but high error rates pointed to a UX problem, not a demand problem.
Crash & Stability Reporting: Firebase Crashlytics provides real-time crash reporting with full stack traces. Within 60 days of launch, we reduced the crash rate to under 0.5%. The signature compression failure and payment gateway state synchronisation issues were both initially surfaced through Crashlytics before being diagnosed with Dynatrace traces.
A/B Testing: Firebase Remote Config powers A/B tests on key user flows. We tested alternate contribution payment flows and scheme selection interfaces. Winning variants improved task completion rates by 18%.
How the Three Tools Work Together
The power of this stack is in how the three tools complement each other. Firebase tells us where users are dropping off. Dynatrace tells us why — whether it’s a slow API, a failed backend call, or an infrastructure bottleneck. Customer.io lets us act on it — nudging users who abandoned, confirming completed transactions, and re-engaging dormant subscribers.
A concrete example: Firebase funnel data showed a spike in registration abandonment at the eKYC stage. Dynatrace traces revealed that the Aadhaar verification API was intermittently timing out during peak hours. We optimised the middleware’s retry logic and timeout handling. Customer.io then sent targeted messages to subscribers who had abandoned at that stage, inviting them to retry — resulting in a measurable recovery of incomplete registrations.
What This Means for Regulated Platforms
If you’re operating a mobile platform in a regulated environment — financial services, pensions, insurance, government services — the lessons from this project are transferable:
A middleware orchestration layer between your mobile client and backend APIs gives you encryption, session management, and data transformation without modifying core platform APIs.
Full-stack observability isn’t optional for mission-critical platforms. Dynatrace’s distributed tracing, Firebase’s in-app analytics, and Customer.io’s lifecycle messaging cover the full spectrum from infrastructure to user engagement.
Security remediation and feature development can run in parallel when the architecture separates concerns properly. We closed VAPT items while simultaneously building new pension features.
Aggressive timelines are achievable when you have a clear API inventory, a well-designed middleware layer, and instrumentation that catches issues before they reach production users.
Fintegration is a fintech product engineering firm with 12+ years of experience building and modernising platforms for financial services, pensions, and regulated industries. FintegrationFS.com
