tailk
Principal System Engineer
About the Role We are seeking a seasoned, hands-on Principal System Engineer to act as a technical vanguard and drive engineering excellence across our organization. Operating in a highly regulated banking environment, you will be the driving force behind our strategic technical direction, platform reliability, and system resilience. This is an influential role where you will balance the hands-on resolution of complex, cross-layer challenges with proactive architectural innovation. You will mentor engineering teams, foster cross-departmental collaboration, and ensure our distributed systems meet the absolute highest standards of compliance, security, and performance.
What You Will Do
- Architect & Operate at Scale: Design, build, and maintain highly available infrastructure using self-managed Kubernetes on Azure. You will manage distributed architectures, stateful sets, database configurations, and advanced resilience patterns (circuit breakers, Istio traffic shifting).
- Drive Reliability & SRE Practices: Anticipate operational bottlenecks, define error budgets, conduct chaos engineering, and implement proactive scaling policies. You will own P0/P1 incident management from rapid cross-layer triage through to blameless Root Cause Analysis (RCA) and permanent mitigation.
- Champion DevSecOps & Automation: Build and manage automated, compliance-aligned CI/CD pipelines utilizing GitHub Actions, ArgoCD (GitOps), and Helm. You will enforce strict quality and security gates (SonarQube, Dependency-Track) and champion Infrastructure as Code (Terraform).
- Ensure Enterprise Security & Compliance: Navigate stringent financial regulations (PCI-DSS, GDPR, SOC 2, ISO 27001) by embedding secure SDLC processes. You will implement least privilege access (RBAC), mTLS, secret management (Azure Key Vault), network segmentation, and robust audit trails.
- Lead & Mentor: Advocate for technical standards, conduct architecture and secure code reviews, and actively mentor other system engineers. You will act as a calm, authoritative technical voice during outages and serve as a strategic consultant to both technical and business stakeholders.
What You Bring to the Team
- Cloud Native Mastery: Expert-level production experience with self-managed Kubernetes, Azure services, advanced Linux systems administration, and deep network routing/troubleshooting.
- Service Mesh & Networking: Expert configuration and operation of Istio Service Mesh (traffic management, mTLS, fault injection) and deep proficiency in DNS management (A, CNAME, SRV, SPF, DKIM).
- Java & Performance Tuning: Strong ability to read, modify, and optimize Java/Spring Boot microservices. You bring hands-on expertise in JVM memory tuning, heap sizing, and Garbage Collection (GC) analysis within containerized limits.
- PKI & Security: Deep practical knowledge of Public Key Infrastructure (PKI), certificate lifecycles, revocation, and security enforcement across distributed environments.
- Comprehensive Observability: Expertise utilizing Prometheus, Grafana, OpenTelemetry, and logging stacks (Loki/ELK) to drive data-informed platform stability.
- Systematic Debugging: Proven ability to dive deep into complex outages using distributed tracing, JVM debugging (heap/thread dumps), and infrastructure tools (kubectl, Envoy logs).
- Communication: Professional fluency in written and spoken English, with the presence and clarity to explain complex technical remediation to developers, security auditors, and executive management.