Job Description
Job Description
Triangle Cyber is seeking a highly skilled and motivated Databricks Engineer to join our team in northern Virginia for a federal contract engagement. As a key contributor, you will help design, implement, and optimize advanced analytics solutions on the Databricks platform to support federal agency objectives. The ideal candidate will have hands-on experience with cloud-based big data technologies, a strong understanding of data engineering principles, and the ability to work within the compliance constraints required for government contracts. You will collaborate with data scientists, analysts, and other stakeholders to deliver scalable, secure, and high-performance data solutions in a mission-critical environment.
Key Responsibilities
- Databricks Platform Development: Design, build, and maintain scalable data pipelines and analytics solutions leveraging Databricks and Apache Spark for federal agency projects.
- Data Integration: Develop and manage the ingestion, transformation, and integration of structured and unstructured datasets from various sources (on-premises, cloud, and external) into Databricks workspaces.
- ETL Process Automation: Implement automated ETL workflows using Databricks notebooks, orchestration tools, and CI/CD methodologies to ensure efficient and repeatable data processing.
- Optimization & Performance Tuning: Monitor, analyze, and optimize the performance of Spark jobs and Databricks clusters, including tuning configurations to meet demanding SLAs.
- Collaboration & Support: Work closely with data scientists, business analysts, and project managers to understand requirements, provide technical guidance, and support advanced analytics, machine learning, and reporting needs.
- Security & Compliance: Ensure all data engineering activities meet federal security and compliance standards (such as FISMA, FedRAMP, and other agency-specific requirements), including data encryption, access controls, and auditing.
- Documentation: Produce and maintain comprehensive technical documentation for data pipelines, architecture, processes, and compliance procedures.
- Troubleshooting: Diagnose and resolve issues related to Databricks environments, Spark applications, data pipelines, and integrations in a timely manner.
- Continuous Improvement: Stay current with emerging Databricks features, best practices, and federal data management requirements, and proactively recommend improvements to enhance the agency's analytics capabilities.
- Stakeholder Communication: Communicate project progress, technical challenges, and solutions to both technical and non-technical stakeholders.
Required Qualifications
- Must be a US Citizen with an active Secret clearance.
- Must have 8+ years of software engineering experience
- Must have a minimum of 4 years of experience performing data engineering work in a cloud environment
- Must have experience with relational, noSQL and/or file-based storage (e.g. Databricks, Elastic, Postgres, etc.), with at least 1 year working directly with Databricks and Apache Spark.
- Must have experience with DevOps tools such as Jenkins, Docker, Ansible, Puppet, Chef, etc.
- Must have a working knowledge of Databricks, Cloud Relational Database Services and Elasticsearch
- Must have a Bachelor's degree in Software Engineering, Computer Science, or a related discipline. Ten (10) years of experience (for a total of eighteen (18) or more years) may be substituted for a degree.
- Must have excellent written and verbal communication skills for cross-functional collaboration and documentation.
Desired Qualifications
- Master's degree in a relevant technical field.
- Databricks certifications (e.g., Databricks Certified Data Engineer Associate, Databricks Certified Data Engineer Professional, Databricks Certified Apache Spark Developer)
- Hands-on experience with Databricks workflows (MLflow), Delta Lake, Databricks Unity Catalog, Databricks Autoloader, and Databricks Delta Live Tables, advanced Databricks features, Apache Spark, and Python/Scala/SQL programming.
- Basic working knowledge of Apache Niagara Files (NiFi)
- Experience with collaboration tools, including MS Teams, MS Outlook, MS SharePoint, and Confluence
- Amazon Web Services (AWS) Professional certification or equivalent.
- Experience implementing compliance frameworks and security protocols for federal data systems.
- DoD 8570.1-M Compliance at IAT Level I (e.g., Certified Information Systems Security Professional (CISSP)) certification is highly desired
- Familiarity with BI tools such as Tableau, Power BI, or similar analytics platforms.
- Experience in building and optimizing scalable big data pipelines on cloud platforms (Azure, AWS, or GCP).
- Strong understanding of data warehousing concepts, ETL processes, and distributed computing.
- Knowledge of source control (Git), CI/CD pipelines, and DevOps practices.
- Experience with REST APIs, data lakes, and data visualization tools.
- Familiarity with federal data security and privacy standards (FISMA, FedRAMP, NIST, etc.).
- Experience with infrastructure-as-code and containerization (Terraform, Docker, Kubernetes).
- Experience supporting federal agency data initiatives and knowledge of government contracting environments.