Egen is a data engineering and cloud modernization firm helping industry-leading companies achieve digital breakthroughs and deliver for the future, today. We are catalysts for change who create digital breakthroughs at warp speed. Our team of cloud and data engineering experts are trusted by top clients in pursuit of the extraordinary. An Inc. 5000 Fastest Growing Company 7 times, and recently recognized on the Crain’s Chicago Business Fast 50 list, Egen has also been recognized as a great place to work 3 times.
As a Staff Software Engineer at Egen you will deliver a key component of a large modernization project by migrating legacy Control M (xml) jobs to run seamlessly through Airflow DAGs. You will design and scope delivery of the new MWAA (Managed Workflows for Apache Airflow) environment on AWS. The Staff Engineer will also be responsible for leading the onboarding and integration of multiple product teams onto the new Airflow platform. This role will also be responsible for creating Platform Run books and user guides as part of the product team onboard. This role will require significant hands-on work as you identify and implement custom Airflow operators with AWS.
Required Experience:
- Airflow expert with experience in setting up and managing MWAA on AWS.
- In-depth knowledge of orchestration concepts, including DAG (Directed Acyclic Graph) creation, task definition, operators, and best practices for workflow management.
- Proficiency in Python programming, with a focus on developing Airflow DAGs and custom operators.
- Strong understanding of Terraform, including the ability to write Terraform code for infrastructure as code (IaC) management.
- Familiarity with AWS services such as S3, Lambda, CloudWatch, Fargate, VPC, security groups, and other related technologies.
- Understanding of AWS Postgres and experience working with databases in the cloud environment.
- Basic-level knowledge of Unix shell scripting for basic automation and maintenance tasks.
Nice to have's (but not required):
- Experience with Python/Golang and cloud SDKs like AWS SDK, Kubernetes clients library, Helm
- Experience with Mysql, PostgreSQL, Bitbucket, GitHub
- Knowledge of service mesh frameworks like Istio, Linkerd, and Hashicorp Consul Connect
- Knowledge of IAM and JWT authorization, SSO/OAuth (Azure AD, Okta)
- Knowledgeable of 12-factor app methodology and how they should be applied
