Principal Data Engineer - Data Platform & Ingress
Movable Ink is a software company that provides marketers with technology and expert services to create unique customer experiences.
Job Description
As a Principal Data Engineer, you will help drive the direction of our Data Warehouse, and be hands on with teammates across all departments to enable access to our data and empower teams to make data driven decisions about the direction of our business. You will play a pivotal role in owning the infrastructure and code for our ETL processes within our Data Platform. You will be responsible for designing, implementing, and optimizing ingestion pipelines, enabling seamless extraction and loading of data from various sources such as customer warehouses, business systems, and events messaging buses. Movable Ink collects campaign data resulting from Billions of requests served each day. Come and help us manage and make sense of the massive amount of data we’re ingesting!
Responsibilities:
---------------------
- Partner with internal operations teams to identify, collect, and integrate data from various business systems, ensuring comprehensive and accurate data capture.
- Design, implement, and maintain robust ETL pipelines that feed data into the Data Warehouse, ensuring high performance, scalability, and reliability.
- Conduct thorough testing of ETL processes to validate data accuracy and integrity, and troubleshoot any issues that arise.
- Facilitate the loading of data into Data Vault 2.0 models, ensuring adherence to best practices and optimizing for performance and scalability.
- Establish and track SLAs for data processing and delivery, ensuring timely and reliable access to data for all users.
- Enforce robust Role-Based Access Control (RBAC) and Row-Level Security (RLS) policies to ensure data security and compliance with regulatory requirements.
Qualifications:
-------------------
- 8+ years of professional experience as a Data Engineer, with a focus on cloud-based Data Warehouse platforms (Redshift, Snowflake, Firebolt, BigQuery).
- Proven expertise in managing and optimizing multi-petabyte, mission-critical databases, ensuring high availability, performance, and reliability.
- Proficiency in high-level programming languages, particularly Python.
- Hands-on experience with data engineering tools and frameworks, including DBT, Apache Pulsar for distributed messaging and Apache Airflow for workflow automation and orchestration.
- Strong skills in designing and implementing scalable data architectures.
- Excellent communication skills for effective collaboration with cross-functional teams.
- Strong understanding of data security best practices within an AWS environment and compliance with regulatory requirements.
- Experience in providing technical leadership, setting best practices, and driving the adoption of new technologies and methodologies. Mentoring other engineers, fostering a collaborative team environment.
As a Principal Data Engineer, you will help drive the direction of our Data Warehouse, and be hands on with teammates across all departments to enable access to our data and empower teams to make data driven decisions about the direction of our business. You will play a pivotal role in owning the infrastructure and code for our ETL processes within our Data Platform. You will be responsible for designing, implementing, and optimizing ingestion pipelines, enabling seamless extraction and loading of data from various sources such as customer warehouses, business systems, and events messaging buses. Movable Ink collects campaign data resulting from Billions of requests served each day. Come and help us manage and make sense of the massive amount of data we’re ingesting!
Responsibilities:
- Partner with internal operations teams to identify, collect, and integrate data from various business systems, ensuring comprehensive and accurate data capture.
- Design, implement, and maintain robust ETL pipelines that feed data into the Data Warehouse, ensuring high performance, scalability, and reliability.
- Conduct thorough testing of ETL processes to validate data accuracy and integrity, and troubleshoot any issues that arise.
- Facilitate the loading of data into Data Vault 2.0 models, ensuring adherence to best practices and optimizing for performance and scalability.
- Establish and track SLAs for data processing and delivery, ensuring timely and reliable access to data for all users.
- Enforce robust Role-Based Access Control (RBAC) and Row-Level Security (RLS) policies to ensure data security and compliance with regulatory requirements.
Qualifications:
- 8+ years of professional experience as a Data Engineer, with a focus on cloud-based Data Warehouse platforms (Redshift, Snowflake, Firebolt, BigQuery).
- Proven expertise in managing and optimizing multi-petabyte, mission-critical databases, ensuring high availability, performance, and reliability.
- Proficiency in high-level programming languages, particularly Python.
- Hands-on experience with data engineering tools and frameworks, including DBT, Apache Pulsar for distributed messaging and Apache Airflow for workflow automation and orchestration.
- Strong skills in designing and implementing scalable data architectures.
- Excellent communication skills for effective collaboration with cross-functional teams.
- Strong understanding of data security best practices within an AWS environment and compliance with regulatory requirements.
- Experience in providing technical leadership, setting best practices, and driving the adoption of new technologies and methodologies. Mentoring other engineers, fostering a collaborative team environment.
