Data Engineer

Location: Bristol, UK

About the role

We are looking for a Data Engineer to join our rapidly growing team. The Data Engineer will be responsible for working with other members of the engineering team to expand and optimise our data models and data pipeline architecture, as well as optimising data flow and collection for cross functional teams. The ideal candidate is an experienced database engineer and pipeline builder who enjoys optimizing data systems and building them from the ground up. The Data Engineer will support our software engineers, systems architects, and product managers on data initiatives, and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of optimizing or even re-designing our company’s data architecture to support our next generation of products and data initiatives.

About the Team

Our SaaS products and services are designed, built and cared for by our engineering teams in Bristol. Each team owns a slice of the product portfolio in close collaboration with product managers. We believe in letting teams self-organise and self-determine by supporting them with design and project expertise when needed.We use Java, Go, PHP, JS, TypeScript and SQL in a diverse suite of applications. Our service oriented architecture processes thousands of transactions an hour while maintaining high-availability and reliability. Our uniform deployment platform (Kubernetes on AWS supported by our global Infrastructure team) means that everyone gets hands-on experience with the full lifecycle of each product.

Your Day-to-Day:

• Participate in a small cross-functional team responsible for optimising database design and performance, driving down hosting costs and coordinating Brightpearl’s data strategy going forward.
• Collaborate with Software Engineers to help them deliver product features and remove defects across the platform, and to coach them in data engineering best practices.
• Partner with DevOps Engineers and SDETs to install performance test infrastructure and automate database operations.
• Identify where additional logging and tracing is required in both the database and application to help diagnose performance issues.
• Introduce tooling to analyse, visualise and understand database performance characteristics.
• Clearly and succinctly present performance data to both technical and non-technical stakeholders.
• Bring past knowledge and experience of query and schema optimisation to analyse these data and assist the Product Owner in creating a backlog of performance fixes.
• Develop a deep understanding of the business domains and how they are / should be mapped in the data layer.
• Work with the Architects Group to shape and progress data strategy in line with the direction of travel of the business.
• Research and present options to educate the Engineering Team when it comes to selecting database technologies, storage options, ETL processes, and so on.

About You:

Technology Skills

• 3+ years experience as a Data Engineer, Database Administrator, Data Architect or similar.
• BSc (or equivalent) in a technical subject (comp.sci./science/engineering)
• Experience with AWS RDS.
• Experience and understanding of the challenges involved in scaling up database provisioning for cloud-based services.
• Experience of database parameter tuning with MySQL and/or Postgres.
• Advanced SQL skills.
• Experience of analysing and optimising query plans.
• Understanding of logical and physical partitioning strategies.
• Understanding of schema design and indexing methods.
• Experience calculating projected hosting costs, and understanding trade-offs.
• Understanding of data security and governance issues.

Professional Skills

• An attitude of continual improvement, lifelong learning and an interest in new technologies.
• Comfortable switching contexts between low-level engineering and high-level architecture as required.
• Ability to assess the relative merits of different database platforms and technologies according to requirements.
• Good communication skills; fluent in English.

Personal Attributes

• A good team player capable of delivering to deadlines.
• Ability to work calmly under pressure to help diagnose performance issues affecting customers in production.
• Comfortable proactively communicating with colleagues and stakeholders.
• Quality focused and value driven.

Ideally, you’ll have some of the following:

• Experience building and maintaining ETL pipelines.
• Understanding of the principles of data warehousing, star / galaxy schemas, OLAP cubes, data marts, data lakes and analytical reporting architecture.
• Experience with NoSQL databases, eg. document stores, column stores key-value stores, graph databases.
• Knowledge of event-driven architectures and technologies eg. Kafka.
• Experience in performance load testing of database-backed services.
• Knowledge of logging and tracing in database applications.
• Knowledge of advanced database monitoring and visualisation tools.
• Experience working with SaaS / microservices.
• Software Engineering experience with Java.