Why this matters:

Commodity hardware refers to the minimal hardware resources needed to run the Apache Hadoop framework, a collection of open-source utilities for using a network of devices to solve large data problems. Hadoop is a ubiquitous technology in the world of big data, so this level of familiarity is important for any big data engineer.

What to listen for:

• In-depth understanding of the concept of commodity hardware, and why it’s important
• Knowledgeable about key Apache Hadoop concepts, especially as they relate to commodity hardware
• Understands the Hadoop Distributed File System, a file system designed to run on commodity hardware

Why this matters:

There are three major steps to deploying a big data solution: ingestion, storage, and processing. Knowing and understanding these three steps helps a big data engineer conceptualize the data capturing process. As such, a qualified candidate should understand each and every aspect of this process to be successful in this role.

What to listen for:

• Can confidently list the three steps core to deploying a big data solution
• Understands how each part of the big data solution deployment process is integral to efficiently capturing data
• Well-versed in big data concepts like ingestion, storage, and processing

Why this matters:

Overfitting is an error in modeling that occurs when a machine learning (ML) algorithm is trained against a limited, poor dataset. This results in an overly complex data model that’s more difficult to analyze and is incapable of performing correctly against proper data. A successful candidate must intuitively understand this concept to avoid overfitting in a work environment.

What to listen for:

• Confidence and familiarity in their knowledge on the concept of overfitting
• Adept understanding of why training an ML algorithm against an unsound dataset can lead to critical errors
• Knowledgeable in methods for structuring datasets that result in proper, streamlined data models

Why this matters:

Solving complex big data problems is what programmers in this role do on a daily basis. Being able to clearly communicate an example of when the candidate had to solve such a problem requires relevant experience and ideally a sense of what they learned from the obstacle. Candidates who can demonstrate their experience solving data challenges will be best prepared to succeed in this role.

What to listen for:

• Confident recollection of a previous experience solving a big data challenge
• A general, highly refined process for tackling difficult challenges
• Any lessons learned from the experience that can be applied toward quicker, more successful problem-solving in the future

Why this matters:

There are many software solutions and APIs that aid in solving specific big data challenges, but broad concepts in this field can apply in various contexts. Candidates who can confidently provide a data transition approach shaped by ample experience demonstrate their versatility, as well as extensive knowledge of the wider field of big data — not just software tools.

What to listen for:

• Experience taking data captured in one software context and applying it toward another toolset or solution
• Demonstrable ability to be dynamic when adapting to new toolsets and workflows
• Eagerness to take on new challenges as they arrive

What to listen for:

• Previous experience dealing with numerous pressing issues in a big data development role
• Ability to perform tasks to a significant degree of quality under time pressure
• Demonstrable areas where efficiency and technical proficiency have improved as a result of this experience

Why this matters:

Many feel a sense of ownership over their work, and may feel distressed when a colleague criticizes their ideas and approaches. Being able to constructively work alongside other engineers to overcome conflicts and create better, more efficient ways to solve technical problems is a valuable trait for any big data engineer candidate.

What to listen for:

• A positive attitude about overcoming conflicts with colleagues
• Able to explain the value of considering and integrating feedback from others when appropriate
• Knowledgeable of broader approaches to bridging gaps between people with different opinions

Why this matters:

As companies continue embracing remote work, issues such as power outages, poor internet infrastructure, and family troubles can interfere with an individual’s productivity. These problems can create bottlenecks in production schedules, so strong candidates should know how to communicate with colleagues and stakeholders to resolve these conflicts.

What to listen for:

• Willingness to be transparent about issues introducing conflict to normal workflows
• Calm, collected demeanor when facing a significant problem interfering with productivity
• A process for overcoming the problem, and mitigating the potential for the complication in the future

Why this matters:

Big data engineers develop software that handles large quantities of data. The development process often entails encountering and overcoming problems with the current codebase — but this can be a significant challenge and time investment. Having a systematic process in place for identifying and fixing the issue is pivotal for any big data engineer.

What to listen for:

• Strong problem-solving skills refined over years of development experience
• Thoughtful process used to search for and fix development problems as they arise
• Acknowledgement of areas to improve personal approach to fixing issues