Big data engineer
Work at the cutting edge of technology and maths and use your analytical skills to solve complex problems.
About the job
What it's like
More than 2.5 quintillion bytes of data are created every day. As a big data engineer, you’ll develop software that collects and analyses large quantities of information. Sometimes, you’ll use Artificial Intelligence (AI) to help you do this.Â
Businesses can then use this to inform decisions or predict behaviour. It’s how websites like YouTube or Netflix show you recommendations based on things you’ve watched before. They use AI software to tap into banks of data, created by a big data engineer, to learn about your preferences and habits. Â
You’ll work with organisations to help them develop, maintain, test and evaluate big data solutions.  Your work will involve collaborating with data architects, data scientists and analysts. You’ll make sure any solution meets the business’ needs. Â
As a big data engineer, you might:Â
create and maintain complex computer systems that use AI to collect dataÂ
programme software to process high volumes of important informationÂ
use AI-powered tools to design mathematical algorithms that simplify data analysisÂ
work with data architects, analysts and data scientistsÂ
make sure the data solutions you build meet all the business needs or requirementsÂ
You could work in a variety of industries, including:Â
finance, including financial technology (fintech)Â
ITÂ
retailÂ
sales and marketingÂ
healthcareÂ
manufacturingÂ
educationÂ
Explore more information about this job
Here are some useful links to learn more about this career:
Like the sound of this career?
Browse courses in Scotland related to 'Big data engineer'
Select qualification level(s)
Other careers that you might like
-
Systems analyst -
Solution architect -
IT project analyst -
Infrastructure engineer -
Systems engineer -
Data visualisation specialist -
Internet-of-things architect -
Cloud services engineer -
Cyber security architect -
Data architect
Related industries
Many jobs can be done in lots of different industries. We've highlighted the ones we think are most important for this job.
-
Financial and professional services -
Energy -
Digital technologies
Top skills
Skills are things you're good at. Whether you know what yours are or not, everyone has them!
It's useful to learn which ones are important in a job so you know the areas you need to brush up on. It can also help you work out if you're suited to a career.
Meta skills
Here are some of the meta skills you'll need to do this job.
- analysing
- attention to detail
- researching
- working with numbers
- problem solving
- working with technology
Your skills are important
Our unique skillsets are what make us stand out from the crowd. Learn about each skill in depth and discover what employers look for in your applications and interviews.
Getting in
Explore each section to find more information about getting into this career.
Colleges and universities will list subjects you'll need for entry to a course. Some useful subjects include:
-
Business Management
-
Economics
-
Maths (Mathematics)
-
Statistics
-
Foundation Apprenticeship: Software Development
-
Applications of Mathematics
You can get a head start in this career by doing a Foundation Apprenticeship in S5 and S6.
You'll get an SCQF level 6 qualification which is the same level as a Higher. You'll also learn new skills and gain valuable experience in a work environment.
Discover what's on offer at your school on Apprenticeships.scot.
There are many ways to get qualified for this job. You could go to college, university or get work-based qualifications, like an apprenticeship.
Many big data engineers will a Higher National Diploma (HND) or degree in maths and technology fields such as:
Computer/Data Science
Computer Engineering
Maths or maths related subjects
Information Technology
Postgraduate data engineering courses are available. A background in a related field such as business, quantitative science or computing is helpful when applying.
Not all employers list specific qualifications they need. They they might ask for relevant experience, usually work based, that demonstrates a range of transferable skills.
You'd need to show up-to-date knowledge and experience of software and programming developments. It's also helpful to have this for many scripting languages.
Find the right course for you
Browse courses in Scotland related to 'Big data engineer'
