1074 data engineer pyspark jobs found

TCS Inviting Application for Azure Data Engineer (Pyspark) Experience- 4 to 10 yrs Joining Location - Pan India JD- Minimum 5 years of PySpark Development experience, especially in Spark SQL and Complex Transformations Minimum 3 years of Python development experience Minimum 2 years of experience in ADF

Company Description About Sopra Steria Sopra Steria, major Tech player in Europe recognised for its consulting, digital services and software development, helps its clients drive their digital transformation and obtain tangible and sustainable benefits. It provides end-to-end solutions to make large companies and organisations more competitive by combining in-depth knowledge of a wide range of business sectors and innovative technologies with a fully collaborative approach. Sopra Steria places people at the heart of everything it does and is committed to putting digital to work for its clients in order to build a positive future for all. With 50,000 employees in nearly 30 countries, the Group generated revenue of €5.1 billion in 2022. The world is how we shape it. Job Description We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine...

Company Description Sopra Steria, a major Tech player in Europe with 56,000 employees in nearly 30 countries, is recognized for its consulting, digital services and software development. It helps its clients drive their digital transformation and obtain tangible and sustainable benefits. The Group provides end-to-end solutions to make large companies and organizations more competitive by combining in-depth knowledge of a wide range of business sectors and innovative technologies with a fully collaborative approach. Sopra Steria places people at the heart of everything it does and is committed to putting digital to work for its clients to build a positive future for all. In 2023, the Group generated revenues of €5.8 billion. The world is how we shape it. Job Description We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine learning models....

Company Description About Sopra Steria Sopra Steria, major Tech player in Europe recognised for its consulting, digital services and software development, helps its clients drive their digital transformation and obtain tangible and sustainable benefits. It provides end-to-end solutions to make large companies and organisations more competitive by combining in-depth knowledge of a wide range of business sectors and innovative technologies with a fully collaborative approach. Sopra Steria places people at the heart of everything it does and is committed to putting digital to work for its clients in order to build a positive future for all. With 50,000 employees in nearly 30 countries, the Group generated revenue of €5.1 billion in 2022. The world is how we shape it. Job Description We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer, you will collaborate closely with our Data Scientists to develop and deploy machine...

Company Description Visa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and government entities in more than 200 countries and territories each year. Our mission is to connect the world through the most innovative, convenient, reliable, and secure payments network, enabling individuals, businesses, and economies to thrive while driven by a common purpose - to uplift everyone, everywhere by being the best way to pay and be paid. Make an impact with a purpose-driven industry leader. Join us today and experience Life at Visa. Job Description The Senior Data Engineer role involves leading and mentoring a team of data engineers, coordinating with other teams to manage and prioritize projects, and driving the strategy, roadmap, and execution of key data engineering initiatives. The role requires understanding and translating business needs into data models, creating...

Our Purpose We work to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart and accessible. Using secure data and networks, partnerships and passion, our innovations and solutions help individuals, financial institutions, governments and businesses realize their greatest potential. Our decency quotient, or DQ, drives our culture and everything we do inside and outside of our company. We cultivate a culture of inclusion for all employees that respects their individual strengths, views, and experiences. We believe that our differences enable us to be a better team - one that makes better decisions, drives innovation and delivers better business results. Title and Summary Senior Software Engineer - Data Engineer / Data Science - ML, AI, Hadoop, Pyspark Who is Mastercard? Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy...

The Applications Development member (data engineering senior programmer) is an intermediate level position responsible for participation in the establishment and implementation of new or revised data platform eco systems and programs in coordination with the Technology team. The overall objective of this role is to contribute to data engineering scrum team to implement the business requirements: Responsibilities: Build and maintain batch or real-time data pipelines in data platform. Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources. Develop ETL (extract, transform, load) processes to help extract and manipulate data from multiple sources. Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users Automate data workflows such as data...

Technical Lead Qualification : Bachelors in science , engineering or equivalent Responsibility : Project Planning and Setup : Understand the project scope, identify activities/ tasks, task level estimates, schedule, dependencies, risks and provide inputs to Module Lead for review. Provide inputs to testing strategy, configuration, deployment, hardware/software requirement etc. Review plan and provide feedback on gaps, timeline and execution feasibility etc as required in the project. Participate in KT sessions conducted by customer/ other business teams and provide feedback on requirements. Requirement Understanding and Analysis : Analyze functional/non functional requirements and seek clarifications for better understanding of requirements. Based on understanding of system upstream & downstream, provide feedback and inputs on gaps in requirements and technical feasibility of requirements. Design : Prepare the LLD/ detailed design documents...

Job Title: Data Engineer with Pyspark, Azure Synapse About Us "Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount . With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery. WHY JOIN CAPCO? You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry. MAKE AN IMPACT Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial...

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Senior Associate Job Description & Summary A career within Data and Analytics services will provide you with the opportunity to help organisations uncover enterprise insights and drive business results using smarter data analytics. We focus on a collection of organisational technology capabilities, including business intelligence, data management, and data assurance that help our clients drive innovation, growth, and change within their organisations in order to keep up with the changing nature of customers and technology. We make impactful decisions by mixing mind and machine to leverage data, understand and navigate risk, and help our clients gain a competitive edge. Creating business intelligence from data requires an understanding of the business, the data, and the technology used to store and analyse that data. Using our Rapid Business Intelligence...

We are seeking a Data Engineer with 2 to 4 years of experience, specializing in SQL, Python, Delta Lake (Delta Tables), and ETL development on Databricks. She/he would work in a Data Mesh architecture, with opportunities to design and manage data products. Knowledge of the Data Mesh framework and its principles would be beneficial. Key Responsibilities: Design, develop, and maintain ETL pipelines using Python, Spark and Databricks, with a focus on Delta Lake for real-time and batch data processing. Ensure efficient data integration from multiple sources and maintain the reliability of data flows, working closely with cross-functional teams. Work within a Data Mesh architecture, contributing to the creation and management of data products. Optimize performance and scalability of data pipelines, ensuring high-quality data delivery. Collaborate with data scientists, analysts, and other stakeholders to ensure the successful implementation of data requirements. Communicate...

The Applications Development Senior Supervisor is an intermediate management level position responsible for providing full leadership and direction to a team of employees in an effort to establish and implement new or revised data platform eco systems and programs in coordination with the Technology team. The overall objective of this role is to lead data engineering systems analysis and programming activities. Responsibilities: Build and maintain batch or real-time data pipelines in data platform. Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources. Develop ETL (extract, transform, load) processes to help extract and manipulate data from multiple sources. Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users Automate data...

Data Engineer - Azure Databricks - Sunderland - £65K I'm looking for a Data Engineer to work on developing products and solutions in Azure. You will have the opportunity to up skill in complex areas of cloud based engineering solutions in the very latest tech. Within their industry, they are certainly one of the most advanced in terms of where they're at tech wise and looking for someone to come in and take this to the next level. This is a very exciting time to join as they shake things up across the industry, so please get in touch asap to hear more! Role & Responsibilities Design and Architecture of Azure Data solutions Be the technical expert with the latest technologies. Make recommendations to the wider business as to how to keep up to date with the best data solutions. Working with and modelling data warehouses. Skills & Qualifications Strong technical expertise using SQL Server and Azure Data Factory for ETL Solid experience with Databricks, PySpark etc....

Data Engineer Lead I - Software Engineering Who We Are: Born digital, UST transforms lives through the power of technology. We walk alongside our clients and partners, embedding innovation and agility into everything they do. We help them create transformative experiences and human-centered solutions for a better world. UST is a mission-driven group of 29,000+ practical problem solvers and creative thinkers in more than 30 countries. Our entrepreneurial teams are empowered to innovate, act nimbly, and create a lasting and sustainable impact for our clients, their customers, and the communities in which we live. With us, you'll create a boundless impact that transforms your career-and the lives of people across the world. Visit us at UST.com. You Are: The Data Engineer will be instrumental in building and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The ideal...

Data Engineer Associate II - Data Science Who We Are: Born digital, UST transforms lives through the power of technology. We walk alongside our clients and partners, embedding innovation and agility into everything they do. We help them create transformative experiences and human-centered solutions for a better world. UST is a mission-driven group of 29,000+ practical problem solvers and creative thinkers in more than 30 countries. Our entrepreneurial teams are empowered to innovate, act nimbly, and create a lasting and sustainable impact for our clients, their customers, and the communities in which we live. With us, you'll create a boundless impact that transforms your career-and the lives of people across the world. Visit us at UST.com. You Are: UST is searching for a Data Engineer who will work with large and complex data sets, provide, and solve routine and non-routine problems by applying necessary analytical methods. The...

Data Engineer Lead I - Software Engineering Who We Are: Born digital, UST transforms lives through the power of technology. We walk alongside our clients and partners, embedding innovation and agility into everything they do. We help them create transformative experiences and human-centered solutions for a better world. UST is a mission-driven group of 29,000+ practical problem solvers and creative thinkers in more than 30 countries. Our entrepreneurial teams are empowered to innovate, act nimbly, and create a lasting and sustainable impact for our clients, their customers, and the communities in which we live. With us, you'll create a boundless impact that transforms your career-and the lives of people across the world. Visit us at UST.com. You Are: The Data Engineer will be instrumental in building and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The ideal...

Data Engineer Lead I - Software Engineering Who We Are: Born digital, UST transforms lives through the power of technology. We walk alongside our clients and partners, embedding innovation and agility into everything they do. We help them create transformative experiences and human-centered solutions for a better world. UST is a mission-driven group of 29,000+ practical problem solvers and creative thinkers in more than 30 countries. Our entrepreneurial teams are empowered to innovate, act nimbly, and create a lasting and sustainable impact for our clients, their customers, and the communities in which we live. With us, you'll create a boundless impact that transforms your career-and the lives of people across the world. Visit us at UST.com. You Are: The Data Engineer will be instrumental in building and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The ideal...

Role Proficiency: Act creatively to develop applications by selecting appropriate technical options optimizing application development maintenance and performance by employing design patterns and reusing proven solutions. Account for others' developmental activities; assisting Project Manager in day to day project execution. Outcomes:< < Interpret the application feature and component designs to develop the same in accordance with specifications.< Code debug test document and communicate product component and feature development stages.< Validate results with user representatives integrating and commissions the overall solution.< Select and create appropriate technical options for development such as reusing improving or reconfiguration of existing components while creating own solutions for new contexts< Optimises efficiency cost and quality.< Influence and improve customer satisfaction< Influence and improve employee engagement within the project...

Responsibilities You will need to be well spoken and have an easy time establishing productive long lasting working relationships with a large variety of stakeholders • Take the lead on data pipeline design with strong analytical skills and a keen eye for detail to really understand and tackle the challenges businesses are facing • You will be confronted with a large variety of Data Engineer tools and other new technologies as well with a wide variety of IT, compliance, security related issues. • Design and develop world-class technology solutions to solve business problems across multiple client engagements. • Collaborate with other teams to understand business requirements, client infrastructure, platforms and overall strategy to ensure seamless transitions. • Work closely with AI and A team to build world-class solutions and to define AI strategy. • You will possess strong logical structuring and problem-solving skills with expert level understanding of database and have an...

Your Role and Responsibilities As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too. Being You @ IBM IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual...

Your Role and Responsibilities As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too. Being You @ IBM IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual...

Your Role and Responsibilities As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too. Introduction In this role, you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients...

Your Role and Responsibilities As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too. Being You @ IBM IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual...

Your Role and Responsibilities As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs. Your primary responsibilities include: Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements. Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization. Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too. Being You @ IBM IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual...