Consultant – Databricks Developer (Unity Catalog, Python, Spark, Kafka) | Genpact Careers
At Genpact, transformation isn’t a reaction — it’s our mission. In today’s dynamic digital era, we’re at the forefront, driving change with AI, data, and cutting-edge technologies. Through initiatives like our AI Gigafactory — the industry’s first — Genpact is delivering powerful, scalable solutions to global enterprises, helping them adapt faster, work smarter, and accelerate growth.
With more than 140,000 professionals across 30+ countries, Genpact (NYSE: G) blends deep industry knowledge with operational excellence and the latest in AI innovation to create tangible, lasting outcomes for enterprises. If you’re passionate about working in a fast-paced, tech-driven environment and love tackling real-world problems, this is your moment.
About Genpact
Genpact empowers some of the world’s top enterprises to navigate complex challenges with bold thinking and innovative solutions. We integrate process excellence with digital transformation to ensure our clients stay ahead in an ever-evolving business environment. Find out more about us at genpact.com and follow us on LinkedIn, X (formerly Twitter), YouTube, and Facebook.
Role Overview: Consultant – Databricks Developer (Unity Catalog, Python, Spark, Kafka)
In this role, you will be responsible for designing, developing, and maintaining robust ETL pipelines on the Databricks platform. Leveraging your expertise in Unity Catalog, Python, Spark, and Kafka, you’ll build scalable, real-time data solutions that drive critical business insights and operational efficiencies.
This role demands a blend of strong technical skills, data engineering expertise, and a proactive, problem-solving mindset to deliver best-in-class solutions.
Key Responsibilities
ETL Pipeline Development
-
Develop Scalable Pipelines: Design and maintain efficient ETL pipelines using Databricks, with a particular focus on Unity Catalog for enhanced data asset management.
-
Data Framework Implementation: Utilize Apache Spark for large-scale data transformation, aggregation, and optimization.
Real-Time Data Processing
-
Integrate Streaming Data: Build and maintain near real-time data processing systems by integrating Apache Kafka with Databricks workflows.
Orchestration and Governance
-
Workflow Automation: Develop and orchestrate data workflows using Databricks Workflows and other industry-standard orchestration tools.
-
Data Governance and Security: Implement data governance policies, set access controls, and enforce security protocols within Unity Catalog.
Performance Optimization
-
Pipeline Monitoring: Monitor the performance of data pipelines, troubleshoot issues, and implement optimizations to enhance scalability and efficiency.
-
Performance Tuning: Write optimized Python scripts and Spark-SQL queries to minimize costs and maximize performance.
Cross-Functional Collaboration
-
Partner with Analysts and Scientists: Collaborate with data scientists and business analysts to deliver data solutions aligned with business objectives.
-
Documentation and Lineage Tracking: Maintain comprehensive data documentation, including data dictionaries, data lineage, and governance frameworks.
Qualifications and Skills
Minimum Qualifications
-
Educational Background: Bachelor’s degree in Computer Science, Data Engineering, or a related field.
-
Technical Expertise:
-
Solid experience in data engineering with a strong focus on Databricks development.
-
Proven skills in Databricks, Unity Catalog, and data lake management.
-
Advanced programming proficiency in Python for ETL development and automation.
-
Hands-on experience with Apache Spark for distributed data processing.
-
Practical knowledge of Apache Kafka for real-time data streaming and event-driven architectures.
-
Strong SQL skills for querying, data transformation, and pipeline development.
-
-
Additional Skills:
-
Deep understanding of data governance, security protocols, and data quality standards.
-
Solid experience in implementing Databricks Lakehouse architecture and concepts.
-
Expertise in Delta Lake, dbConnect, Databricks APIs, and Databricks workflows orchestration.
-
Hands-on experience with both batch and streaming data pipelines.
-
Familiarity with Spark and Hive-based data processing frameworks.
-
Cloud platform experience (Azure, AWS, or GCP) and related services such as ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, and cloud databases.
-
Strong skills in writing unit tests and integration tests to ensure code quality.
-
Excellent communication skills with experience working in teams of 5+ members.
-
High proficiency in data structures and algorithms.
-
Preferred Qualifications
-
Governance Expertise: Experience with Unity Catalog and basic data governance.
-
Advanced Databricks Knowledge: Familiarity with Databricks SQL Endpoint.
-
CI/CD Practices: Experience with building CI/CD pipelines for Databricks jobs.
-
Migration Projects: Exposure to migration projects for building unified data platforms.
-
Additional Tools: Familiarity with DBT, Docker, and Kubernetes is an added advantage.
-
Innovation Mindset: A strong attitude towards continuous learning and upskilling.
Why Choose Genpact?
At Genpact, we empower transformation leaders by giving them the tools, support, and environment to thrive.
1. Lead at the Cutting Edge
Work with emerging technologies such as AI, automation, and digital solutions to lead the future of enterprise transformation.
2. Drive Meaningful Impact
Tackle real-world business problems that matter. Build solutions that drive substantial outcomes for global enterprises.
3. Accelerate Your Growth
Take advantage of mentorship programs, hands-on projects, and continuous learning platforms designed to propel your career forward.
4. Collaborate with the Best
Join a team of 140,000+ professionals who are redefining what’s possible across industries.
5. Thrive in a Values-Driven Culture
Integrity, respect, curiosity, and inclusion form the foundation of Genpact’s work culture. Here, your voice matters and your ideas fuel progress.
Equal Opportunity Commitment
Genpact is an Equal Opportunity Employer. We are committed to fostering a diverse and inclusive work environment where respect, integrity, and innovation thrive. All qualified applicants are considered for employment without regard to race, religion, gender, age, disability, or any other protected status.
Important Notice: Genpact does not charge fees at any stage of the hiring process. Please be cautious of recruitment scams requesting payments for application, training, or recruitment services.
Job Summary:
-
Job Title: Consultant – Databricks Developer (Unity Catalog + Python, Spark, Kafka for ETL)
-
Primary Location: Hyderabad, India
-
Job Category: Full-Time
-
Education Requirement: Bachelor’s Degree or Equivalent
-
Posting Date: June 9, 2025
-
Application Status: Ongoing
Build the Future with Genpact
If you are ready to innovate, solve complex problems, and build the next generation of enterprise data platforms, Genpact is the place for you. Step into a future where your expertise drives innovation and transformation at scale.