Skill: Data Engineer
- Design, develop, monitor, and operate data pipelines and maintenance of existing data pipelines.
- Collaborating with product owner and analysts to provide data-driven solutions and insights.
- Implementing data transformations and aggregations for large datasets.
- Ensuring data security and compliance with data governance policies.
- Developing in languages like Pyspark and Python.
- Preprocess of structured and unstructured data.
- Analyze, transform, and create SQL queries.
- Developing and optimizing Spark-based data processing applications on the AWS Databricks platform.
- Troubleshooting and debugging data-related issues on the AWS Databricks platform.
- Integrating Databricks with other data storage and processing systems.
What we are looking for:
- Expertise in Apache Spark along with Spark streaming & Spark SQL.
- Highly proficient with Apache Spark, Scala/ Python and PySpark.
- Good understanding & hands-on experience Gitlab with CI/CD.
- Hands-on experience in AWS Services like S3, Glue is required.
- Working knowledge of AWS Databricks, Cluster Management.
- Design, build, and test data pipelines and workflows using AWS Databricks.
- Having the knowledge about Optimizing AWS Databricks Streaming jobs.
- Have a deep understanding of AWS Cloud platforms and data engineering technologies.
- Strong SQL skills to perform data analysis and understanding of source data.
- Familiarity with NoSQL databases and JSON formats.
- Experience in building ETL / data warehouse transformation processes.
- Strong knowledge of Data Management principles.
- Good to have experience in analytical tool.
- Good to have experience in Alteryx/Tableau.
#LI-NR3
More Information
Application Details
-
Organization Details
TCS / Tata Consultancy Services
Recommended Comments
There are no comments to display.
Join the conversation
You are posting as a guest. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.