Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

RmR2SnhRcUxGREpibGVKRG1aMmNZYVpNa2c9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

NetHire

Sales Representative B2B - Remote Job at NetHire

 ...B2B Sales Representative - Remote We help companies hire the people they need with job postings and recruitment services This is...  ...them with a solution Try to close in one call Schedule an online presentation and or send email information to close sales Follow... 

BEM Systems, Inc.

Environmental Permitting Specialist/Ecologist Job at BEM Systems, Inc.

 ...and Sections 9 and 10 of the Rivers and Harbors Appropriation Act. Experience with the New York State Department of Environmental Conservation (NYSDEC), United States Coast Guard (USCG), NJ Pinelands Commission (Pinelands), Delaware and Raritan Canal Commission (DRCC),... 

TJ Maxx

Customer Experience Coordinator Job at TJ Maxx

TJ MaxxAt TJX Companies, every day brings new opportunities for growth, exploration, and achievement. You'll be part of our vibrant team...  ...global Home Offices, Distribution Centers or Retail Stores--TJ Maxx, Marshalls, Homegoods, Homesense, Sierra, Winners, and TK Maxx,... 

Two Maids - Gilbert

Day Shift Entry Level Housecleaner Job at Two Maids - Gilbert

 ...tools, and support you need to truly do your best. * Our PHC position is perfect for anyone looking to start a career, students in night school, working moms looking for work-life balance, or for anyone that just simply loves to clean! Why you should be a part of our... 

The Principal Financial Group

Director, Investment Funds and Financial Risk Management - Asset Management Job at The Principal Financial Group

 ...Were looking for a Director, Investment Funds and Financial Risk Management to join our Asset Management Risk team. In this role, youll support the execution of the second line global investment risk management program that meets business objective Management, Risk, Director...