Senior Software Engineer (Information Technology)
As an organization, Black Book is open, transparent, and very customer focused. Leaders in the organization consistently showcase the following key attributes and look to build teams that embrace these qualities every day:
- Shift Left Mindset
- Constant desire for self-improvement and growth
- Embracing change and ambiguity
- Strong sense of ownership, focusing on accountability
- Oriented around the internal and external customer experience
- Foster and celebrate an inclusive environment
Us
Black Book: Black Book provides vehicle information and valuations to dealerships, financial institutions, fleet owners and other companies in the automotive industry. We develop APIs and web and mobile applications that provide insights on a variety of motor vehicles of various markets.
Web Crawler Team: We are seeking a highly skilled software engineer to join Black Book's Web Crawler team. The main responsibility of the team is to build, enhance, support, and innovate on a key automotive retail listings web crawler project. The ideal candidate will have extensive experience with Scrapy and Python for web scraping, and strong knowledge of PostgreSQL and SQLAlchemy for database management. Proficiency in browser automation using Playwright and Selenium is crucial, alongside expertise in Redis for data storage, and ElasticMQ for local message queuing and AWS SQS in the cloud. Familiarity with Kubernetes and Docker Desktop for deployment, and competency in using Poetry for dependency management, are also required. This is an exciting opportunity for someone who thrives in a fast-paced environment and is enthusiastic about technological innovation and problem-solving.
You
You are comfortable working with developers and members outside your core team to achieve success. You enjoy taking opportunities to fix problems, mentor your peers, and like to step outside your comfort zone to develop your skillset.
Responsibilities:
- Develop and maintain secure, efficient, and scalable web scraping solutions using modern technologies.
- Analyze, design, and implement strategies for robust data extraction and processing systems.
- Ensure compliance with web scraping ethics, privacy, and legal standards.
- Troubleshoot, debug, and resolve complex issues in web crawling processes effectively.
- Optimize and refactor crawling solutions for enhanced performance and scalability.
- Monitor, maintain, and support current web crawling systems to ensure reliability and efficiency.
- Collaborate with cross-functional teams to integrate data solutions with stakeholder needs.
- Engage in rigorous peer review and code review sessions to ensure high code quality standards.
- Develop and execute comprehensive automated testing frameworks for web scraping solutions.
- Stay updated on the latest trends and advancements in web scraping and data extraction technologies.
- Proactively address and resolve challenges with innovative and agile solutions.
- Demonstrate ownership and accountability for project milestones and deliverables.
- Provide clear documentation and training for team members on web crawling solutions and best practices.
What You Will Bring
- Bachelor's degree in computer science or equivalent work experience in software development.
- 5+ years of experience in developing web-based solutions, proficient in Python and modern technologies.
- Demonstrable expertise in web scraping frameworks, such as Scrapy, and browser automation tools like Selenium and Playwright.
- Solid understanding and practical experience with PostgreSQL for relational database management.
- Skilled in deploying and managing applications in Docker and Kubernetes environments.
- Proficiency in utilizing cloud-based services, specifically AWS SQS, with exposure to ElasticMQ for local development.
- Experience in implementing and maintaining Continuous Integration/Continuous Deployment pipelines.
- Seasoned in both unit and integration testing methodologies to ensure code quality and reliability.
- At least 2 years of working in agile environments, promoting collaboration and continuous improvement.
- Experience with version control systems, particularly Git, to manage and track code changes.
- Excellent communication skills - the ability to think creatively and adapt the message to the audience
- Strong problem-solving skills, proactive, adaptable, with the ability to take initiative and drive projects forward.
Helpful Experience (nice to have):
- Experience working with Angular, React, Next.js, and NestJS.
