Fathom

Backend or Full-Stack: - Drive development of what Andrej Karpathy calls the most underappreciated and yet most important portion of the ML Software Stack: building the human-machine labeling loop (https://vimeo.com/272696002) - Design and build the human-in-the-loop software that lets humans quickly clarify the points of confusion for the automated machine learning system. Intelligently integrate human judgment with what the machine “knows” with high confidence, while leveraging external knowledge bases to further help the human rapidly make decisions. - Construct a platform which must scale across many users, all while staying highly secure, and providing the analytics and controls to monitor both the users and the underlying machine learning, to ensure that all parts of the system (human and the machine) are operating both efficiently and effectively.

Data: Implement the build-out of a data pipeline to support scaled, long-term training of a language model on proprietary data: - Construct a pre-processing pipeline capable of cleaning and aggregating tens or hundreds of millions of documents - Extend our ML training systems to support interfacing with a highly scaled solution, such as Google’s TPUs, Google’s ML API distributed Tensorflow interface, or an internal Kubernetes-based distributed Tensorflow - Put in place the systems and monitoring to ensure successful training over an extended period of time (days or even weeks) - Implement analytics to track and determine efficacy of training against a wide variety of data sets

Infrastructure: Stand up a CI/CD pipeline that supports real-time ML inference: - Use Spinnaker to set up a highly scalable, red-black continuously deployment bake off - Implement robust canary testing to push sample data through “new” and “old” ML pipelines; generate and track the metrics (accuracy, throughput, etc.) to automate deployment of new code/models