How to Use ML-Assisted Tooling to Benefit from ML Model/Human Synergy
Here’s how machine learning-assisted tooling works—and how it’s being applied in different industries.
As machine learning matures, the advances are changing the way businesses interact with ML models. Among the most promising of these advances: ML-assisted tooling. Rather than treat an ML model as a black box in a larger system, to be queried when predictions are needed, ML-assisted tooling introduces the idea that humans and ML systems should work together to produce the best results.
ML models can identify complex relationships far faster than humans can, providing powerful insights across numerous applications, while humans can identify nuance and make decisions on a level that ML models can’t achieve. Additionally, while ML models typically fail in systematic ways, the errors they produce are exactly the type of failures that humans are best equipped to catch.
ML-assisted tooling provides the best of both worlds: Humans benefit from the vast computational abilities of ML models while still providing the final insight when evaluating decisions.
ML-assisted tooling uses specific types of ML—active learning and reinforcement learning—to allow the model to learn not only from data, but also from human actions. ML-assisted tooling can speed up labor-intensive processes and can augment decision making when you can’t safely relegate decisions to an algorithm.
Many observers see ML-assisted tooling as the next stage of ML products, and these tools will only get better with time. Here’s what you need to know about ML-assisted tooling and how it’s being used across a wide variety of industries.
Applications of ML-Assisted Tooling
We all interact with ML-assisted tools every day without realizing it. These are just a few examples.
Recommendation Systems
“If you enjoyed this book, you might also like …” and “based on your viewing history, you will probably enjoy …” are the sorts of messages we’ve all seen on sites including Amazon and Netflix. But without input from a human, they wouldn’t function at all.
ML-assisted tooling plays a crucial role in creating well-designed recommendation systems. Fully automated recommendation systems typically suggest content based on user engagement, so they can be biased toward sensationalist content. When YouTube identified a bias issue with its recommendation system, it pivoted to an ML-assisted tooling approach by introducing human moderation into the equation.
YouTube relies on human evaluators to assess recommended content to determine whether the channels and videos being suggested are a quality source of information. Evaluators assign videos a quality score based on whether they are intended to relay information, and if so, the quality of the information they provide.
Videos with high quality scores are more likely to be promoted by the recommendation algorithm. Alternatively, videos with lower ratings, such as conspiracy theory videos or videos spreading misinformation, are demoted in the recommendations.
Content Creation
Many businesses use ML-assisted tooling to help users write emails and documents. The autocomplete in Google’s email composer is a popular example of this. As users type their email, the technology provides possible completion options, making email composition easier and faster.
Google has also incorporated this technology into Google Docs as the Smart Compose feature. Like the email composer, this tool predicts what the user will type next and provides autocomplete suggestions. It also identifies and corrects mistakes as the user is typing. These suggestions and corrections help users to write better documents more quickly.
Data Labeling
ML-assisted tooling can also dramatically speed up the process of labeling and segmenting data. Creating a dataset for a task such as autonomous driving typically involves semantic segmentation, where a human identifies and labels every pixel in every image. Manually tracing the outlines of complex objects can be time-consuming, and tracing the outline of just one car typically takes about one minute.
With ML-assisted tools, this process can be sped up drastically. With such tools, users draw a bounding box around an item to be labeled and the tool auto-segments the object. Users can adjust the outline if needed, and then they just need to add the semantic label.
Compared to the minute it takes for a person to outline a car manually, it typically takes just three seconds to draw a border around the box and label it as a car.
Gaming
ML-assisted tooling can even be useful for game development, allowing designers to discover game imbalances without months of play-testing. Game designers typically balance a game by going through thousands of play-testing sessions with users, redesigning the game based on feedback, and then repeating that process until the designers are satisfied with the game.
With ML-assisted tooling, however, game designers can drastically speed up this process. At Google, a team developed a prototype for a multiplayer card game called Chimera. They trained AIs to play the game, making it possible to simulate millions of games with different player decks.
With this technique, they were able to run through a huge number of play-testing sessions in a very small window of time, making it easy for the team to quickly identify and address imbalances in the deck. After making appropriate changes, they were able to test out the AIs on the new deck. This process allowed the team to balance their game much faster than with traditional play-testing.
Medicine
Medicine has seen a revolution in the past decade thanks to ML, and it is now beginning to embrace the benefits of ML-assisted tooling as well. Two years ago, DeepMind made history when its AI, called AlphaFold 2, solved the decades-old biochemistry problem of folding proteins. Since then, investment in startups designing and discovering drugs has exploded.
Drug discovery with AI follows a standard process. First, scientists provide input for some characteristics of the drug they are looking for; for example, Exscientia helped healthcare company Sanofi in 2019 to find a drug specifically for metabolic pathways.
Here’s how the process works: They let the AI develop candidate drugs, which are then tested in a wet lab for the desired properties. This goes through a few iterations until they end up with several promising candidates that can then be used for further testing.
Previously, this process took a decade and billions of dollars. Now, with AI filtering drug candidates, it can take a fraction of the time and cost. This approach also makes it possible to search through molecules on a scale that simply isn’t possible with brute-force screening, unlocking even more possibilities for drug development.
In addition to advancing drug development, ML-assisted tooling is streamlining medical image analysis. Microsoft Research’s Project InnerEye, for example, uses ML to provide tools for analyzing and segmenting 3D medical images. Medical professionals in the radiology field are using this technology to help plan treatments.
Rather than manually drawing contour lines around tumors and organs in the 3D images, specialists can instead use InnerEye to automatically add contours to the images. They can then update the proposed contours as needed to ensure their accuracy. Microsoft’s research study showed that their tool allowed clinicians to segment images up to 13 times faster.
Robotics
ML-assisted tooling is also changing the way that people work with industrial robotics. The Veo FreeMove, for example, is a collaborative robot that allows humans to work safely within a defined area called a “work cell.” These robots use sensors to identify objects in the work cell, and if a person moves too close to the robot, the robot automatically stops.
Allowing a person to remain in the work cell can make tasks like palletizing more efficient. Industrial robots occasionally experience faults, preventing the robot from proceeding with its task until the fault is manually fixed. For safety, noncollaborative robots must operate within a locked cage, so their recovery time after a fault is often long, leading to a window of lost productivity.
Fixing a fault is much faster when working with a collaborative robot. The human is already in the work cell and can fix the fault, and the robot can quickly resume palletizing without safety issues.
Getting the Best of Both Worlds, Human and ML
ML algorithms are limited in their ability to function entirely autonomously. With ML-assisted tooling, companies can benefit from the power of ML technology, while still allowing humans to make the final decisions.
ML techniques help speed up labor-intensive processes such as data labeling and segmentation and help to improve the quality of technologies such as recommendation systems and content moderation. Having a human in the loop can also help address limitations of current technologies, such as the tendency of industrial robots to experience faults.
Across industries, ML-assisted tooling allows companies to produce better results. ML-assisted tools can accomplish the tedious work that is more effectively performed by machines, while freeing up humans to look at the big picture and provide valuable insights in the decision-making process.