Artificial intelligence (AI) is increasingly used by chemists and materials scientists in the design of new materials. How well do these methods work for guiding laboratory experimentation? In this talk, I will describe efforts to benchmark AI-based experiment planning methods for the exploratory synthesis of novel metal halide perovskitoid bulk solids, as well as related studies on crystallization modification additives and ligands for nanocrystals. I will describe how a combination of laboratory automation and comprehensive data capture enabled us to extract new scientific insights, develop and test new active and meta-learning algorithms, and compare a variety of algorithms in an experimental setting. I will describe some recent generalizations of this approach to perovskite nanocrystal ligand selection and rare-earth element separations.  Finally, I will comment on some of the limitations of existing AI methods for discovering new materials, and highlight possible research directions for improving the algorithms.