“Oh, so it’s like Shazam, but for art?” a friend asked after I spent the past two minutes explaining my latest project.
It was a platform that could help art enthusiasts identify which artist made a painting and she had just explained it in 5 words; ‘Like Shazam, but for art’.
This article is about why I started the project and also how I built the first version.
Why?
I’ve loved art all my life.
I’ve spent years researching art movements, making & exhibiting my own artwork, and generally hanging out around artists and galleries. Over time, I developed the ability to guess (with high levels of accuracy) which artist made a painting, especially if it was an African artist.
In the AI renaissance of 2022, I got inspired and thought it would be fun to see if I could train computers to achieve the same thing, so I started a small personal project.
While working on it, I quickly realised that there were bigger problems. Problems around the documentation, recognition and sharing of African art knowledge. For example, there are old African masters whose paintings are difficult to find online. You have to go to old art textbooks or personal collections to find them. I became concerned that as the world moves fully digital, a lot of these works may be lost in history.
Experiencing this challenge made me decide to make the project public and extend it to digitally preserving and sharing knowledge on African art and artists.
It’s focused on African art because I knew that no one else would do it, but it had to be done, so it had to start with me.
How it works
Using the platform is simple.
You upload a photo of a painting or choose one of the sample images available.
The platform processes the image, identifies the artists whose art styles are most similar to the uploaded artwork, creates a confidence score for each artist based on how similar the uploaded photo is to their art style and selects the artist with the highest confidence score.
Information about the artist is then displayed in the format below.
The confidence score is publicised to show you how it came to that prediction and to allow you to decide how you want to use the result provided.
There’s also a feedback section (thumbs up or down) so that you can contribute to the project and help improve the model.
This is a simplified explanation of how the platform works. Below is more information (with as little technical jargon as possible) on how it was built.
The Tech / Data preparation
Data gathering and processing
This wasn’t my first ML project so I knew that the juice was going to be in the training data. I had to find and prepare enough high-quality data for the model in order to get good results. That turned out to be harder than I thought.
I started collating and categorising training data but quickly realised that it would take a lot of time so I got my sister and two interns to work on this with me for some months. (They were paid for their work)
The next step was cleaning and standardising the data.
- Removing duplicates and normalizing the data
- Ensuring that the data was correctly labelled
- Ensuring that each artist had roughly the same amount of training data to avoid bias in the model due to class imbalance.
- Ensuring that training data on each artist included enough variation of their different painting styles and mediums to avoid overfitting to a certain style/medium from an artist.
- Ensuring that the training photos had paintings in different contexts. Photos of paintings in different environments, photos of paintings upside down, black and white versions of the paintings etc. This diversity would help the model perform well in most situations.
It was a lot of work but getting this right would make the model perform well.
Once the data preparation process was complete, I split the data into training data and testing data (control) to make it easier to use and to verify accuracy.
Building/Training the Models
When thinking of the models, I had two options.
Build on an open-source foundation model or rely on a Cloud-based proprietary model e.g. GCP’s AutoML.
I attempted both approaches, but since the project was initially supposed to be a proof of concept, cost management was the biggest factor for me to consider.
I eventually settled on using the open-source foundation model, EfficientNet. It’s free to use, It’s good at identifying features in data, it doesn’t require a lot of compute to run, it offers flexibility in deployment and I had sort of used it in the past.
I found a Tensorflow implementation of EfficientNet and used that as the backbone of the model. It had support for Keras so that also helped to manage complexity too.
It was efficient enough that I could train test versions on a Macbook M1 Pro with 16GB of RAM (I had to deal with my laptop occasionally freezing up though).
Under the hood, the model processes the photos to identify key features like brush strokes, colour schemes, composition etc, and condenses the information to keep the important parts. It uses this information to identify similarities between newly provided photos of paintings and it’s training data.
I experimented with various epoch values and early stopping to see what worked best and I eventually landed on interesting metrics.
I continued to add more artists and experiment with different values to get something accurate enough to launch with.
The final implementation involved two models. One for the painting style recognition, and another for the painting medium recognition.
I got help refining the models further and eventually deploying them with API wrappers.
Design & Development of the application
Once the models were sorted, the rest of the platform was straightforward to build. The final implementation looked like this:
Platform Backend/API: I built the backend/API with Laravel to handle logic and routing requests to the actual models efficiently. This layer handles all the image processing, storage, validation, artist information, and request triage i.e controlling requests sent to the models.
Frontend: The frontend was built with React. I really don’t like writing front-end code so I got help with building this part. It basically controls the display of information and user requests sent to the backend.
Server/Hosting: I hosted the entire application on Heroku (models included). I picked Heroku because it allowed me to get a lot of ready-built services and support at a manageable cost. After months of work to gather data, train the models and build the core platform, I didn’t want any more work, I just wanted to go live!
Final Notes
Ethics
I trained the models solely on publicly available data.
As I developed the project, I also checked in with various artists to see if they had any issues with the project and I didn’t receive any pushback. However, if any artist that I have included in the project wants to be excluded, please let me know. I will do so immediately.
Future plans
I see a lot of use cases for this in education, art discovery, tourism and commerce. There are also a couple of features I’ve been thinking about; an artist directory, web-based art games, Artist recommendations and so much more.
My plan is to continue improving the project and learning what artists, industry stakeholders and art enthusiasts want to see next.
I’m really happy I started this project. On one hand, it gave me a unique opportunity to combine two things I’m passionate about (art + tech), on the other hand, it challenged me and forced me to think creatively around various problems. I also feel like I’m actively contributing to the growth and preservation of African art, and that feels great.
Project link(beta): https://beta.whomadethisartwork.com/
Note*: The beta version is restricted to 100 African artists with plans to include more soon.*
Demo:
If you enjoyed reading this article it’s because I used a writing technique called ‘the slippery slope’ to make it fun to read. It’s a technique used by a lot of great authors and also copywriters. This technique and a lot more are taught in a course called CopyThat, and it’s the only writing course you’ll ever need. The best part is that it takes only 10 days to complete. 10 days to be a really good writer. Visit CopyThat now to start your journey to becoming a great writer.
