Can OpenAI Techniques Help Machines Paint Watercolours? A Look At TRL And OpenEnv

🔍 Read the full analysis: Can OpenAI Techniques Help Machines Paint Watercolours? A Look At TRL And OpenEnv on ThorstenMeyerAI.com

An independent engineer has completed a full open-source reproduction of Surya Narreddi’s viral watercolour-painting language model, utilizing TRL and OpenEn frameworks on Hugging Face infrastructure. The project, which includes datasets, training scripts, and trained models, aims to explore whether reinforcement learning can optimize aesthetic taste rather than verifiable answers. For more details, see the original analysis on training a coding model to paint watercolours. This development makes the original research more accessible and testable for the community, marking a significant step in AI art and reinforcement learning research.

The reproduction closely follows Narreddi’s original approach, which trains a language model to generate JavaScript code that produces watercolour-style paintings through the p5.js library. The process involves reinforcement learning with a reward function combining four terms: code correctness, length, style similarity judged by a vision model, and preferences based on human-labeled data. This approach is discussed in detail in the original analysis. The core innovation is the open release of all datasets, environment code, and trained models, allowing others to replicate and extend the work. You can find a comprehensive overview in the original analysis.

The project leverages the Qwen model, specifically Qwen3-VL-30B-A3B, guided by a reward mix that emphasizes style preferences. The training involves 110 steps, with 240 episodes per step, and eight generations per episode, conducted via Hugging Face’s infrastructure. The reward components include a style judge, a code correctness gate, a length bias, and a human preference model trained on pairs of images. The style judge, for example, compares generated paintings against four references, scoring based on visual attributes like translucency and soft edges.

According to the developer, the open artifacts—datasets, scripts, models—are now hosted in a single Hugging Face collection, making this a comprehensive resource for researchers interested in AI art, reinforcement learning, and interpretability. The project aims to test the hypothesis: can reinforcement learning be guided by aesthetic taste, which is inherently subjective, rather than objective correctness? The results, while promising, are still being analyzed, and no definitive conclusions about the effectiveness of the reward mixes have yet been published.

Implications for AI-Generated Artistic Creativity

This project demonstrates that reinforcement learning can be directed toward subjective aesthetic goals, not just factual correctness or functional performance. If successful, this could expand AI’s role in creative fields, enabling models to produce art that aligns more closely with human taste and style preferences. The open-source nature of the project lowers barriers for researchers to experiment with aesthetic reinforcement learning, potentially accelerating innovations in AI art and design.

Furthermore, because the output is code that produces watercolour paintings, it remains inspectable and editable, unlike pixel-based image generation. This transparency allows artists and developers to understand and modify the decision-making process behind each brushstroke, fostering more collaborative human-AI creative workflows. The work also raises questions about the nature of taste and subjectivity in AI, challenging the assumption that AI can only optimize for measurable, objective rewards.

Historical and Technical Background of AI Art Reproduction

The project is situated within a broader lineage of AI art experiments dating back to DeepDream (2015), GAN-based portraits like Edmond de Belamy (2018), and neural network artworks by Mario Klingemann. Early efforts focused on generating images directly, often resulting in abstract or surreal outputs. Over time, artists and researchers shifted toward more controllable and interpretable methods, such as training models on curated datasets or using code-based approaches.

Surya Narreddi’s original project, which went viral in August 2023, trained a language model to generate JavaScript code that produces watercolour effects through the p5.js library. The model’s output is human-readable code, allowing for inspection and editing, and it was trained with reinforcement learning guided by a reward function emphasizing style and aesthetic qualities. The original work used a curated reference pool and a mixed reward scheme, but artifacts were not publicly released until now.

The open reproduction builds on this foundation, providing transparency and reproducibility, which are often lacking in AI art projects. The use of TRL (Training Reinforcement Learning) and OpenEn frameworks on Hugging Face infrastructure represents a modern, accessible approach to training complex models with reinforcement learning based on subjective preferences, rather than strict correctness.

“Releasing all artifacts openly enables the community to rigorously test whether reinforcement learning can truly optimize for aesthetic taste, not just correctness.”

— Thorsten Meyer, AI researcher

Unconfirmed Effectiveness of Reward Schemes

While the reproduction provides all artifacts and details, it remains unclear how well the different reward mixes perform in producing aesthetically pleasing watercolours. The author notes that the results are preliminary, and no definitive quantitative comparison or final verdict on the best reward configuration has been published. The ultimate question—whether reinforcement learning can reliably optimize for aesthetic taste—remains open and subject to further testing and validation.

Next Steps for Validation and Community Testing

The upcoming technical report from Narreddi will likely provide more detailed analysis and quantitative results. Meanwhile, the open artifacts invite independent researchers and artists to test, modify, and extend the models. Future work may include refining reward functions, exploring different artistic styles, and benchmarking against pixel-based image generation methods. The community’s engagement will be crucial in assessing whether reinforcement learning can genuinely capture subjective aesthetic preferences in AI-generated art.

Key Questions

What is the main goal of this open reproduction?

The primary aim is to test whether reinforcement learning can optimize AI models for aesthetic taste, specifically in generating watercolour-style paintings through code, with all artifacts openly shared for community validation.

How does this reproduction differ from the original project?

It provides a complete, open-source implementation with datasets, training scripts, and trained models, whereas the original project by Narreddi did not release artifacts and was limited to a blog post and a viral video.

Can reinforcement learning really teach AI to prefer beauty?

This is an open research question. The project aims to explore this possibility by using human-style preference models as rewards, but conclusive evidence is still pending.

Why is making the code and datasets open important?

Open access allows independent validation, fosters collaboration, and accelerates research into AI creativity and aesthetic optimization, making the field more transparent and reproducible.

What are the potential implications for AI art?

If successful, this approach could enable AI to produce more human-like, style-aware artwork, expanding creative possibilities and integrating AI more deeply into artistic workflows.

Primary source: Hugging Face · via ThorstenMeyerAI.com