Spark: Synergistic Policy And Reward Co-Evolving Framework

📖Paper | 🤗Models | 🤗Datasets | 🤗Daily Paper

🌈 Introduction: We propose SPARK, a unified framework that integrates policy and reward into a single model for joint and synchronous training. SPARK can automatically derive reward and reflection data from verifiable reward, enabling self-learning and self-evolution.

🤗 Models: We release the checkpoints at internlm/Spark-VL-7B.

🤗 Datasets: Training data is available at internlm/Spark-Data.

💻 Training Code: The training code and implementation details can be found at InternLM/Spark.

📸 Upload an image and enter a prompt or 🖼️ choose the input from the example gallery (image + prompt).

Image

Prompt

max_new_tokens

512 4096

temperature

0 1.5

top_p

0 1

top_k

1 200

Examples

📐 Math Reasoning Examples

Math Example 0

Prompt

🎯 Reward Model Examples

Reward Example 0

Prompt

You are a highly capable multimodal AI assistant tasked with evaluating answers to visual questions. Please analyze the following image and question, then determine which of the two provided answers is better.

Question:
Create a detailed description describing the elements of the given worldwide landmark image. Identify the name of the landmark and describe the objects, colors, perspectives, etc.
Answer 1: 
"The image showcases a breathtaking view of a mountainous region with a deep, green lake nestled amidst the rocky terrain. The perspective is from a high vantage point, providing a panoramic view of the landscape. The landmark, identified as \"sa_1563\", is a testament to the natural beauty and diversity of our world. The combination of the rugged mountains, the clear blue sky, and the vibrant green lake creates a serene and awe-inspiring scene."

Answer 2:
"The image showcases a serene mountain lake, nestled amidst rocky terrains. The deep green color of the water contrasts with the surrounding landscape, creating a tranquil scene. This landmark is a natural beauty, located in a hilly or mountainous region. The perspective of the image gives a sense of being high above the lake, looking down upon it. However, without specific identifiable features, it's challenging to determine the exact landmark from the description."

Please evaluate both answers based on the following criteria:
1. Accuracy: How well does the answer align with the visual information in the image?
2. Completeness: Does the answer fully address all aspects of the question?
3. Clarity: Is the answer easy to understand and well-articulated?
4. Relevance: Does the answer directly relate to the question and the image?
  
After your evaluation, please:
1. Explain your reasoning for each criterion.
2. Provide an overall judgment on which answer is better (Answer 1 or Answer 2). For example: Overall Judgment: Answer X is better.
  
Your response should be structured and detailed, demonstrating your understanding of both the visual and textual elements of the task.

Model Output

If you find this project useful, please kindly cite:

@article{liu2025spark,
  title={SPARK: Synergistic Policy And Reward Co-Evolving Framework},
  author={Liu, Ziyu and Zang, Yuhang and Ding, Shengyuan and Cao, Yuhang and Dong, Xiaoyi and Duan, Haodong and Lin, Dahua and Wang, Jiaqi},
  journal={arXiv preprint arXiv:2509.22624},
  year={2025}
}