Social Annotation (SA) tools can be used to facilitate active and collaborative learning when students have to study academic texts. However, making these tools available does not ensure students participate in argumentative discussions. Scaffolding students by means of collaborations scripts geared towards collaboration and discussion encourages students to engage in meaningful, high-quality interactions. We conducted an experiment with students (n=59) in a course running at a Dutch university, using the SA tool Perusall. A control group received normal instructions, while an experimental group received scaffolding through collaboration scripts. The results showed a significant increase in the number of responses to fellow students for the experimental group compared to the control group. The quality of the annotations, measured on levels of Bloom’s taxonomy, increased significantly for the experimental group compared to both its baseline measurement and the control group. However, when scaffolding was faded out over subsequent assignments these differences became non-significant. The experimental groups’ increased quality of annotations did not remain over time, suggesting that internalization of the scripts was not achieved.