Hindsight Trust Region Policy Optimization

Reinforcement Learning (RL) with sparse rewards is a major challenge. We pro- pose Hindsight Trust Region Policy Optimization (HTRPO), a new RL algorithm that extends the highly successful TRPO algorithm with hindsight to tackle the challenge of sparse rewards. Hindsight refers to the algorithm’s ability to learn from information across goals, including past goals not intended for the current task. We derive the hindsight form of TRPO, together with QKL, a quadratic approximation to the KL divergence constraint on the trust region. QKL reduces variance in KL divergence estimation and improves stability in policy updates. We show that HTRPO has similar convergence property as TRPO. We also present Hindsight Goal Filtering (HGF), which further improves the learning performance for suitable tasks. HTRPO has been evaluated on various sparse-reward tasks, including Atari games and simulated robot control. Experimental results show that HTRPO consistently outperforms TRPO, as well as HPG, a state-of-the-art policy 14 gradient algorithm for RL with sparse rewards.

Free access to antiretroviral treatment and protection against the risk of catastrophic health expenditure in people living with HIV: evidence from Cameroon

Background To foster access to care and reduce the burden of health expenditures on people living with HIV (PLHIV), several sub-Saharan African countries, including Cameroon, have adopted a policy of removing HIV-related fees, especially for antiretroviral treatment (ART). We investigate the impact of Cameroon’s free antiretroviral treatment (ART) policy, enacted in May 2007, on catastrophic health expenditure (CHE) risk according to socioeconomic status, in PLHIV enrolled in the country’s treatment access program. Methods Based on primary data from two cross-sectional surveys of PLHIV outpatients in 2006–2007 and 2014 (i.e., before and after the policy’s implementation, respectively), we used inverse propensity score weighting to reduce covariate imbalances between participants in both surveys, combined with probit regressions of CHE incidence. The analysis included participants treated with ART in one of the 11 HIV services common to both surveys (n = 1275). Results The free ART policy was associated with a significantly lower risk of CHE only in the poorest PLHIV while no significant effect was found in lower-middle or upper socioeconomic status PLHIV. Unexpectedly, the risk of CHE was higher in those with middle socioeconomic status after the policy’s implementation. Conclusions Our findings suggest that Cameroon’s free ART policy is pro-poor. As it only benefitted PLHIV with the lowest socioeconomic status, increased comprehensive HIV care coverage is needed to substantially reduce the risk of CHE and the associated risk of impoverishment for all PLHIV.

Understanding the Reasons for Deferring ART Among Patients Diagnosed Under the Same-Day-ART Policy in Johannesburg, South Africa

We aimed to examine the correlates of antiretroviral therapy (ART) deferral to inform ART demand creation and retention interventions for patients diagnosed with HIV during the Universal Test and Treat (UTT) policy in South Africa. We conducted a cohort study enrolling newly diagnosed HIV-positive adults (≥ 18 years), at four primary healthcare clinics in Johannesburg between October 2017 and August 2018. Patients were interviewed immediately after HIV diagnosis, and ART initiation was determined through medical record review up to six-months post-test. ART deferral was defined as not starting ART six months after HIV diagnosis. Participants who were not on ART six-months post-test were traced and interviewed telephonically to determine reasons for ART deferral. Modified Poisson regression was used to evaluate correlates of six-months ART deferral. We adjusted for baseline demographic and clinical factors. We present crude and adjusted risk ratios (aRR) associated with ART deferral. Overall, 99/652 (15.2%) had deferred ART by six months, 20.5% men and 12.2% women. Baseline predictors of ART deferral were older age at diagnosis (adjusted risk ratio (aRR) 1.5 for 30–39.9 vs 18–29.9 years, 95% confidence intervals (CI): 1.0–2.2), disclosure of intentions to test for HIV (aRR 2.2 non-disclosure vs disclosure to a partner/spouse, 95% CI: 1.4–3.6) and HIV testing history (aRR 1.7 for > 12 months vs < 12 months/no prior test, 95% CI: 1.0–2.8). Additionally, having a primary house in another country (aRR 2.1 vs current house, 95% CI: 1.4–3.1) and testing alone (RR 4.6 vs partner/spouse support, 95% CI: 1.2–18.3) predicted ART deferral among men. Among the 43/99 six-months interviews, women (71.4%) were more likely to self-report ART initiation than men (RR 0.4, 95% CI: 0.2–0.8) and participants who relocated within SA (RR 2.1 vs not relocated, 95% CI: 1.2–3.5) were more likely to still not be on ART. Under the treat-all ART policy, nearly 15.2% of study participants deferred ART initiation up to six months after the HIV diagnosis. Our analysis highlighted the need to pay particular attention to patients who show little social preparation for HIV testing and mobile populations.

Art and Research: A Portrait of a Humanities Faculty as an Inclusive Workspace

At a time when monuments are falling, learning processes and discourses accelerating, it seems apposite to pay attention also to artworks commissioned by established institutions in order to give form to good intentions. This essay focuses on a commissioned portrait of female professors, on art (history) education, Dutch art policy / politics and the former colonial (VOC) site that the University of Amsterdam occupies, in order to aide this institution’s desired process to become more inclusive. It proposes Art(istic) Research as a realm that can contribute a differentiated and thoughtful positioning of research and universities in visual and public domains: a necessary ally. Since the essay was written in Summer 2019 (with later additions), much has happened: the Faculty in question has been found to be the locus of ongoing sexual harassment. The student victims did not feel that the (internal) complaints procedures were safeguarding them sufficiently. They went to the media. The university’s ombudsperson asked to investigate found no systemic deficiencies. Since George Floyd’s death, such a conclusion is no longer possible, and the Board of the university has admitted shameful and systemic failures.[1] We now know better what “systemic” means. This essay’s case study is meant to show that art (history) and philosophy can jointly analyze systems and organizations as a basis for necessary conversations, followed by a broad range of people taking responsibility and acting accordingly. In Covid lock-down times, it became deceptively obvious how unimportant art is. Through an essay such as this (and the Faculty’s “art committee” that I established), it is hopefully also evident how art can be an indicator of institutional culture: e.g. in relation to how embedded the principle to consult specialists is, even if this seems to be unimportant. And that is a matter of life and death. [1] https://www.folia.nl/actueel/138876/fred-weerman-en-geert-ten-dam-dit-is-ongehoord-beschamend

From Barbara to Bia: Art, policy and science dialoguing with rights and health of lesbians in Brazil

Issue Discrimination and homophobia are latent and can affect health. Chico Buarque composed at least three songs about lesbian love: Barbara (with Ruy Guerra - 1972), Sea and Moon (1980) and Blues for Bia (2017). Problem Beginning with the view that art can manifest itself as narrative, this paper analyzes the alignment of the content of the three songs mentioned above with Brazilian policies ensuring LGBT rights and the trend of studies on lesbian health in Brazil from 1970 to 2019. The results can assist the promotion of discussions in other places. We performed 1) a search of governmental policies aimed at protecting the rights of the LGBT population; 2) a systematic search for studies on lesbian health in PUBMED; 3) a musical analysis of the songs. Results The 1970s was marked by a military government when the love addressed in the song 'Barbara' was closeted and sad. The song (composed in a minor tone) uses half-step intervals, generating melancholy feelings. In the 1980s, still under a dictatorial government, several social movements initiated a militant stance for the LGBT movement and the song “Sea and Moon” addressed the suicide of a lesbian couple. We found no government actions in this period or studies conducted in Brazil in the PUBMED database up to 1998. Government policies emerged in the 2000s, especially through the program entitled 'Brazil without Homophobia' in 2004. Other policies followed and intensified in the 2010s. The number of studies increased from seven up to the year 2009 to 105 from 2010 onward. “Blues for Bia” addresses sexual freedom with lightheartedness. Lessons The increase in policies for the rights of the LGBT population was aligned with a trend in studies on lesbian health and with the three songs analyzed. Songs began with sad, negative tones until finally arriving at issues of autonomy and freedom, suggesting the importance of the maintenance of these actions that can alleviate problems related to LGBT rights and health. Key messages Policies for LGBT rights and studies on lesbian health have increased in Brazil since 2000s, especially after 2010. The tone of the songs studied changed from pain to lightness, in line with increase of the research and policies, emphasizing the importance of such actions.

Efficient Deep Reinforcement Learning via Adaptive Policy Transfer

Transfer learning has shown great potential to accelerate Reinforcement Learning (RL) by leveraging prior knowledge from past learned policies of relevant tasks. Existing approaches either transfer previous knowledge by explicitly computing similarities between tasks or select appropriate source policies to provide guided explorations. However, how to directly optimize the target policy by alternatively utilizing knowledge from appropriate source policies without explicitly measuring the similarities is currently missing. In this paper, we propose a novel Policy Transfer Framework (PTF) by taking advantage of this idea. PTF learns when and which source policy is the best to reuse for the target policy and when to terminate it by modeling multi-policy transfer as an option learning problem. PTF can be easily combined with existing DRL methods and experimental results show it significantly accelerates RL and surpasses state-of-the-art policy transfer methods in terms of learning efficiency and final performance in both discrete and continuous action spaces.