An Experimental Study on State Representation Extraction for Vision-Based Deep Reinforcement Learning

Scaling end-to-end learning to control robots with vision inputs is a challenging problem in the field of deep reinforcement learning (DRL). While achieving remarkable success in complex sequential tasks, vision-based DRL remains extremely data-inefficient, especially when dealing with high-dimensional pixels inputs. Many recent studies have tried to leverage state representation learning (SRL) to break through such a barrier. Some of them could even help the agent learn from pixels as efficiently as from states. Reproducing existing work, accurately judging the improvements offered by novel methods, and applying these approaches to new tasks are vital for sustaining this progress. However, the demands of these three aspects are seldom straightforward. Without significant criteria and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the previous methods are meaningful. For this reason, we conducted ablation studies on hyperparameters, embedding network architecture, embedded dimension, regularization methods, sample quality and SRL methods to compare and analyze their effects on representation learning and reinforcement learning systematically. Three evaluation metrics are summarized, including five baseline algorithms (including both value-based and policy-based methods) and eight tasks are adopted to avoid the particularity of each experiment setting. We highlight the variability in reported methods and suggest guidelines to make future results in SRL more reproducible and stable based on a wide number of experimental analyses. We aim to spur discussion about how to assure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

SNP Genotyping and ApoE Genotyping v1

This protocol details the steps for DNA extraction from a human blood sample, quality control, and SNP and APOE genotyping. The protocol has been adapted from the PRoBaND SNP Genotyping and ApoE Genotyping Protocol. The overall protocol for PRoBaND /Tracking Parkinson’s is published: Malek, N., Swallow, D. M. A., Grosset, K. A., Lawton, M. A., Marrinan, S. L., Lehn, A. C., Bresner, C., Bajaj, N., Barker, R. A., Ben-Shlomo, Y., Burn, D. J., Foltynie, T., Hardy, J., Morris, H. R., Williams, N. M., Wood, N., & Grosset, D. G. (2015). Tracking Parkinson’s: Study Design and Baseline Patient Data. Journal of Parkinson’s Disease, 5(4), 947–959. https://doi.org/10.3233/JPD-150662

Procedural Level Generation for Sokoban via Deep Learning: An Experimental Study

Deep learning for procedural level generation has been explored in many recent works, however, experimental comparisons with previous works are rare and usually limited to the work they extend upon. This paper's goal is to conduct an experimental study on four recent deep learning procedural level generators for Sokoban to explore their strengths and weaknesses. The methods will be bootstrapping conditional generative models, controllable & uncontrollable procedural content generation via reinforcement learning (PCGRL) and generative playing networks. We will propose some modifications to either adapt the methods to the task or improve their efficiency and performance. For the bootstrapping method, we propose using diversity sampling to improve the solution diversity, auxiliary targets to enhance the models' quality and Gaussian mixture models to improve the sample quality. The results show that diversity sampling at least doubles the unique plan count in the generated levels. On average, auxiliary targets increases the quality by 24% and sampling conditions from Gaussian mixture models increases the sample quality by 13%. Overall, PCGRL shows superior quality and diversity while generative adversarial networks exhibit the least control confusion when trained with diversity sampling and auxiliary targets.

Procedural Level Generation for Sokoban via Deep Learning: An Experimental Study

Deep learning for procedural level generation has been explored in many recent works, however, experimental comparisons with previous works are rare and usually limited to the work they extend upon. This paper's goal is to conduct an experimental study on four recent deep learning procedural level generators for Sokoban to explore their strengths and weaknesses. The methods will be bootstrapping conditional generative models, controllable & uncontrollable procedural content generation via reinforcement learning (PCGRL) and generative playing networks. We will propose some modifications to either adapt the methods to the task or improve their efficiency and performance. For the bootstrapping method, we propose using diversity sampling to improve the solution diversity, auxiliary targets to enhance the models' quality and Gaussian mixture models to improve the sample quality. The results show that diversity sampling at least doubles the unique plan count in the generated levels. On average, auxiliary targets increases the quality by 24% and sampling conditions from Gaussian mixture models increases the sample quality by 13%. Overall, PCGRL shows superior quality and diversity while generative adversarial networks exhibit the least control confusion when trained with diversity sampling and auxiliary targets.

Procedural Level Generation for Sokoban via Deep Learning: An Experimental Study

Deep learning for procedural level generation has been explored in many recent works, however, experimental comparisons with previous works are either nonexistent or limited to the works they extend upon. This paper’s goal is to conduct an experimental study on four recent deep learning procedural level generation methods for Sokoban (size = 7 × 7) to explore their strengths and weaknesses and provide insights for possible research directions. The methods will be bootstrapping conditional generative models, controllable & uncontrollable procedural content generation via reinforcement learning (PCGRL) and generative playing networks. We will propose some modifications to either adapt the methods to the task or improve their efficiency and performance. For the bootstrapping method, we propose using diversity sampling to improve the solution diversity, auxiliary targets to enhance the models’ quality and Gaussian mixture models to improve the sample quality. The results show that diversity sampling at least doubles the unique plan count in the generated levels. On average, auxiliary targets increases the quality by 24% and sampling conditions from Gaussian mixture models increases the sample quality by 13%. Overall, PCGRL shows superior quality and diversity while generative adversarial networks exhibit the least control confusion when trained with diversity sampling and auxiliary targets.

Procedural Level Generation for Sokoban via Deep Learning: An Experimental Study

Deep learning for procedural level generation has been explored in many recent works, however, experimental comparisons with previous works are either nonexistent or limited to the works they extend upon. This paper’s goal is to conduct an experimental study on four recent deep learning procedural level generation methods for Sokoban (size = 7 × 7) to explore their strengths and weaknesses and provide insights for possible research directions. The methods will be bootstrapping conditional generative models, controllable & uncontrollable procedural content generation via reinforcement learning (PCGRL) and generative playing networks. We will propose some modifications to either adapt the methods to the task or improve their efficiency and performance. For the bootstrapping method, we propose using diversity sampling to improve the solution diversity, auxiliary targets to enhance the models’ quality and Gaussian mixture models to improve the sample quality. The results show that diversity sampling at least doubles the unique plan count in the generated levels. On average, auxiliary targets increases the quality by 24% and sampling conditions from Gaussian mixture models increases the sample quality by 13%. Overall, PCGRL shows superior quality and diversity while generative adversarial networks exhibit the least control confusion when trained with diversity sampling and auxiliary targets.

A real-world study evaluating ultrasound guided percutaneous non-targeted liver biopsy needle failures and pathology sample quality assessment in both end-cut and side-notch needles

Objectives: To determine biopsy device failures, causative factors, complications and sample quality of the 16G end-cut Biopince™ and side-notch Bard™ needles. Methods: All ultrasound-guided non-targeted liver biopsies between 01/01/2016 to 31/12/2018 were included. Operator, device, number of failures, complications and repeat biopsies were recorded. Histopathology samples were reviewed for all cases of needle failure and a group with no failures, and graded “yes/no” for presence of steatosis, inflammation and fibrosis. The pathology slides from these cases were reviewed to assess biopsy sample quality (length and portal tract number). The failure and no-failure groups were compared in terms of device type/ histology and sample quality was compared between the needle types. Results: 1004 patients were included. 93.8% (n = 942) required one needle pass to obtain a sample and 6.2% (n = 62) required >1 pass due to needle failure. Total of 76 needle failures, more with end-cut than side-notch needles (8.7% vs 2.9%) (p < 0.001). No needle failures resulted in complication. The presence of liver fibrosis was associated with fewer needle failures (p = 0.036). The major complication rate was 0.4% (4/1044). A biopsy with >10 portal tracts was obtained in 90.2% of specimens > 20 mm long, compared with 66% of 16–20 mm biopsies and 21% of <16 mm biopsies. The target of >10 portal tracts was achieved in 10/26 (38.5%) of side-notch biopsies and 64/90 (71.1%) of end-cut biopsies (p = 0.004). Conclusion: Ultrasound-guided liver biopsy is safe and sample quality is consistently good when a core >20 mm long is obtained. The end-cut biopsy device generated reliably good quality biopsy samples, however the needle failure rate was significantly higher than the side-cut needle. Advances in knowledge: Ultrasound guided liver biopsy specimen quality is consistently good when a core >20 mm long is obtained which can be achieved with a single pass using the 16G BiopinceTM end-cut needle, although the needle failure rate is significantly higher than the 16G Max-Core™ Bard™ side-notch needle.

HOME vs. LAB hair samples for the determination of long-term steroid concentrations: a comparison between hair samples collected by laypersons and trained research staff

Hair segment analysis is a valuable tool for the assessment of cumulative long-term steroid secretion. Preliminary findings suggest comparable cortisol concentrations in hair collected by instructed laypersons and research staff. However, it remains unclear whether hair sample quality and hair steroids other than cortisol are affected by level of experience (laypersons vs. research staff), home collection circumstances (instructions, familiarity to participant, performance confidence), and characteristics of the layperson (conscientiousness). Sixty participants (23.6 ± 3.9 years; 43 females) provided hair samples twice: first collected by laypersons (HOME) according to provided instructions (written vs. written/video-based instructions) and second by trained research staff (LAB) on the same day or the day after the HOME collection. Hair steroid concentrations (cortisol, cortisone, DHEA, progesterone) were determined using LC–MS/MS. Hair sample quality was evaluated using nine predefined criteria. Laypersons completed questionnaires for the assessment of potential factors of hair outcome measures (hair steroid concentrations, hair sample quality). Hair steroids from HOME and LAB samples were positively correlated (rs between 0.76 and 0.89) and did not significantly differ, with the exception of cortisone. The quality of hair samples was significantly higher for LAB than for HOME samples. Neither HOME collection circumstances nor layperson-related characteristics had an impact on hair outcome measures. However, a low self-reported performance confidence predicted a high absolute difference between HOME and LAB DHEA. In summary, our findings suggest higher quality of hair samples collected by trained research staff compared to instructed laypersons. However, these differences might be negligible, considering the high correlation between HOME and LAB hair steroid concentrations, with the characteristics of the layperson or collection circumstances having a minor impact on hair steroids and hair sample quality. These findings provide further support for the notion that well-instructed laypersons can be enabled to collect hair samples.