News

MAQC Society 1Q2023 Newsletter

MAQC Society annual meeting – Sept 2023

The President of the MAQC Society, Dr. Andreas Scherer, is diligently identifying keynote speakers, organizing sessions and planning for our 6^th annual MAQC Society 2023 meeting with the main theme “AI/ML in Health Science: Benefits and Challenges,” in Helsinki, Finland (Sept 11-12, 2023).  We will have a full two days of keynote and invited speakers as well as sessions involving abstracts contributed by MAQC Society members and related participants.  Registration for the MAQC 2023 meeting is now available.  Feel free to contact the Scandic Park Hotel in Helsinki to make reservations for the conference now.

Abstracts for MAQC 2023

Please submit abstracts for MAQC 2023. If your work involves AI/ML in Health Science, multi-omics, single-cell research, or more traditional examinations of reproducibility in health and -omics disciplines, we would like to review your abstract.  Also, if you would like to organize a session, please contact Dr. Andreas Scherer (andreas.scherer AT helsinki.fi).

Annual Society Awards

Nominations for the annual MAQC society awards including the 2023 Outstanding Reproducibility in Science Award and the 2023 MAQC Society award are now being accepted.  Nominees must have had a key role (first, corresponding, or primary author) of a significant published scientific article promoting, establishing, or contributing to reproducible science, especially of massive datasets in human health sciences in 2022.

Wendell Jones, PhD

Executive Chair, MAQC Society

MAQC Society 3Q2022 Newsletter

We have just completed a successful 5th annual meeting of the MAQC Society, a hybrid meeting where in-person attendees came to the US FDA headquarters in Silver Spring, MD.  We had 140 individuals register with 130+ people attending at least one of the sessions.  Attendance was around 80-85 for a good portion of the first day and over 60 for the second day.  In comparison to last year’s meeting, attendance was noticeably higher, sometimes twice the attendance level in 2022 as 2021.

5th Annual Meeting Highlights

We were honored to have Dr. Namandjé Bumpus, FDA Chief Scientist, to provide opening remarks at the meeting.  Dr. Bumpus highlighted the importance of reproducibility in experiments and discussed some of her own personal experiences in the lab with these challenges.  The first invited talk generated a great deal of discussion at the meeting.  Sayash Kapoor discussed “Leakage and the Reproducibility Crisis in ML-based Science.”   He highlighted data leakage and other issues in Machine Learning (ML) in scientific paper reviews across a broad range of scientific areas but which also included MAQC Society-relevant areas such as bioinformatics, genomics, toxicology, medicine, and clinical epidemiology.  Common issues that affected bioinformatics, genomics, medicine and clinical epidemiology included conducting normalization and/or feature selection using both the training and test sets simultaneously, which is a form a data leakage that will lead to overestimation of the model’s predictive accuracy.  This misguided practice, which was largely avoided in MAQC II due to careful management of the training and test sets, still persists in many scientific studies, leading to erroneous conclusions.

Dr. Marzyeh Ghassemi (MIT) was able to attend the conference in-person and receive the Outstanding Reproducibility in Science Award.  Dr. Ghassemi spoke on a variety of topics including reproducibility challenges in AI/ML.  We were also happy to have Dr. Brian Nosek, Open Science Foundation and UVa faculty member (who was the award runner-up), speak on reproducibility issues encountered with pre-clinical data.  One of the more interesting aspects to his talk for MAQC members was the lack of reproducibility even in the effect sizes of many pre-clinical studies.  While many members have an understanding that in multi-analyte testing, p-values and extreme effect sizes (both very large and very small) may not replicate well, Dr. Nosek demonstrated it can be a challenge to reproduce the effect size estimates observed across a broad range of the values observed in pre-clinical studies.  In fact, 97% of the effect size estimates in this replication study were smaller than in the original.  The median effect size in the replication study was 85% smaller than in the original studies.

Acknowledgements

We also want to thank the FDA/NCTR for being a co-sponsor of the meeting.  We thank the FDA and the facilities personnel at the FDA’s White Oak Conference Center for providing the meeting facility and support for the meeting.  Also, we thank the precisionFDA team for helping to organize the meeting and also for having the top performing teams of the FDA NCTR Indel challenge give their presentations at the meeting.  An interesting aspect of the precisionFDA Indel challenge was the diversity of the performance across the different key performance metrics and the diversity of approaches that each of the top performing teams took.   

Thanks to all the presenters at MAQC 2022 for their presentations and preparations as well as the following members of the program committee:  Dr. Joshua Xu, Dr. Rebecca Kusko, Dr. Samir Lababidi, Dr. Dongying Li, Dr. Joaquin Dopazo, and Dr. Shraddha Thakkar.  I also want to thank Dr. Binsheng Gong for all of his conference support and efforts as well as Dr. Cesare Furlanello and his team for supporting the society award process.

Video Recordings of Presentations

We plan to make most of the MAQC 2022 conference sessions available via video recording in the near future.  To view the recordings, you must either have registered for the 2022 meeting or be a society member.

6th Annual Meeting Plans

For 2023, we currently plan to have the 6th annual meeting of the MAQC Society in late summer or early fall in Europe.  The MAQC Board of Directors will likely decide this by year end 2022, along with the location and venue, but you can use this information to plan for budgeting purposes for 2023 now.

Ideas for New MAQC Working Groups

Finally, we ended the session with a discussion of future projects within the MAQC Society.   Among ideas shared were

• Restarting the single-cell multi-lab multi-platform study involving sc-RNAseq, flow cytometry (high dimensional) and bulk RNAseq.  Challenges include identifying samples (we need at least 2-3 cryo-preserved blood samples per subject) and labs that have all of these capabilities at one site as refreezing the sample for transport is not really an option.
• Reference material/samples for single cell methods
• Having a precisionFDA challenge examining data leakage in radiology or pathology, understanding predictions that may be using background instead of the expected features
• Having a challenge where we combine AI/ML modeling with having participants explain how their models work.  What are reasonable criteria?
• Examining Alzheimer’s disease with multi-omics approaches.  Blood samples only.  Much data is available.  Are platforms consistent?

Other potential topics included massive COVID data, Drug Development Process QC (eg, RNA therapies), handling multi-omics data integration.  The MAQC Society leadership plans to find proper champions for new working groups and present them to the society at-large over the next few months.  Stay tuned for more information.

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society

MAQC Society 1Q22 Newsletter

Greetings from the MAQC Society,

In 2022, the MAQC Society was impacted by the Omicron COVID-19 surge and we have had to postpone our scheduled April annual meeting to later in the year.  We will announce in a special session of the society on May 5 the time and place for the 5^th annual meeting of the MAQC Society.  At that same special session, we will recognize and award lead authors and scientific articles both from within the MAQC Society and outside the society that promote, enhance, or exemplify reproducibility in science, especially with massive data sets.  In the next few weeks, please look for an opportunity to register and (virtually) attend this very important event (a free event).

Despite the pandemic, the work of the MAQC Society continues.  We have scheduled (and there is still time to register) for a free webinar series of additional articles associated with the SEQC2 Consortium effort.  This six-part series for 2022 began on March 22 and continues into May.  Please go to Webinar | MAQC society (themaqc.org)and register for this free webinar series.  Also, if you missed our outstanding 12-part 2021 SEQC2 Webinar Series, you can access most of last year’s presentations directly from same web page.  Original scientific articles continue to be published this year from the SEQC2 Consortium (Papers | MAQC society (themaqc.org) ) and more are on the horizon.  So stay tuned for SCIENCE!

In addition to the webinar series, there is a new project that is currently being initiated that will appeal to many of the variant-calling pipeline builders/owners that are associated with the MAQC Society.   The SEQC2 dataset from the Oncopanel Working group has spawned a precisionFDA challenge called the NCTR Indel Calling from Oncopanel Sequencing Data Challenge – precisionFDA Challenge .  All who diligently work with variant calling in humans understand the specific challenges associated with accurate insertion and deletion calls, especially from tumor samples. In particular, this challenge “asks the participants to develop, validate, and benchmark indel calling pipelines to identify indels in the oncopanel sequencing datasets generated by the SEQC2 Oncopanel Sequencing Working Group.”   If you would like to participate in this challenge, you can pre-register at the web site shown.

Finally, I want to make you aware that the MAQC Society is conducting a formal membership drive.  The cost of membership for professionals is US$100 and for students it is US$50.  For 2022, Society membership benefits include:

• US$100 off (Professional Membership) or US$50 off (Student/Postdoc Membership) of registration at the 2022 late summer/early fall meeting of the MAQC Society
• Free access to videos of the 2022 SEQC2 Webinar series (the 2021 Webinar Series presentations are already freely available at our website)
• Free registration to a planned MAQC Society summer event where we will have up to 3 events with panel members discussing some of the challenges and breakthroughs in Massive Data Analysis and Quality Control.
• Ability to initiate (with approval) MAQC Society projects

The MAQC Society uses membership to help fund our meetings, projects, student and professional awards, and operational costs.  We are a 501c nonprofit, charitable organization who makes things happen via the efforts of volunteers and participants like yourself.  All membership money goes directly to the activities of the MAQC Society.  I hope you will join us in this journey to advance reproducible science by becoming a member and actively participating in society activities.  You can register for membership at MAQC 2022 Society Membership (jotform.com).

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society

Announcing the MAQC Society Spring 2022 Webinar Series and the 2022 MAQC Society Awards

Webinar

The MAQC Society is pleased to announce a new series of SEQC2 (Sequencing and Quality Control Phase 2) Webinars addressing the many challenges of analysis of massive datasets that come from Next-Gen Sequencing (NGS).  In 2021, we provided a series of eleven webinars with a large focus on laboratory and bioinformatic methods that primarily examined challenges in oncology-related sequencing (detecting low-frequency somatic variants, performance of cancer panels, suitable reference material) but also addressing fundamental aspects of molecular biology (epigenomic sequencing QC, metagenomics standards, etc.).  These webinars corresponded to scientific articles that appeared in 2021 in journals such as Nature Biotechnology and Genome Biology.

This year, in 2022, we offer a new webinar series on diverse topics related to NGS:  copy-number pathogenicity, structural variation detection, use of internal standards and synthetic controls for variant detection, formalin-fixation impacts, and general reproducibility challenges. Each webinar will be live and presented at 11 am ET to 12 noon ET on the date indicated, which will be provided on consecutive Tuesdays starting March 22, 2022. For more information about each webinar, go to our Webinar page  Registration is free.

Awards

Nominations are now being accepted for the MAQC Society’s 2022 Outstanding Reproducibility in Science Award

The MAQC Society’s objective is to communicate, promote, and advance reproducible science principles and quality control for analysis of the massive data generated from the existing and emerging technologies in solving biological, health, and medical problems. The MAQC Awards bring attention to the importance of reproducibility in science and celebrate the most impactful scientific contributions that enhance, promote, and exemplify reproducibility in science.  The Outstanding Reproducibility in Science Award was designated to celebrate the scientific contribution of a particular nominee that publishes an outstanding scientific article promoting, establishing, or contributing to reproducible science, especially of massive datasets in human health sciences.  The reproducibility award winners in 2021 included Davide Chicco for his paper “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation” in BMC Genomics which has already generated 1000 citations in less than two years, as well as Charles Wang for his paper “A multicenter study benchmarking single-cell RNA sequencing technologies using reference samples” which appeared in Nature Biotechnology.

Nominees for this award must be primary or corresponding authors of a relevant scientific article relative to the award.  The related scientific article must have been peer-reviewed and published (electronically or otherwise) in the previous calendar year (2021).  The application deadline is March 31, 2022.  More information and access to a form for nominations is provided at https://themaqc.org/awards/.

MAQC Society 4Q2021 Newsletter

Season’s Greetings from the MAQC Society,

On December 1st, we announced we were accepting abstracts for the 5th Annual Meeting of the MAQC Society (April 27-29, 2022, Joint with MCBIOS April 25-27).  Go to https://themaqc.org/conferences/ for more information.  Our theme this year is Explainable and Reproducible AI/ML methods and applications with additional subthemes regarding large COVID-19 datasets and public health.  In addition, we are interested in traditional MAQC Society-related papers in single-cell genomics, image analysis, reference samples and methods, and reproducible and better science in general when working with massive datasets.  MAQC 2022 is scheduled to take place in Cary, NC USA (and via Whova/Zoom) at the SAS Institute Campus if we are successful in reducing COVID-19 infection rates.  I am looking forward to the day when Greek letters go back to math class instead of headlining public health concerns.  We will provide additional information in the early January 2022 time frame as to whether the meeting will be a hybrid virtual/in-person meeting or 100% virtual.   Early abstract submissions can receive early review by mid-January.  We will also begin registration for the meeting in January.

I’m happy to report that the Society members and colleagues continue to publish their results from the SEQC2 working groups.  A listing of MAQC Society-related peer-reviewed papers is available at https://themaqc.org/papers.  For example, Foox et al. recently published their efforts related to epigenomics quality control (EpiQC) in Genome Biology while Willey et al. published their results related to the use of internal standards to assess true actionable mutations in ctDNA in Cell Reports Methods.  Tim Mercer, Joshua Xu, Chris Mason, and Weida Tong have provided a succinct yet comprehensive summary of SEQC2 efforts in an editorial recently published in Genome Biology.

In addition to the stream of papers coming from SEQC2, the MAQC Society is spreading its influence in additional ways.  For example, two papers were recently published where the MAQC Society demonstrated support for general initiatives in promoting standards and reproducibility.  These efforts include the recent Mirzayi et al. paper in Nature Medicine on the STORMS checklist which will improve analysis and reporting of metagenomic and microbiome sequencing as well as the Mammoliti et al. paper in Nature Communications on ORCHESTRA, a cloud-based platform enhancing reproducibility of multimodal biomedical data.  The Mammoliti paper illustrates the use of guiding principles called FAIR (Findable, Accessible, Interoperable, Reusable) for research that all should strongly consider for their own research data.  The FAIR principles were first stated by Wilkinson et al. (https://www.nature.com/articles/sdata201618)  in their 2016 paper in Scientific Data; several Society board and regular members are active in supporting FAIR Principles.

As we are at the end of the year, I ask that you reflect on papers that you have seen published in 2021 that emphasized and advanced reproducibility and transparency in health science as we will seek nominations for the Outstanding Reproducibility in Science Award and the MAQC Society Award soon.  Until we meet together again in-person or virtually, I wish you all good health, to take appropriate precautions during the pandemic relative to your locale, and to enjoy the holidays and the new year.   Society members should look back on 2021 as “well done” for all the work over the past few years that led to tangible results, publications, standards, and recommendations.  Thanks to everyone for each of your efforts.

Feel free to contact me (wendell.jones  AT  q2labsolutions.com ) with any questions.

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society

Press Release Announcing Nature Biotechnology SEQC2 Collection