News

 

MAQC Society 3Q2022 Newsletter

We have just completed a successful 5th annual meeting of the MAQC Society, a hybrid meeting where in-person attendees came to the US FDA headquarters in Silver Spring, MD.  We had 140 individuals register with 130+ people attending at least one of the sessions.  Attendance was around 80-85 for a good portion of the first day and over 60 for the second day.  In comparison to last year’s meeting, attendance was noticeably higher, sometimes twice the attendance level in 2022 as 2021. 

5th Annual Meeting Highlights

We were honored to have Dr. Namandjé Bumpus, FDA Chief Scientist, to provide opening remarks at the meeting.  Dr. Bumpus highlighted the importance of reproducibility in experiments and discussed some of her own personal experiences in the lab with these challenges.  The first invited talk generated a great deal of discussion at the meeting.  Sayash Kapoor discussed “Leakage and the Reproducibility Crisis in ML-based Science.”   He highlighted data leakage and other issues in Machine Learning (ML) in scientific paper reviews across a broad range of scientific areas but which also included MAQC Society-relevant areas such as bioinformatics, genomics, toxicology, medicine, and clinical epidemiology.  Common issues that affected bioinformatics, genomics, medicine and clinical epidemiology included conducting normalization and/or feature selection using both the training and test sets simultaneously, which is a form a data leakage that will lead to overestimation of the model’s predictive accuracy.  This misguided practice, which was largely avoided in MAQC II due to careful management of the training and test sets, still persists in many scientific studies, leading to erroneous conclusions.

Dr. Marzyeh Ghassemi (MIT) was able to attend the conference in-person and receive the Outstanding Reproducibility in Science Award.  Dr. Ghassemi spoke on a variety of topics including reproducibility challenges in AI/ML.  We were also happy to have Dr. Brian Nosek, Open Science Foundation and UVa faculty member (who was the award runner-up), speak on reproducibility issues encountered with pre-clinical data.  One of the more interesting aspects to his talk for MAQC members was the lack of reproducibility even in the effect sizes of many pre-clinical studies.  While many members have an understanding that in multi-analyte testing, p-values and extreme effect sizes (both very large and very small) may not replicate well, Dr. Nosek demonstrated it can be a challenge to reproduce the effect size estimates observed across a broad range of the values observed in pre-clinical studies.  In fact, 97% of the effect size estimates in this replication study were smaller than in the original.  The median effect size in the replication study was 85% smaller than in the original studies.

Acknowledgements

We also want to thank the FDA/NCTR for being a co-sponsor of the meeting.  We thank the FDA and the facilities personnel at the FDA’s White Oak Conference Center for providing the meeting facility and support for the meeting.  Also, we thank the precisionFDA team for helping to organize the meeting and also for having the top performing teams of the FDA NCTR Indel challenge give their presentations at the meeting.  An interesting aspect of the precisionFDA Indel challenge was the diversity of the performance across the different key performance metrics and the diversity of approaches that each of the top performing teams took.   

Thanks to all the presenters at MAQC 2022 for their presentations and preparations as well as the following members of the program committee:  Dr. Joshua Xu, Dr. Rebecca Kusko, Dr. Samir Lababidi, Dr. Dongying Li, Dr. Joaquin Dopazo, and Dr. Shraddha Thakkar.  I also want to thank Dr. Binsheng Gong for all of his conference support and efforts as well as Dr. Cesare Furlanello and his team for supporting the society award process.

Video Recordings of Presentations

We plan to make most of the MAQC 2022 conference sessions available via video recording in the near future.  To view the recordings, you must either have registered for the 2022 meeting or be a society member.

6th Annual Meeting Plans

For 2023, we currently plan to have the 6th annual meeting of the MAQC Society in late summer or early fall in Europe.  The MAQC Board of Directors will likely decide this by year end 2022, along with the location and venue, but you can use this information to plan for budgeting purposes for 2023 now. 

Ideas for New MAQC Working Groups

Finally, we ended the session with a discussion of future projects within the MAQC Society.   Among ideas shared were

  • Restarting the single-cell multi-lab multi-platform study involving sc-RNAseq, flow cytometry (high dimensional) and bulk RNAseq.  Challenges include identifying samples (we need at least 2-3 cryo-preserved blood samples per subject) and labs that have all of these capabilities at one site as refreezing the sample for transport is not really an option.
  • Reference material/samples for single cell methods
  • Having a precisionFDA challenge examining data leakage in radiology or pathology, understanding predictions that may be using background instead of the expected features
  • Having a challenge where we combine AI/ML modeling with having participants explain how their models work.  What are reasonable criteria?
  • Examining Alzheimer’s disease with multi-omics approaches.  Blood samples only.  Much data is available.  Are platforms consistent?

Other potential topics included massive COVID data, Drug Development Process QC (eg, RNA therapies), handling multi-omics data integration.  The MAQC Society leadership plans to find proper champions for new working groups and present them to the society at-large over the next few months.  Stay tuned for more information.

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society

MAQC Society 2Q2022 Newsletter

As a follow-up to the 1Q22 Newsletter, we have identified the date and place for the 2022 MAQC Society annual meeting.  We will conduct our 2022 annual meeting Sept. 26 & 27 in Silver Spring, MD USA at the White Oak Conference center at the US FDA HQ. We are excited to announce that the FDA-NCTR will be a co-sponsor of the MAQC 2022 meeting.   Of course, this is also subject local protocols if COVID-19 rates were to suddenly surge later this year.  We will plan to provide a virtual offering for the meeting in any event, but I hope you will plan to attend in-person.   Registration will only be $100 for professionals and $50 for students/postdocs.  Individuals who are already formally members of the MAQC Society (as well as US FDA employees) can register for free.  We plan to begin registration this week.  Please visit our web site MAQC society | (themaqc.org) later this week for details.  Due to the significance of having the US FDA co-sponsor the meeting, we have decided to announce our 2022 Outstanding Reproducibility in Science Award and the MAQC Society Award for individuals publishing outstanding scientific papers in 2021 at that September meeting. 

The primary themes for the 2022 MAQC Society meeting include a) Explainable AI/ML, b) Reproducible AI/ML methods and applications, c) AI/ML in massive health–related datasets, and d) Large datasets related to SARS-CoV-2 and public health.  Also, we are wanting other abstracts to be submitted in traditional areas such as single-cell assays, image analysis, reference material, and other methods promoting reproducibility in -omics.  Please go to Conferences | MAQC society (themaqc.org) to submit abstracts for the meeting. 

We are very excited the US FDA be a co-sponsor of our September MAQC Society meeting.  Among the events planned is a session featuring the precisionFDA’s NCTR Indel Calling from Oncopanel Sequencing Data Challenge Phase 1, a challenge where submissions just concluded last week.  The Office of Health Informatics at the US FDA will have a special session and presentation describing this challenge and other challenges in their precisionFDA program as well as recognition of outstanding performers in the Indel Calling challenge.  This challenge will use familiar data to the many involved in SEQC2 as it springs from the oncopanel sequencing data and results published this past year in Genome Biology.

I hope that many of you were able to attend the 2022 weekly MAQC Society Webinar series that was conducted over March 22 – May 3.  In this series, which was an extension of the webinars provided in 2021, we highlighted additional SEQC2 efforts that have been recently published in peer-reviewed journals.  If you were not able to attend, you can still view the 2022 webinar series by becoming a member of the MAQC Society (see below for more info).  Webinars from 2021 are available at our website (Webinar | MAQC society (themaqc.org).  Presentations included Deep oncopanel sequencing reveals within block position-dependent quality degradation in FFPE processed samples by Dr. Joshua Xu corresponding to an article was just published in June by Genome Biology.

Finally, I want to remind you that the MAQC Society is conducting a formal membership drive.  The cost of membership for professionals is US$100 and for students it is US$50.  For 2022, Society membership benefits include:

  • US$100 off (Professional Membership) or US$50 off (Student/Postdoc Membership) of registration at the 2022 September meeting of the MAQC Society
  • Free access to videos of the 2022 SEQC2 Webinar series (the 2021 Webinar Series presentations are already freely available at our website)
  • Free registration to a planned MAQC Society summer/fall events where we will have up to 3 events with panel members discussing some of the challenges and breakthroughs in Massive Data Analysis and Quality Control.
  • Ability to initiate (with approval) MAQC Society projects

The MAQC Society uses membership to help fund our meetings, projects, student and professional awards, and operational costs.  We are a 501c nonprofit, charitable organization who makes things happen via the efforts of volunteers and participants like yourself.  All membership money goes directly to the activities of the MAQC Society.  I hope you will join us in this journey to advance reproducible science by becoming a member and actively participating in society activities.  You can register for membership at Membership | MAQC society (themaqc.org).

Regards

Wendell Jones, PhD

Executive Chair of the MAQC Society

MAQC Society 1Q22 Newsletter

Greetings from the MAQC Society,

In 2022, the MAQC Society was impacted by the Omicron COVID-19 surge and we have had to postpone our scheduled April annual meeting to later in the year.  We will announce in a special session of the society on May 5 the time and place for the 5th annual meeting of the MAQC Society.  At that same special session, we will recognize and award lead authors and scientific articles both from within the MAQC Society and outside the society that promote, enhance, or exemplify reproducibility in science, especially with massive data sets.  In the next few weeks, please look for an opportunity to register and (virtually) attend this very important event (a free event).

Despite the pandemic, the work of the MAQC Society continues.  We have scheduled (and there is still time to register) for a free webinar series of additional articles associated with the SEQC2 Consortium effort.  This six-part series for 2022 began on March 22 and continues into May.  Please go to Webinar | MAQC society (themaqc.org)and register for this free webinar series.  Also, if you missed our outstanding 12-part 2021 SEQC2 Webinar Series, you can access most of last year’s presentations directly from same web page.  Original scientific articles continue to be published this year from the SEQC2 Consortium (Papers | MAQC society (themaqc.org) ) and more are on the horizon.  So stay tuned for SCIENCE!

In addition to the webinar series, there is a new project that is currently being initiated that will appeal to many of the variant-calling pipeline builders/owners that are associated with the MAQC Society.   The SEQC2 dataset from the Oncopanel Working group has spawned a precisionFDA challenge called the NCTR Indel Calling from Oncopanel Sequencing Data Challenge – precisionFDA Challenge .  All who diligently work with variant calling in humans understand the specific challenges associated with accurate insertion and deletion calls, especially from tumor samples. In particular, this challenge “asks the participants to develop, validate, and benchmark indel calling pipelines to identify indels in the oncopanel sequencing datasets generated by the SEQC2 Oncopanel Sequencing Working Group.”   If you would like to participate in this challenge, you can pre-register at the web site shown.

Finally, I want to make you aware that the MAQC Society is conducting a formal membership drive.  The cost of membership for professionals is US$100 and for students it is US$50.  For 2022, Society membership benefits include:

  • US$100 off (Professional Membership) or US$50 off (Student/Postdoc Membership) of registration at the 2022 late summer/early fall meeting of the MAQC Society
  • Free access to videos of the 2022 SEQC2 Webinar series (the 2021 Webinar Series presentations are already freely available at our website)
  • Free registration to a planned MAQC Society summer event where we will have up to 3 events with panel members discussing some of the challenges and breakthroughs in Massive Data Analysis and Quality Control.
  • Ability to initiate (with approval) MAQC Society projects

The MAQC Society uses membership to help fund our meetings, projects, student and professional awards, and operational costs.  We are a 501c nonprofit, charitable organization who makes things happen via the efforts of volunteers and participants like yourself.  All membership money goes directly to the activities of the MAQC Society.  I hope you will join us in this journey to advance reproducible science by becoming a member and actively participating in society activities.  You can register for membership at MAQC 2022 Society Membership (jotform.com).

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society

Announcing the MAQC Society Spring 2022 Webinar Series and the 2022 MAQC Society Awards

 

Webinar

The MAQC Society is pleased to announce a new series of SEQC2 (Sequencing and Quality Control Phase 2) Webinars addressing the many challenges of analysis of massive datasets that come from Next-Gen Sequencing (NGS).  In 2021, we provided a series of eleven webinars with a large focus on laboratory and bioinformatic methods that primarily examined challenges in oncology-related sequencing (detecting low-frequency somatic variants, performance of cancer panels, suitable reference material) but also addressing fundamental aspects of molecular biology (epigenomic sequencing QC, metagenomics standards, etc.).  These webinars corresponded to scientific articles that appeared in 2021 in journals such as Nature Biotechnology and Genome Biology.

This year, in 2022, we offer a new webinar series on diverse topics related to NGS:  copy-number pathogenicity, structural variation detection, use of internal standards and synthetic controls for variant detection, formalin-fixation impacts, and general reproducibility challenges. Each webinar will be live and presented at 11 am ET to 12 noon ET on the date indicated, which will be provided on consecutive Tuesdays starting March 22, 2022. For more information about each webinar, go to our Webinar page  Registration is free.

 
Awards

Nominations are now being accepted for the MAQC Society’s 2022 Outstanding Reproducibility in Science Award

The MAQC Society’s objective is to communicate, promote, and advance reproducible science principles and quality control for analysis of the massive data generated from the existing and emerging technologies in solving biological, health, and medical problems. The MAQC Awards bring attention to the importance of reproducibility in science and celebrate the most impactful scientific contributions that enhance, promote, and exemplify reproducibility in science.  The Outstanding Reproducibility in Science Award was designated to celebrate the scientific contribution of a particular nominee that publishes an outstanding scientific article promoting, establishing, or contributing to reproducible science, especially of massive datasets in human health sciences.  The reproducibility award winners in 2021 included Davide Chicco for his paper “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation” in BMC Genomics which has already generated 1000 citations in less than two years, as well as Charles Wang for his paper “A multicenter study benchmarking single-cell RNA sequencing technologies using reference samples” which appeared in Nature Biotechnology.

Nominees for this award must be primary or corresponding authors of a relevant scientific article relative to the award.  The related scientific article must have been peer-reviewed and published (electronically or otherwise) in the previous calendar year (2021).  The application deadline is March 31, 2022.  More information and access to a form for nominations is provided at https://themaqc.org/awards/.

 

2022 MAQC Society 5th Annual Meeting is postponed until 2H2022

Due to the ongoing COVID-19 pandemic, we will be postponing our annual meeting until the 2H2022, most likely around September.  We expect to announce the exact dates and venue/location for the meeting on May 5 2022 at a special meeting/webinar of the MAQC Society where we will recognize the winners of the MAQC Society Award and the Outstanding Reproducibility in Science Award for outstanding articles published in 2021.  Please continue to submit abstracts for the meeting and we will look forward to getting together in-person and virtually in the 2nd half of this year.  Also, please see the Award section of the society’s web pages for nominations for the MAQC Society Awards.

MAQC Society 4Q2021 Newsletter

 

Season’s Greetings from the MAQC Society,

On December 1st, we announced we were accepting abstracts for the 5th Annual Meeting of the MAQC Society (April 27-29, 2022, Joint with MCBIOS April 25-27).  Go to https://themaqc.org/conferences/ for more information.  Our theme this year is Explainable and Reproducible AI/ML methods and applications with additional subthemes regarding large COVID-19 datasets and public health.  In addition, we are interested in traditional MAQC Society-related papers in single-cell genomics, image analysis, reference samples and methods, and reproducible and better science in general when working with massive datasets.  MAQC 2022 is scheduled to take place in Cary, NC USA (and via Whova/Zoom) at the SAS Institute Campus if we are successful in reducing COVID-19 infection rates.  I am looking forward to the day when Greek letters go back to math class instead of headlining public health concerns.  We will provide additional information in the early January 2022 time frame as to whether the meeting will be a hybrid virtual/in-person meeting or 100% virtual.   Early abstract submissions can receive early review by mid-January.  We will also begin registration for the meeting in January.

I’m happy to report that the Society members and colleagues continue to publish their results from the SEQC2 working groups.  A listing of MAQC Society-related peer-reviewed papers is available at https://themaqc.org/papers.  For example, Foox et al. recently published their efforts related to epigenomics quality control (EpiQC) in Genome Biology while Willey et al. published their results related to the use of internal standards to assess true actionable mutations in ctDNA in Cell Reports Methods.  Tim Mercer, Joshua Xu, Chris Mason, and Weida Tong have provided a succinct yet comprehensive summary of SEQC2 efforts in an editorial recently published in Genome Biology.

In addition to the stream of papers coming from SEQC2, the MAQC Society is spreading its influence in additional ways.  For example, two papers were recently published where the MAQC Society demonstrated support for general initiatives in promoting standards and reproducibility.  These efforts include the recent Mirzayi et al. paper in Nature Medicine on the STORMS checklist which will improve analysis and reporting of metagenomic and microbiome sequencing as well as the Mammoliti et al. paper in Nature Communications on ORCHESTRA, a cloud-based platform enhancing reproducibility of multimodal biomedical data.  The Mammoliti paper illustrates the use of guiding principles called FAIR (Findable, Accessible, Interoperable, Reusable) for research that all should strongly consider for their own research data.  The FAIR principles were first stated by Wilkinson et al. (https://www.nature.com/articles/sdata201618)  in their 2016 paper in Scientific Data; several Society board and regular members are active in supporting FAIR Principles.

As we are at the end of the year, I ask that you reflect on papers that you have seen published in 2021 that emphasized and advanced reproducibility and transparency in health science as we will seek nominations for the Outstanding Reproducibility in Science Award and the MAQC Society Award soon.  Until we meet together again in-person or virtually, I wish you all good health, to take appropriate precautions during the pandemic relative to your locale, and to enjoy the holidays and the new year.   Society members should look back on 2021 as “well done” for all the work over the past few years that led to tangible results, publications, standards, and recommendations.  Thanks to everyone for each of your efforts.

Feel free to contact me (wendell.jones  AT  q2labsolutions.com ) with any questions.

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society

MAQC Society 3Q2021 Newsletter

Greetings from the MAQC Society,

This past quarter has been a prominent time for the MAQC Society.  Nature Biotechnology released their September 2021 issue highlighting and acknowledging the research of the Sequencing and Quality Control Phase 2 project by publishing four SEQC2-related papers as well as providing editorials about the history of the MAQC-related consortiums and the importance and impact of the MAQC Society’s work.  One of the editorials was provided by Marc Salit (Director of Joint Initiative for Metrology in Biology at Stanford Univ.) and Janet Woodcock (Acting Commissioner of the US Food and Drug Administration).  The MAQC Society is honored to have such prominent persons in science and health provide a perspective on the public-private partnership that the members of the MAQC Society have fostered.  To quote the editorial “…genomic data, with the confidence lent by the MAQC efforts, underpins many drug and device applications in the precision medicine arena.”  This event follows the successful publication of four other SEQC2 papers in NBT, Genome Biology and Genome Medicine that have appeared over the last five months.   Check out the complete collection athttps://themaqcorg/papers.  Also, many of these papers have video presentations available by the primary investigators available at https://themaqcorg/webinar, which were part of our earlier SEQC2 Webinar series.

 

Wendell Jones, PhD

Wendell Jones, PhD

Chairperson of MAQC Society

However, this is not the totality of the SEQC2-related publications.  More SEQC2 papers will appear in a variety of journals in the coming months, highlighting efforts in epigenetics, structural variant detection, applications of AI/ML in detecting somatic SNVs, quality control materials, and other areas.  Look for a new series of Webinars related to these SEQC2 research efforts this coming November (2021). 

 

Finally, I want to pre-announce that we will be inviting you to submit abstracts for MAQC 2022 very soon, most likely in the middle of October.  MAQC 2022 is scheduled to take place April 27-29, 2022 in Cary, NC USA (and via Whova/Zoom) at the SAS Institute Campus if we are successful in reducing COVID-19 infection rates.  We will provide additional information in the early January 2022 time frame as to whether the meeting will be a hybrid virtual/in-person meeting or 100% virtual.  The tentative theme for MAQC 2022 is Reproducibility and Explainable AI in Health.  In addition to papers related to reproducible and explainable AI, we will solicit papers examining massive data efforts related to COVID-19 evolution, tracking, and sequencing as well as traditional MAQC reproducibility-related topics in NGS, medical imaging, single-cell methods, and emerging molecular methods.   Feel free to contact me (wendell.jones  AT  q2labsolutions.com ) with questions. 

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society

Press Release Announcing Nature Biotechnology SEQC2 Collection

MAQC Society 2Q2021 Newsletter

 

Greetings from the MAQC Society,

I am glad to report that we had a very successful virtual MAQC Society meeting at the end of April.  This joint meeting together with MCBIOS included over 40 presentations in the MAQC program and enabled the society to have a dual track for the first time.  These presentations highlighted long-term efforts by the SEQC2 consortium and additional interesting research related to imaging, ML, synthetic genomes, RNA and DNA-based biomarkers, multi-omics methods, general methods, single-cell methods, and standards.  We presented awards for best presentations (winners can be found on our home page – https://maqcsociety.org).  We also presented awards for members and non-members publishing scientific papers in 2020 that promoted the mission and objectives of the society.  Finally, we had the opportunity to honor Dr. Weida Tong by naming him Chair Emeritus of the MAQC Society. 

 

Wendell Jones, PhD

Wendell Jones, PhD

Chairperson of MAQC Society

This was our first attempt in conducting a virtual meeting.  I want to especially thank Dr. Steven Foley, Dr. Shraddha Thakkar, Dr. Binsheng Gong, and the rest of the support staff at FDA-NCTR for evaluating conference software, configuring the meeting setup, and for administering and managing the joint conference itself.  This was a large effort on their part and we were able to make every speaker’s presentation available and accessible to the community at the designated time (no technical glitches!).   I also want to thank all the speakers for their contributed work, for the recordings of their presentations, and for being available virtually for Q&A, which helped make the conference run smoothly.  From the meeting and the promotion efforts leading up to it, we now have a society membership of roughly 200 persons.  This is a significant milestone as, up to this point, the society had not kept formal membership rolls.   As much as we enjoyed interacting at the virtual MAQC Society meeting, please make plans now to come in-person to the 5th annual MAQC Society meeting on SAS Campus in Cary, NC USA April 27-29, 2022.  I look forward to the time when I can greet you face-to-face. 

Another first for the society was conducting a webinar series on SEQC2 efforts and research.  This 12-part Webinar series was well-attended and served as a means for not only educating the community on important research being conducted by the SEQC2 consortium but also provided a mechanism to promote society membership and advertise the 2021 annual meeting.  I want to thank our speakers:  Dr. Weida Tong, Dr. Wenming Xiao, Dr. Li Tai Fang, Dr. Charles Wang, Dr. Mohammad Sahraeian, Dr. Mehdi Pirooznia, Dr. Joshua Xu, Dr. Binsheng Gong, Dr. Christopher Mason, Dr. Jonathan Foox, and Dr. Huixiao Hong for highlighting their (and the consortium’s) research during the webinar series.  We hope to make the webinar presentations available to MAQC Society members and otherssoon.

In addition to the presentations, publications by members of the society and the SEQC2 consortium are starting to appear on-line or in print.  Check out our Papers page Papers | MAQC society (themaqc.org) for the latest.  The SEQC2 has two papers already appearing in Nature Biotechnology, two papers appearing in Genome Biology, and one paper describing the scRNA-seq dataset in Scientific Data.  In addition, other papers from the Somatic Sequencing working group have been accepted by Nature Biotechnology and still others from SEQC2 will appear in Cell Genomics and other journals over the next few months.  Keep checking the society web pages for updates.

Finally, I wanted to make you aware of a new MAQC Society project.  The area is inflammatory biomarkers and the immune system. The study is attempting to examine optimal methods for cell identification by comparing single-cell technologies such as the different methods of cell cytometry including standard flow, mass cytometry, and spectral flow cytometry as well as different methods of single cell RNA sequencing.  The sample requirements to perform a multi-lab, multi-platform study of single cells are challenging in that these methods typically need a large number of viable cells to perform optimally.  We are in the middle of identifying and acquiring the necessary number of patients/cells suitably preserved for this study, which is a nontrivial task.  We will provide an update next quarter as to our success and needs.  If you have an interest in participating in this study or have an idea for a suitable new study that can leverage the knowledge and insight of the MAQC Society, please contact me (wendell.jones  AT  q2labsolutions.com ). 

 

Regards,

Wendell Jones, PhD

Executive Chair of the MAQC Society